The first step towards meaningful, actionable results is understanding statistical significance and having confidence in test results.
Intelligems uses Bayesian statistics to analyze A/B tests. Because groups are randomized, we are able to measure significance through standard statistical methods, such as z-tests for Bernoulli distributions, such as conversion rates, and t-tests for more discrete metrics, such as revenue and profit per visitor.
As a rule of thumb, we typically need 150+ orders per group to accurately measure a 10% change in conversion rate. Once your test has reached these order volumes and has run for at least one full week, we recommend using the Stat Sig tab in the analytics dashboard in the app to get a read on confidence in the results.
See below for an example of what our statistical significance analysis looks like on a revenue basis! On the x-axis is '% Revenue per Visitor Opportunity Cost' and the y-axis '% Probability' — so the plot shows the probability that the winning group is truly the winner from a revenue perspective, or, if it’s not better, then at least not worse by more than [x]%. For example, in the below analysis, we’re 79% confident that the current winner of this test is a winner (or not worse by more than 1%) from a revenue standpoint. We usually like to use the 1% opportunity cost value and aim to see this value around 80-90%.