The first step towards meaningful, actionable results is understanding statistical significance and having confidence in test results.
Intelligems uses Bayesian statistics to analyze A/B tests. Because groups are randomized, we are able to measure significance through standard statistical methods, such as z-tests for Bernoulli distributions, such as conversion rates, and t-tests for more discrete metrics, such as revenue and profit per visitor.
As a rule of thumb, we typically need 200-300 orders per group to accurately measure a 10% change in conversion rate. Once your test has reached these order values and has run for at least one full week, feel free to reach out to us at firstname.lastname@example.org if you'd like us to run a statistical significance analysis.
See below for an example of what our statistical significance analysis looks like on a revenue basis! On the x-axis is '% Revenue per Visitor Opportunity Cost' and the y-axis '% Probability' — so the plot shows the probability that the winning group is truly the winner from a revenue perspective, or, if it’s not better, then at least not worse by more than [x]%. For example, in the below analysis, we’re 79% confident that the current winner of this test is a winner (or not worse by more than 1%) from a revenue standpoint. We usually like to use the 1% opportunity cost value and aim to see this value around 80-90%.
✨ Note that we are also able to run this analysis on a profit basis!