1. Knowledge Base
  2. Analyzing Results

Determining Statistical Significance

The first step towards meaningful, actionable results is understanding statistical significance and having confidence in test results.

Intelligems uses Bayesian statistics to analyze A/B tests. Because groups are randomized, we are able to measure significance through standard statistical methods, such as z-tests for Bernoulli distributions, such as conversion rates, and t-tests for more discrete metrics, such as revenue and profit per visitor.

As a rule of thumb, we typically need 200-300 orders per group to accurately measure a 10% change in conversion rate. Once your test has reached these order values and has run for at least one full week, feel free to reach out to us at support@intelligems.io if you'd like us to run a statistical significance analysis.

See below for an example of what our statistical significance analysis looks like on a revenue basis!  On the x-axis is '% Revenue per Visitor Opportunity Cost' and the y-axis '% Probability' — so the plot shows the probability that the winning group is truly the winner from a revenue perspective, or, if it’s not better, then at least not worse by more than [x]%. For example, in the below analysis, we’re 79% confident that the current winner of this test is a winner (or not worse by more than 1%) from a revenue standpoint. We usually like to use the 1% opportunity cost value and aim to see this value around 80-90%.

✨ Note that we are also able to run this analysis on a profit basis!