The minimum detectable effect (MDE) is the smallest real difference between a control and a variant that an A/B test is set up to detect reliably. If the true improvement is smaller than your MDE, the test will probably finish without a clear winner even though the change did help.
How the minimum detectable effect works
Four numbers are tied together in any A/B test, and fixing three of them decides the fourth:
- Baseline conversion rate How often the current page converts.
- MDE The smallest lift you want to be able to catch.
- Significance level How much risk of a false positive you accept, commonly 5%.
- Statistical power The chance of detecting the effect if it really exists, commonly 80%.
Together they set the sample size each variant needs. The relationship is steep: halving the MDE roughly quadruples the sample you need, because small differences are much harder to tell apart from random noise.
MDE can be stated two ways. An absolute MDE is in percentage points, such as 3.0% rising to 3.5%. A relative MDE is a percentage of the baseline, so the same change is a 16.7% relative lift. Testing tools differ in which they use, so check before comparing plans.
A worked example. A landing page converts at 3%, and you want to detect a relative lift of 20%, from 3.0% to 3.6%. Using the standard sample size formula for comparing two conversion rates, at 5% significance and 80% power, that needs roughly 14,000 visitors per variant, so about 28,000 in total. If the page gets 2,000 visitors a month, the test would take over a year. Raise the MDE to 50% (3.0% to 4.5%) and the requirement falls to around 2,500 per variant, which that page would reach in about two and a half months.
Why it matters
Most UK small and mid-sized business sites do not have the traffic that testing case studies assume. Setting an MDE before the test forces an honest question: can this page, with this traffic, detect the kind of change you are making? If not, you either test something bolder, test on a higher-traffic page, or accept that you are making a judgement call rather than running an experiment.
Skipping this step leads to tests that run for weeks, end inconclusive, and get called on gut feel anyway. Worse, people stop early when the result looks good, which inflates false positives. That habit is known as peeking.
Common mistakes
- Not setting an MDE at all and letting the test run until something looks significant.
- Choosing a relative MDE of 2% or 5% on a low-traffic site, then waiting months for an answer that never comes.
- Mixing up absolute and relative MDE when comparing tools or plans.
- Testing tiny changes, such as button shade, that could never produce an effect large enough to detect.
- Treating “no significant result” as proof the change made no difference. It may simply have been smaller than the MDE.
How to act on it
- Get the baseline conversion rate and weekly traffic for the page you want to test.
- Decide the smallest lift that would be worth acting on commercially. If a 5% improvement would not change any decision, do not design a test to find it.
- Use a sample size calculator to see the visitors needed, then convert that into a test duration using your real traffic.
- If the duration is unrealistic, test a bigger change or a page with more traffic.
- When the test ends, check the result with an A/B test significance calculator and report the confidence interval, not just a winner.
On landing pages for paid traffic I usually recommend bold, clearly different variants for exactly this reason, and plan the test length before launch as part of PPC landing page design.