Multivariate testing is a way of testing several changes on a page at the same time, by showing visitors different combinations of those changes and measuring which combination produces the best result. Where an A/B test compares whole versions of a page, a multivariate test looks at the parts and how they work together.
How multivariate testing works
You choose the elements to test and the variations of each. Say a landing page has three headlines, two hero images and two button labels. Every combination is a version of the page: 3 × 2 × 2 gives 12 versions. The testing tool splits traffic between them and records the outcome you care about, such as form submissions or purchases.
At the end, you get two kinds of answer. The first is which combination won. The second, and often more useful, is how much each element contributed on its own, and whether some pairs work better together than apart. A friendly headline might do well with a photo of people and badly with a product shot. That is an interaction effect, and an A/B test run one element at a time can miss it.
Testing every combination is called a full factorial design. Some tools offer reduced designs that test a subset of combinations and estimate the rest, trading some accuracy for speed.
Why it matters
The appeal is efficiency: one experiment instead of several in a row, and an answer about how the parts of a page relate. For a business with a high-traffic page that matters a great deal, such as a national retailer’s checkout or a lender’s application form, that can be worth the setup.
The catch is traffic. Every combination needs enough visitors and conversions to reach statistical significance on its own. A 12-version test needs roughly six times the traffic of a two-version A/B test to give each version the same number of visitors. Most UK small and medium-sized businesses do not have it. A local service page with 1,500 visits a month and 30 enquiries cannot support a 12-cell test; even a two-version A/B test may take months to settle.
That is why, for most small and medium-sized businesses, multivariate testing is something to understand rather than something to run. A/B tests on bigger, bolder changes usually teach them more, faster.
Common mistakes
- Skipping the sample size calculation and discovering after two months that no result is reliable.
- Testing tiny changes, such as a slightly different shade of button, that could never produce a difference big enough to detect.
- Calling the winner early because one combination is ahead on day five.
- Adding a new element halfway through, which resets the comparison.
- Measuring clicks on the tested button rather than the outcome that pays, such as completed enquiries.
- Running other changes on the same page during the test, so the results are mixed with things you did not plan.
How to act on it
Start with the numbers. Find the page’s monthly visitors and conversions, decide the smallest improvement worth detecting, and calculate how long each combination would need. If the answer is more than a couple of months, run a simpler A/B test instead, with fewer and larger changes.
If the traffic is there, write a clear hypothesis for each element, keep the number of elements low (two or three) and fix the run time in advance. Check the result with an A/B test significance calculator or your tool’s own statistics, and confirm the outcome in your analytics, not only in the testing tool.
Testing is only as good as the page being tested. When I build landing pages for ad campaigns, I plan which elements are worth testing from the start, so the page can be improved on evidence once traffic builds.