Test and learn is a way of running marketing in which decisions come from small, controlled experiments rather than opinion. You form a clear idea of what might improve results, try it on part of your audience or budget, measure what happened against a comparison group, and let the outcome decide what you do next.
How test and learn works
The approach runs as a loop:
- Start with a question that matters. For example: would showing prices on our service pages bring in better-qualified enquiries?
- Write a hypothesis. “Adding a ‘from £’ price to the boiler installation page will reduce enquiries slightly but increase the share that become booked jobs.” A good hypothesis names the change, the expected effect and the measure.
- Design a fair test. Change one thing, keep a control, split traffic or budget at random, and decide in advance how long the test runs and what result would count.
- Run it and leave it alone. Stopping early because the first week looks promising is the most common route to a wrong answer.
- Record the result and act. Roll out what worked, drop what did not, and write down what you learned, including tests that showed no difference.
The method applies almost anywhere: A/B testing of landing pages, email subject lines and ad copy; campaign experiments in Google Ads that split traffic between a control and a trial; holdout groups who receive no emails, to measure the true effect of a sequence; or pausing a channel in some regions to see what changes.
Why it matters
Marketing is full of confident opinions, mine included, and plenty of them turn out to be wrong for a particular business. A test-and-learn habit replaces arguments about what should work with evidence about what does, on your own customers.
It also limits risk. Instead of rebuilding a whole website or moving an entire budget to a new platform, you try the change on a portion first. If it fails, you have lost a little; if it works, you scale it with confidence.
Over time, the record of tests becomes an asset in its own right. A UK retailer with two years of logged email, offer and landing page tests knows things about its customers that a competitor cannot copy.
Common mistakes
- Testing trivia. Button colour tests rarely move a business. Test offers, prices, headlines, audiences and channels first.
- Too little traffic. Small sites cannot detect small differences. Work out the sample size you need before starting, and test bolder changes if traffic is low.
- Calling it too early. Let the test reach its planned sample size and duration before you check for statistical significance; checking daily and stopping at the first significant reading inflates false positives. Cover at least one full week so weekday and weekend behaviour are both included.
- Changing several things at once without a design that can separate their effects.
- Ignoring the calendar. A test run over a bank holiday weekend or the run-up to Christmas may not hold in an ordinary month.
- Not writing results down. Unrecorded tests get repeated, and the lessons leave with whoever ran them.
How to act on it
Keep a simple test log: a spreadsheet with the question, hypothesis, dates, audience, result and decision. Then commit to one test a month that addresses a real commercial question.
Prioritise by likely impact and ease. A new headline on your busiest landing page is quick and can matter; a full redesign is slow and hard to read. For paid media, use the platforms’ built-in experiment tools, which handle the traffic split for you.
Agree with everyone involved that a test showing no difference is still a result: it stops you spending time and money on something that does not help. Building a steady testing programme into ongoing campaigns is part of the performance marketing work I do.