Introduction

Ask most teams what makes a successful A/B test and they'll mention aspects such as a redesigned page, a rebuilt journey leading to resource intensive development and design to create and push the test live. This leads to the assumption that if you put more into a test, you'll get stronger results.

We wanted to know whether that's true. So, we analysed Fresh Egg's experiment repository, analysing hundreds of concluded A/B tests, run across more than 30 clients and across 10 industries and compared results by build size.

This revealed an interesting trend, bigger isn’t always better. Our smallest builds won more often, and delivered bigger uplifts when they did win. The more useful question is when a larger build is worth the investment - and the data has a clear answer.

The challenge

For businesses, development time is often the most expensive resource in a testing programme. Every large build ties up design, development and QA capacity that could otherwise fund several smaller experiments.

That trade off only makes sense if larger tests drive stronger return on investment. If they don't, programmes are slowing their own velocity and insight, often without realising it, because one large test feels like more progress than four small ones.

It's also a barrier to getting started. For organisations new to experimentation, the perception that A/B testing means large redesigns and budgets can impact a programme getting started.

From our analysis, we wanted to answer one question with evidence - do larger test builds deliver stronger outcomes?

What we analysed and how

We analysed every concluded test in our experiment database, spanning over 9 years. We categorised each test by:

  • Build size: Small, medium or large based on development resource, design and QA

  • Outcome: Winner, loser or inconclusive

  • Strategic or tactical: Whether the test challenged the proposition (strategic), or refined the presentation and UX (tactical)

We calculated the win rate as the share of all concluded tests that produced a winner, with losers and inconclusive results also counted in the total. The median uplift across test size and other related categories was calculated to understand the impact on performance.

The findings

When analysing the data, across build size, industries and test type, the analysis highlighted 3 key findings.

  1. Small builds win more often, with larger uplifts

    Small builds won 46.5% of the time. That's around 34% higher than large experiments.

    Build size

    Win rate

    Loss rate

    Uplift (winners)

    Small

    46.5%

    15.5%

    +8.8%

    Medium

    31.8%

    21.6%

    +6.4%

    Large

    34.7%

    22.4%

    +6.0%

    They were also the lowest risk option, losing just 15% of the time compared with around 22% for medium and large builds. And when small builds win, uplifts tend to be stronger.

    That results in a higher win rate, stronger uplifts and a lower build cost, meaning small builds deliver a far stronger return on investment.

    Some of the most impactful winners were among the simplest: For example, simply using a ‘Which? Best Buy’ logo increased pet insurance sales by +20%. To ensure data integrity, the median uplift across experiments has been used due to large conversion uplift outliers.

  2. Large builds more impactful for strategic testing

    None of this means large builds have no place in a testing programme. But the analysis and insights are clear about when large builds are most successful.

    Large builds that tested a strategic change such as, a new proposition, offer or way of selling won 42% of the time. Large builds used for tactical, presentational changes won just 28% of the time. The same pattern was seen for medium builds.

    This insight suggests larger builds are important when testing a genuinely different, more strategic idea or feature. When it's testing a different look for the same idea, a smaller, more focused change is likely to be more impactful.

  3. Charities: proof that you can make large impacts with smaller build tests

    Charity was the sector where small builds stood out the most. Low development effort charity tests won 58% of the time, with medium builds in the same sector winning just 22.2%.

    The wins weren't marginal, either. For example, a small change in the donation form, to drive persuasion increased donations by +10%, and a simple membership page copy test saw donations increase by +21%. Both are lower effort tests for design and build but make a huge difference for charity donations.

    For charities, this matters. Many charities do not prioritise A/B testing because they assume it needs significant budget and development resource. Our data suggests the opposite. Some of the strongest results in the sector came from simple, tactical changes such as banner messaging and optimised calls-to-action.

What this means for your testing strategy

Build size isn't a proxy for impact. The tests that win most often are focused, well-evidenced changes aimed at a clear behavioural barrier and not always large, resource intensive builds.

If you're planning your next testing roadmap, here's what we'd take from the data:

  • Focus on small tactical optimisations first. Small builds win more often, lose less often and cost less to build. This means faster testing and increased likelihood of a winning experiment.

  • Larger builds for more strategic testing and insights. Use larger builds to test a new proposition or feature.

  • Don't let budget stop you starting. Particularly for charities and smaller teams, simple changes can deliver significant results from day one. You will be surprised how simple changes can make a big impact.

Get in touch

If you'd like help building a testing programme that learns faster and wins more often, we're here to help. Talk to us about your experimentation challenges or upcoming projects and learn how we can help you drive results through experimentation.

Get in touch

Get Fresh Thinking in your inbox

Get expert insights, event invites, and practical guides on digital marketing - every fortnight, in our newsletter.

This site is protected by reCAPTCHA.
You may unsubscribe from these communications at any time. Please review our Privacy Policy.