Our testing methodology
Fresh Egg tests substantially, scientifically, and safely
What do we test?
Put simply, the elements of a website we test are those that are predicted to have the biggest impact on conversions, which can vary greatly between sites. It may be that your navigation really needs improving, or perhaps your USPs are not clear enough to users.
In reality there are often thousands of site elements that can impact conversions and that could be refined through testing. Our discovery research tells us what the biggest conversions issues are and we test these first, but over time it’s possible we’ll test every major part of your site.
Segmentation and personalisation
Some tests that we run aim to measure the value of segmenting users based on certain qualities and offering a different experience to each segment.
We may also test personalising site appearance and content based on individual user traits or behaviours.
Opportunities to segment or personalise your website can be identified in the CRO discovery, but are often found after analysing conversion experiment results once a test has concluded. Understanding how different audience groups behaved when exposed to different test variations can highlight where new segmentation or personalisation can add value.
Test periods and statistical significance
Concluding tests without sufficient or robust data will likely do you more harm to your conversions than good. Fresh Egg only declares a test as concluded once it has met the following criteria:
- The test variation sample sizes and conversion numbers are large enough to give confident results
- The test is run for a long enough period to account of relevant conversion cycles
- The test reaches as a statistical significance level of 95% or higher (accepted scientific standard)
Tests may be stopped earlier however if they are not demonstrating the necessary signs that suggest they may go on to be conclusively positive. This ensures that the next round of tests is not unnecessarily held up and opportunity costs are minimised.
Before any experiments are set live we undertake a series of quality assurance processes to ensure the experiments will return valid data, and no undue harm will be caused to conversions or to the user experience as a result of the technology used.
Fresh Egg also ensures that analytics tracking is working as it should before new pages are tested and that there are no issues with the experiment that could negatively affect SEO.