4. Measuring and understanding impact
We monitor tests to ensure everything is functioning and that user data is registering against the variants, with all correct analytics events and goals firing. Close monitoring happens until a test reaches statistical significance against primary goals.
Statistical measurement ensures the results are not just due to random chance. We deep dive all our tests – winners, losers and inconclusive results are all gold dust in terms of data and learnings - We segment data to understand how audiences reacted and assess click behaviour to understand changing user behaviours.
Once we have properly understood the impact of the test, we make our onward recommendation, be it full deployment of the winning variant or further refinement.