At a time when consumer indicators are in decline, protecting your cash flow is more of a priority than ever. That's why we've decided to share with you an essential approach for maximizing the impact and profitability of your marketing actions: AB testing.
Why AB testing?
This article may seem complex, with its mathematical concepts and specialized terms, but we invite you to read it carefully.
Indiscriminate media spending is a thing of the past, and ROAS are not what they used to be. To guarantee profitable growth, it's imperative to keep a tight rein on your advertising investments.
AB Testing is the only scientifically validated method for proving the profitability and effectiveness of your marketing actions.
The fundamentals of AB testing
A randomized trial (a general method which includes AB testing) is a type of scientific study used in many fields, particularly in medicine, where they are considered to be part of the gold standards for proving the effects of different therapeutic approaches (drugs, surgery, etc.).
Understanding the steps involved, from randomization to statistical validation, will enable you to integrate this powerful tool into your marketing strategy.
However, it is essential to stress that statistical validation is a matter for experts. It requires an in-depth understanding of complex and abstract concepts (e.g. binomial distribution, p-value...) to assess the significance of AB testing results.
A real-life example of how to conduct an AB test
To optimize an advertising campaign on Facebook, you want to evaluate the difference in performance between two creatives (creative A and creative B). Suppose you randomly assign a population A (unexposed to the creative) and a population B (exposed), and manage to guarantee the absence of bias and migration between these two populations (these essential conditions for certifying results are already very challenging to guarantee):
After a while, you notice that creative A has generated 100 sales, while creative B has generated 120.
If you stick to this rough comparison, you could conclude that creative B performs better (+20 sales) and allocate 100% of the budget to it.
In general, actors who have not mastered AB testing stop at this conclusion, and therefore make the wrong decision.
Indeed, without statistical validation, analysis of the situation can be misleading. For the difference in sales (+20) cannot be interpreted as an absolute figure, but rather as a statistical law. Every day you calculate the cumulative incremental sales figure, the result draws this curve:
It's a Gaussian curve.
And the more you carry out this calculation, the more the figure will tend towards the top of this Gaussian (law of large numbers) and fit into a confidence interval, indicating the potential variability of the results.
At this point, a statistical specialist should be called in to perform complex calculations and determine whether the difference in sales between A and B is statistically significant or simply due to chance.
Following multiple statistical rules, he or she might conclude: "Crea B has a 95% chance of achieving between +5 and +25 sales, but the most likely is +12".
Or, even if you have found a difference of +20 at an instant T, which remains a probable value, conclude that the result is "between -10 and +30 sales"... So the difference could be "-10" (in favor of A).
I'll leave to you to imagine the consequences if you'd chosen creative B.
Another pitfall of AB testing is the lack of a "red thread"
Another (real) example...
Let's imagine an e-tailer who, following an AB test in September 2022, discovers a ROAS of 1.2 for META. On the basis of this information alone, he significantly increases his investments. However, by neglecting continuous monitoring through AB testing, he finds himself with results well below expectations six months later.
The decision to increase his META investments was based on the rule of three comparing the ROAS of AB testing (1.2) with the ROAS announced by META (1.8), and relying on the reality of variations (trends) in the ROAS displayed by META to adjust his rule of 3.
In fact, confidence in the figures announced by META proved to be a source of error, above all in assuming that the trends displayed by META were accurate. After all, throughout this period, META only ever showed slight variations in ROAS trends (+/-10%)... Whereas a new AB test proved, six months later, that its ROAS had fallen to 0.8 (-35%!).
Ignoring this reality had significant consequences for the long-term profitability of its advertising investments. This example highlights the risks of blindly relying on ROAS calculations (especially trends) provided by platforms like META, underlining the need to question these figures and base decisions on in-depth analysis rather than belief.
Conclusion :
Accurate assessment of your key indicators, such as ROAS or LTV, is of crucial importance. The resulting decisions, whether to adjust budgets or prioritize a project, have an immediate impact. In these critical times, you need to avoid the penalties of making choices based on erroneous analyses or hunches.
Secondly, even if these figures are valid at a given moment, they evolve over time. Not following them over the long term means making decisions on the wrong basis, thus compromising the stability of your growth.
On the other hand, skilfully implementing an AB test, mastering its interpretation (even if the math can sometimes be intimidating), and understanding how to integrate it consistently into all your marketing analyses gives you the ability to steer your growth in a solid, sustainable and secure way. By investing in these practices, you equip yourself with the tools you need to make informed decisions and keep your business thriving.