Note! To be able to use the A/B Test Module, you first need to implement the Tweakwise Event Tag. You can find more information on this in our documentation.
The A/B Testing module in Tweakwise allows you to compare the performance of two different configurations (for example, two different sorting orders, builder templates, or filter templates). You use A/B tests to determine in a data-driven way which adjustments in your Tweakwise setup yield the best results, such as a higher conversion rate or more revenue. You test a control version (A) against a variant (B) to see which performs better.
A/B testing in Tweakwise consists of three parts: configuration and measurement. Follow the steps for each part.
Configuration
Navigate to the A/B Test module in the Tweakwise App and click on + New A/B Test.
2. Type
Select the type of Tweakwise configuration you want to test: Builder templates, filter templates, or sorting templates.
Click on Next.
3. Parameters
Name and description: Enter the name of the test and a brief description. This helps you quickly recognize the test later in the overview.
Start date and end date: Plan the duration of your test here. The advice is to run a test for a minimum of three weeks. This ensures you include two full weekly cycles and minimizes the chance of skewed data.
Select the templates:
A control template: Select the template that serves as the basis. This is the current, live configuration you are comparing the variant against.
B variant template: Select the template with the adjustments you want to test. Ensure the modifications are clear and targeted.
4. Click on Next to go to Step 3: Confirm and finalize the test. After saving the test, you must publish it to set the test live.
Measurement
In the A/B testing overview, you see the performance of your test. Statistics are refreshed daily (at midnight, CET).
Here is how the performance of the control group (A) compares to the variant (B), and what this means:
Revenue: the total revenue generated by visitors in this test group.
CR (Conversion Rate): the percentage of visitors in this group who made a purchase.
AOV (Average Order Value): the average value of purchases made by this group.
UPT (Units per Transaction): the average number of items purchased per transaction by this group.
Status: indicates the statistical reliability of the test: Significant (clear winner), Inconclusive (not enough data yet), or No Change (no meaningful difference).
2. Important notes⚠️
Statistical significance: always run a test until you have collected sufficient data. The recommended minimum duration of three weeks helps with this.
Scope of the test: the A/B test is automatically applied to all categories to which the Control Template (A) is linked.
Always publish: after creating an A/B test, you must set it to Active / Publish before it goes live on your webshop.
Focus on one variable: only test one aspect at a time (one template or one setting) to ensure that the measured gain is attributable to that specific change.
3. FAQ ❓
How long should I run an A/B test?
Run a test until you achieve statistical significance. Adhere to the recommended minimum duration of two weeks to limit noise from daily or weekly fluctuations.
What happens when I turn off the test?
As soon as you turn off the test and publish, all visitors will revert to seeing the standard configuration (either the Control group A or the global default, depending on your implementation).
How does Tweakwise know if a purchase belongs to test A or test B?
The customer's unique ProfileKey is linked to the A/B group. When the purchase is registered via the Analytics JS, this ProfileKey is transmitted, allowing the purchase to be assigned to the correct A/B group.
Why do I see 0% for the distribution of my new test?
The statistics are refreshed daily (around midnight). It is possible that you have just created the test and the initial data still needs to be collected and processed. Check the next day to see if the distribution has started.
Can I extend or stop the test prematurely? Yes, you can adjust the test at any time.
To extend it, adjust the End date in the test parameters and publish again. To stop it, manually stop the test and then publish.
How can I verify the test?
This can be done in the Demoshop. Open the Demoshop for a category that has an active A/B test; you will immediately see a notification that a test is running and which variant you are in. You can change your personalization key in the Demoshop's configuration to check if you land in the other variant (B), allowing you to view both results.