A/B/X testing

Recommendation A/B/X testing is designed to perform multivariate testing of your existing recommendation campaigns directly in the Synerise platform. This feature lets you identify the best-performing recommendation configuration for a page section or even an entire page without the need for external tools or side calculations. The feature doesn’t require any changes to your production deployment.

The test is run on the baseline recommendation, which means you request the same recommendation ID regardless of the variant, as the traffic split is handled entirely automatically by Synerise. At the end of the test, you can overwrite the baseline variant configuration with the winning one without any development work on your side. Winners are declared automatically for given metrics based on statistical significance.

Key features

• Launch tests without any changes to your existing setup.
• Compare up to 5 variants, including your baseline campaign.
• Precisely control user exposure to each variant.
• Access pre-built dashboards showcasing critical metrics (such as conversion rate, click through rate, average revenue, and more)
• Automatically declare winners based on statistical significance.
• Implement winning variants without developer intervention.
• Convenient outcome preview - you can preview and compare recommendation results straight in the module.

Feature overview

1. A/B/X Test Setup

   • You can create up to 5 variants (including baseline) in a single test.
   • Each variant requires selecting one recommendation (draft or active status).
   • Variants can be distributed equally or manually with allocation between 1% and 99%.

2. Variant Assignment

   • When a request for the baseline variant for a customer is made, the customer is randomly assigned to one of the test variants.
   • UUID is the profile identifier used to assign customers to groups. If a customer logs in on multiple devices or browsers, this customer can be assigned to more than one variant.
   • Profiles are assigned to a variant for approximately 180 days.
   • A variant.assign event is generated on the profile card.

3. Usage Restrictions

   • Each recommendation can only be used in one A/B/X test at a time.
   • There are no limits on the number of active A/B/X tests.
   • This feature ignores the global control group - customers who belong to the global control group can be shown recommendations regardless.

4. Combination Restrictions

   • The baseline recommendation type determines the allowed recommendation types in A/B/X testing:
     • If Section page recommendation is the baseline, all variants must be of the Section page type.
     • If Attributes recommendation is the baseline, all variants must be of the Attributes type.
     • No restrictions apply if any other type is the baseline.

5. Displaying Recommendations

   • Refer to guidelines in the “Embedding a recommendation in templates” section.
   • Recommendations are presented to the defined audience based on recommendation settings.

6. Performance Evaluation

   • A/B/X tests run indefinitely by default. Performance metrics such as clicks, CTR, and revenue are continuously calculated. You can stop the A/B/X test manually.
   • Once the test is complete, you can select the best-performing variant. Regardless of the selected variant, the ID of the recommendation will remain the same and no further actions are required.

7. Custom Analysis

   • You can perform analyses using recommendation events and the following event parameters:
     • variantId: Unique identifier of the variant.
     • variantName: The name of the variant.
     • experimentId: Unique identifier of the test.

Requirements

• You must create at least two recommendations.
• The recommendations you select for A/B/X testing must have the same preview attributes.
  In the “Selecting attributes for preview” section you can learn to configure preview attributes.

Configuration

1. Go to Communication > Recommendation A/B/X tests > New A/B/X test.

2. On the top of the screen, you can change the name of the A/B/X test.

3. In the Variants and profile allocation section, click Define.
   Result: A configuration form for variants is displayed. Two variants are automatically created: Base variant and Variant.
   

   

4. In the Base variant tab, from the Select recommendation campaign dropdown list, select a recommendation that will be used as the base variant.

5. Go to the “B” Variant tab.

   

6. From the Select recommendation campaign dropdown list, select a recommendation for this variant.

7. To add more variants, click and select recommendations for them.

8. To customize variant allocation, in the Profile allocation section, use the slider and set preferred variant allocation.

9. Confirm the settings by clicking Apply.

10. To:

    • save the test as a draft, click Save.
    • save and activate the test, click Save & Run.

Statistics

Statistics are based on the recommendation events, product.buy and transaction.charge events. The following metrics are calculated:

• Clicks - The number of unique clicks on a recommendation frame.
• Unique users - The number of customers for whom a recommendation was generated.
• CTR - Click-through rate: the number of clicks on a recommendation frame divided by the number of generated recommendations.
• Conversion rate - The number of unique purchased items divided by the number of all clicks on an item within a recommendation frame.
• Average revenue - The average revenue (per customer) the recommendation generated during the test. It is calculated by dividing the total revenue by the number of unique customers who made a purchase.

How to interpret winners?

The winner status is declared exclusively for individual metrics and not for the entire variant. The following metrics can be declared as winners: CTR, conversion rate, and average revenue.

• For CTR and CR, we use one-tailed 2-proportion Z-test with 95% confidence level, so if the variant metric is labeled “WINNER”, it means that there is a 95% probability that the observed metrics difference is not due to random chance.
• For Average Revenue, we use one-tailed 2-sample T-test, with n-2 degrees of freedom, so if the variant metric is labeled “WINNER”, it means that there is a statistically significant difference in Average Revenue between the groups.

Where to find statistics?

To view the A/B/X test statistics:

1. Go to Communication > Recommendation A/B/X tests.
2. On a running or finished test, click
3. From the dropdown list, click Show statistics.
   Result: The preview of A/B/X test statistics is displayed.

   

Statistics overview

*Revenue - Revenue generated by the recommendations. It is counted when a customer buys an item within 24 hours after clicking the item in the recommendation frame.