Configuring A/B tests

A/B tests allow you to seemlessly test various index settings. This is extremely helpful while optimizing your search engine settings. You can create indices with different settings and compare them in real time to your production index or you can test two or more indices in one test, these are just examples of business scenarios.

The advantage of the feature is that when the test is set up, the traffic that reaches the base index is relocated among the base index and variants, according to the customer allocation set up in the test. So you won’t need to implement any changes on your frontend side.

Before you configure an A/B test, you must create indicies with settings that you want to test. For the test to be conclusive these should be singular changes to the base index.

Prerequisites

• Import a product feed to Synerise. You can find instructions here.
• Implement the transaction events and item.search.click event that is generated when an item shown in the search results is clicked. Use this API method.
• Create indexes
  Tip: In order to avoid creating the settings from scratch you can duplicate the base index (with or without its rules and synonyms). Go to AI Search > Indexes, and on the index you want to duplicate, click > Duplicate. We recommend introducting singular changes to each version of the base index, so the test will be conclusive.

Create A/B test

1. To create a new test, go to AI Search > A/B tests > New A/B test.
2. Enter the name of your A/B test.
3. In the Variants and profile allocation section, click Define.
4. Click the Base variant tab.
5. From the dropdown list, select the index which will be your baseline.
   It’s highly recommended to choose a production index.
6. To add the next variant, click .
7. From the Index dropdown list, select an index for this variant. You can rename the tab for better management of the versions.
   
   WARNING: You can’t use an index which is being used in another A/B test.
8. To add the next variant repeat steps 6 and 7.
   
   Important: You can add up to 5 variants.
9. In the Profile allocation section, assign the percentage of customers to each variant.
   Once a customer is assigned to an index, the customer will be receiving items scored by that index in the search results.
10. Confirm the settings by clicking Apply.
11. If you want to save the test without launching it, click Save.
12. When the test is ready to launch, click Save&Run. After the test runs so long that all variants have searches, you can check the Statistics.

After the test, you can export the settings of a winning index to the baseline index (provided you selected the production index as your baseline index).

Export settings of the winning version

Exporting settings overrides any settings of the base index and finishes the A/B test. You may also end the test, analyze the results and export the settings later.

1. Go to AI Search > A/B tests.
2. From the list, select the A/B test.
3. Click the Statistics tab.
4. On the variant, click Export settings.
5. On the pop-up, click Confirm.