A/B test in AI Search

Published November 19, 2021

Modules

Difficulty

Selected Clients

Industry

By preparing A/B tests in AI Search, you can find out which configuration of the AI search brings better results in terms of conversion, revenue, and CTR in your business. This use case describes an implementation that compares the results of presenting search results promoting bestsellers against personalizing the results.

The bestsellers are selected according to item popularity, whereas personalization uses the history of a customer’s activity.

Prerequisites

Import a product feed to Synerise. You can find instructions here.
Implement the item.search.click event that is generated when an item shown in the search results is clicked. Use this API method.

Process

In this use case, you will go through the following steps:

Prepare two search indexes

As the first part of the process, create two separate search indexes (personalization and bestseller) which will differ only in the settings of search results hierarchy.

Go to AI Search > Indexes.
Click Add index.
Follow the instructions on the interface until the configuration of Ranking (step 4). If you need more information, you can read instruction on configuring AI Search index.
Set the ranking for the personalization search index. You will create the besteller version later.
- For the personalization search index:
  - Frequency of item page views to 70%
  - Frequency of item purchases to 0%
  - Personalization to 100%
- For the bestseller search index:
  - Frequency of item page views to 20%
  - Frequency of item purchases to 80%
  - Personalization to 0%
Save the index. The index will be ready in several minutes.

View after saving the index
Create the bestseller search index. Apply the same settings as with the personalization search index apart from the Ranking settings.

Prepare AB test

Go to AI Search > A/B tests > New A/B test.
Enter the name of the test.
In the Variants and customer allocation section, click Define.
Click the Base variant tab.
From the dropdown list, select one of the indices you created in the previous part of the process.
Add the next variant by clicking .
From the Index dropdown list, select the second index you created in the previous part of the process. You can rename the tab for better management of the versions.
In the Customer allocation section, assign the percentage of customers to each variant.
Confirm the settings by clicking Apply.
Click Save&Run.
After some time you can check the statistics of the A/B test.

Analyze results

When the A/B tests has been running for long enough, you can analyze the statistics for each variant.

Go to AI Search > A/B tests.
From the list, select the test.
Choose the Statistics tab. You can find there the results for key performance indicators such as a conversion rate or revenue.

Tip: You can find more information about A/B test statistics here.
Compare the statistics of the two indexes you created.
You can export the settings of the winning variant to your base index.

Check the use case set up on the Synerise Demo workspace

You can check the:

Index A,
Index B,
A/B test, directly in Synerise Demo workspace.

If you don’t have access to the Synerise Demo workspace, please leave your contact details in this form, and our representative will contact you shortly.