Enrich with Catalog

The Enrich with Catalog transformation step allows you to enhance your dataset by adding information from a selected catalog. It works by matching records in your data with records in the catalog, using a designated item key column from your dataset and the primary key column from the catalog.

Catalogs store different types of data and are often used to enrich profiles imported into Synerise or to add more details to data sent outside the system. This step makes data transformation more powerful and helps Automation Hub and the Catalogs feature work better together.

Additional important notes

• When enriching data, if the enriched columns already exist in your dataset, the values will be overwritten by the catalog values.
• The catalog’s primary key column will not be included in the enriched dataset.
• Once enriched data is imported into Synerise, the enrichment cannot be undone or reverted.

Requirements

• You must be assigned with a user role with the catalog permissions.
• You must have at least one Synerise catalog with data.

Step configuration

1. In the Data Transformation Studio, add the Enrich with Catalog step.
2. Click Select catalog and select the catalog which will be used for enriching the data.
3. From the Match catalog item by column dropdown list, select the sample data column that will be used to match records from the catalog.
   The values from the declared column should correspond to the values in the primary key column of the catalog you select for enrichment.
4. In the Enrich data with columns section, click Choose columns.
   Result: A pop-up appears.
5. Select the columns from the catalog which will be added to the data output.
6. Optionally, you can add more catalogs to enrich the data by clicking Add rule and repeating steps 2-5.
   Make sure these catalogs and the data being enriched contain a shared key (column).
7. To define what happens if rows contain errors, from the dropdown list next to Add rule, select one of the options described in Handling incomplete data.
8. Confirm by clicking Apply.
9. You can check the result in the Output tab.

Handling incomplete data

When there is no matching record in the catalog for a given item key or if cell values fail (for example, Jinja rendering failure), an error occurs and you can choose how the step behaves:

• Skip row if error occurred - Skips the row if no matching record is found or an error occurs and continues processing other rows.
• Skip if result is null or empty string - If a value from a catalog is null or empty, such rows will be skipped.
• Stop further transformation - Stops the transformation process with an error.
• Insert null if error occurred - If there is no matching record in the catalog, a selected column will be added with the null value.
• Insert empty string if error occurred - If there is no matching record in the catalog, a column with empty strings is added.

Example of use

In this example, we will create a transformation which will enrich information in transaction.charge events during the import of transactions to Synerise. Based on the store ID in the event, we will enrich it with the store’s address data stored in the Catalog.

For the purposes of this example, you must create a catalog which contains the ID of the store (this column will be a primary key), and other address details in separate columns.

shopId,shopName,address,city,state,zipCode,country,phoneNumber,email,openingHours,managerName,hasFuelStation,numberOfEmployees 001,Shop XYZ Lake Highlands,123 Main St,Dallas,TX,75201,USA,214-555-0101,downtown@example.com,06:00-23:00,John Smith,Yes,15 002,Shop XYZ Lakewood,456 Oak Ave,Dallas,TX,75202,USA,214-555-0102,midtown@example.com,06:00-23:00,Sarah Johnson,No,12 003,Shop XYZ Oak Cliff,789 Pine Rd,Dallas,TX,75203,USA,214-555-0103,central@example.com,07:00-22:00,Mike Brown,No,10

client.id,orderId,paymentInfo.method,products.finalUnitPrice.amount,products.sku,products.quantity,revenue.amount,revenue.currency,value.amount,value.currency,source,shopId,products.finalUnitPrice.currency 21052,ORD12345,Credit Card,29.99,SKU001,2,59.98,USD,60,USD,WEB_DESKTOP,001,USD 21052,ORD12346,PayPal,15.50,SKU002,1,15.50,USD,15.50,USD,MOBILE_APP,002,USD

Creating a transformation

1. Go to Automation Hub > Transformations > New transformation.
2. Add sample data with transactions you will import to Synerise.
3. Click Add step and select Enrich with Catalog. In the configuration of the step:
   1. Click Select catalog and select the catalog that contains shop information (it must contain a shopId column).
   2. From the Match catalog item by column dropdown list, select shopId.
   3. In Enrich data with columns, click Choose columns and select the columns from the catalog which will be added to the data output.
   4. Confirm by clicking Apply.
      Result: In the Output tab, the transactions contain the columns from the catalog.
4. Click Save.

Creating a workflow which imports transactions

In this part of the example, you will create a workflow which imports transactions, but before the import, the data will be enriched with shop information according to the transformation created in the first part of the example.

1. Start with the Scheduled Run node. In the configuration of the node define the frequency with which the workflow will be triggered.
2. Retrieve data for import by using one of the following nodes:
   • SFTP > Get File
   • HTTPS > Get File
   • Amazon S3 > Get File
   • Azure > Get File
   • Google Cloud Storage > Get File
   • Local File
3. Add the "Data Transformation Studio" node. In the configuration of the node:
   1. Click Templates library.
   2. Select the transformation you have created in the previous part of the process.
   3. Click Apply.
4. Add the Import transactions node. This node doesn't require configuration.
5. Add the End node. This node doesn't require configuration.
   Result:

   

6. Start the workflow by clicking Save & Run.
   Result: The transaction is added to the profile activity list as a transaction.charge event, which is enriched with the shop's address and the shop manager's name