Enhanced Conversion Lift Report

[CommunityMod_WM]

The Enhanced Conversion - Lift report features conversion attribution insights for enhanced content on select websites where we have access to conversion data and can run experiments.  

When to Use the Lift Report:

The Enhanced Conversion - Lift report should be leveraged to measure the impact of enhanced content on the behavior of shoppers. For all companies, the insights available in this report are crucial in measuring the success of an enhanced content program and identifying opportunities for optimization to promote higher conversion rates. In cases where the enhanced content program is set up for success, this report provides the value of conversions that can be directly attributed to the presence of enhanced content and the ROI for the program overall.

Metrics & Terminology

Sessions – A single session consists of one or more page loads by a unique shopper during a one-hour period

Conversions – The number of sessions – on pages where enhanced content is published - resulting in products being added to the cart or purchased (depending on type of conversion being reported on by site)

Conversion Rate – The percentage of sessions that result in a conversion

• Formula:   Conversions / Sessions

Value – The revenue generated from conversions on pages where enhanced content is published

• Formula: Product price * Units added to cart

Conversion Rate Lift (as calculated in the Lift view) – The percentage lift in conversion rate observed when enhanced content is present. Achieved by running the Holdout Experiment.

• Formula: (Treatment Group Conversion Rate – Holdout Group Conversion Rate) / Holdout Group Conversion Rate

Conversion Rate Lift (as calculated in the Content view) – The percentage lift in conversion rate observed when shoppers interact with enhanced content. This calculation utilizes only the Treatment Group data collected as part of the Holdout Experiment.

• Formula: (Conversion Rate of shoppers who interact – Conversion Rate of no-interaction shoppers) / Conversion Rate of no-interaction shoppers

Time On Page Lift – The percentage lift in time shoppers spend active in their sessions when they interact with enhanced content. This calculation utilizes only the Treatment Group data collected as part of the Holdout Experiment.

• Formula: (Time on Page of shoppers who interact – Time on Page of no-interaction shoppers) / Time on Page of no-interaction shoppers

Incremental Value – The value of conversions which can be attributed to enhanced content being present. Achieved by running the holdout experiment.

• Formula: Treatment Group Total Value - (Treatment Group Sessions * Holdout Group Conversion Rate * Average Treatment Order Value)

Holdout Experiment – On select websites from which Syndigo is able to obtain conversion data and conduct A/B tests, enhanced content is withheld from approximately 5% of new shoppers visiting the URLs where enhanced content has been published. These shoppers are classified into the cohort known as the Holdout Group. Enhanced content continues to load for the remaining 95% of traffic, called the Treatment Group. The difference observed in the same-session conversion rates and other metrics across both groups is utilized to calculate the KPIs and populate the visualizations in the Enhanced Conversion – Lift report.

Holdout Group – New shoppers, representing approximately 5% of traffic, from whom enhanced content is withheld (not visible) during every visit to the URLs where the content was published. Shoppers who are qualified into the Holdout Group or Treatment Group remain in their assigned cohort for 12 months, after which they are eligible to be reassigned upon a subsequent visit. Unique shoppers are tracked over a 7-day period starting with their initial visits, such that attribution can be derived from their add-to-cart or purchasing activity.

Treatment Group – New shoppers, representing approximately 95% of traffic, for whom enhanced content is available to view during every visit to the URLs where the content was published. Shoppers who are qualified into the Holdout Group or Treatment Group remain in their assigned cohort for 12 months, after which they are eligible to be reassigned upon a subsequent visit. Unique shoppers are tracked over a 7-day period starting with their initial visits, such that attribution can be derived from their add-to-cart or purchasing activity.

Treatment Filter – Found in the filter panel of the report, this setting refers to the minimum count of visitors required to deem the data set statistically significant and tells the system to calculate the KPIs. In other words, it helps establish the desired Minimum Sample Size. There are six preset options to choose from: 0, 10, 50, 75, 100, and 1,000 – the default is set to 50. 

Minimum Sample Size – This term refers to the minimum number of visitors required for the data to be considered statistically significant. Does your organization consider fewer than 100 visitors a large enough number to determine the results are telling a compelling story of conversion rate lift, or is it too few? This number may vary depending on organization guidelines, by retailer website, or even by individual product lines. That is why the Treatment Filter may be modified to adjust the minimum sample size to your liking. 

Minimum Effect Size – In order for KPIs to be calculated in the Enhanced Conversion – Lift report, the data set – once the optional filters are applied – must encompass both a minimum sample size and a minimum effect size. The minimum effect size for the Holdout Experiment requires that a consistent and measurable difference in conversion rate and value is observed between the Treatment Group and the Holdout Group shoppers.

Before Report Generation

The standard filter options available across other reports are available to apply or adjust. There are several options here worth special consideration:  

• Date: The Lift reports are unique in that results may vary drastically depending on the volume of data utilized in the KPI calculations. Take time to expand and shrink the date range, regenerating the report each time, to better understand which timeframe best captures the story of conversion rate lift.  
• Currency: It is strongly recommended to filter down to only one currency option if there are multiple currency options available. This will normalize the Incremental Revenue KPIs.  
• Treatment Filter: This number determines the minimum count of Treatment Cohort visitors required to consider a dataset worthy of KPI calculations.  Choose one of the larger minimum visitor count options if you’d like to reinforce stricter standards of statistical significance. Choose a smaller number or even zero if statistical significance is not a priority for your organization, and you would like to observe how the KPIs may be calculating with much less data than is recommended. 50 is the default minimum Treatment Cohort visitor count Syndigo recommends starting with, but return to this filter and adjust for the various options to observe how the story of lift may change.  
• Additional filters: As with other reports in the suite, additional filter options may be added via the gear icon in the upper right corner of the filter panel. Specific product attributes, for instance, may be added and applied through this interface.  

Make modifications to the default settings or preserve the defaults presented and click the Generate button when ready.  

Account & Site Overview

The first page of the report features the KPIs derived from Syndigo’s ongoing holdout experiments calculated from the context of two different levels of aggregation – all data available in the account based on the filters applied and the data for each site.

At the top of the report, the summary cards contain the calculations for the Conversion Rate Lift and Incremental Revenue KPIs based on all data available after the optional filters in the report are applied (i.e. date range). If no filters are applied to the report generation beyond date range, all data from all URLs where the Holdout Experiment was conducted is utilized in the calculations for the summary cards.

The second section of the report, the Retailer Summary, features two charts representing the KPIs as calculated per website. The websites presented must qualify for inclusion in the report based on the filters applied and the minimum sample size and effect size criteria. The table on the left displays both the conversion rate lift and incremental revenue calculated from the data available from each site. The pie chart on the right captures how the site-level incremental revenue KPIs compare against each other. 

The Content View

The Content view highlights the relative value of including specific widget types in enhanced content. All visualizations in this view are calculated based on only the Treatment Group in the Holdout Experiment.

At the top of the Content tab, the summary cards highlight the value of shopper interactions (clicks) on enhanced content:

The Conversion Rate Lift in the first summary card represents the lift in conversion rates observed in sessions during which the shoppers engaged with enhanced content versus the sessions where shoppers did not engage with the enhanced content present on the URL.

The Time on Page Lift presented in the second summary card represents how much longer the shoppers remained active in their sessions when they interacted with enhanced content versus the shoppers who did not interact with enhanced content at all during their sessions. This is derived by counting the amount of time, in seconds, that each shopper spends actively viewing a URL.

The Views by Widget table highlights the frequency by which each widget type is viewed by shoppers during the sessions that result in conversions. The values are provided by each widget type and each applicable experience type where the widgets were viewed – Inline, Toolbar, and Hotspots. This heatmap can provide guidance on which type of content is most likely to generate a conversion when included in an enhanced content layout.

While this may be perceived as a complex visualization, the purpose of the heatmap is to enable a quick analysis of the assortment of widget types potentially influencing conversions. The higher the number is indexing above 1.0 and the deeper the shade of green associated with the cell, the more frequently the associated widget type was observed across the sessions that resulted in conversions.

The underlying data is a comparison of the frequency by which each specific widget type was viewed in relationship to the frequency by which all the other widget types in the same experience were viewed during the same conversion resulting sessions. The baseline is 1.0, which means that if every widget type available was viewed in every session that resulted in conversions, then every value in this heatmap would be 1.0. As that scenario is not terribly likely, certain widget types index higher and others index lower than the baseline of 1.0. The higher the number, the more often the widget type was viewed across these conversion-resulting sessions compared to the others in the same experiences. 

The last table in the Content view is the Product Summary. The interactions (clicks), carts (conversions), and cart rate (conversion rate) associated with each URL visited by the Treatment Group is broken out further to represent each widget type present during those sessions.