Troubleshooting the Enhanced Conversion Lift Report

[CommunityMod_WM]

This article contains guidance on how to interpret and address missing records and “More data is required” messaging.

Overview of Syndigo’s Criteria for Inclusion in the Lift Report and Various Potential Scenarios

The objective of the Enhanced Conversion – Lift report is to present only the insights for which there is a high degree of confidence that the observed results reflect reality accurately. To prevent the presentation of inaccurate findings from the Holdout Experiment, Syndigo enforces a series of criteria to qualify data for inclusion in the report – at all levels of aggregation represented.

In the case where no data qualifies for inclusion in any visualization of the report, the following messaging is presented in each visualization impacted: “More data is required to calculate the metrics in this report. We recommend expanding the timeframe, sites, and product filter options applied. Note: It generally takes 30 days for newly published enhanced content data to appear.” This message is being presented because, after the timeframe and optional filters are applied, the resulting data does not meet all the criteria set by Syndigo to promote statistical significance in the results of the Holdout Experiment.

The same applies to the perception of “missing” records in the Retailer Summary and Product Summary tables in the Lift view - The data at each level of aggregation must be inclusive of a minimum sample size, which is generally achieved when at least 2 - 3 thousand unique shoppers are qualified into the Holdout Experiment. A minimum effect size must also be observed. As a result, the following rules are enforced during report generation.

Based on the timeframe specified in the filter panel, in addition to any optional filters applied:

• When all the data available does not meet the inclusion criteria, the overall KPIs displayed in the summary cards will not be calculated.
• When all the data for enhanced content URLs associated with a single site does not meet the inclusion criteria, the KPIs will not be calculated at the site level. The site will be excluded from the Retailer Summary visualizations.
• When all of data for a single enhanced content URL does not meet the inclusion criteria, the KPIs will not be calculated for the product + page combination. The product + page record will be excluded from the Product Summary table.

A lack of available data is not always an indication that there is an issue with data collection or processing. To better understand why the KPIs could not be calculated, please review the following example and Syndigo’s recommendations to analyze the data further.

Recommended Troubleshooting Steps:

Let’s consider the following example:

There are 100 URLs where enhanced content is published. The Enhanced Conversion – Lift report is generated for the month of January. Optional filters are applied, so the remaining data consists of 3 out of 5 retailer websites and 50 product + URL records where enhanced content was available. Outcome upon report generation: No data was qualified for any visualization on the Lift view or in the summary cards of the Content view. The “more data required” messaging is displayed above the summary tiles on the Lift view and potentially across other visualizations, as well.

Root Cause: After the date range and any optional filters are applied, all remaining Holdout Experiment data available does not meet Syndigo’s criteria for inclusion in the report.

• During the month of January, the traffic across all 50 remaining URLs after the optional filters are applied is less than 3,000 total unique shoppers. Since only net-new unique shoppers are qualified into the Holdout Experiment, the actual number of qualified shoppers is likely much less than the overall traffic that visited these 50 URLs.
• During the month of January, the difference in conversion rate between the Treatment Group and Holdout Group was not calculable or negligible.

How to interpret this outcome: The summary card KPIs were not calculated because there is no overall story of conversion rate lift to tell with the given filters applied due to a lack of qualified data.

Further analysis recommended:

• Confirm the volume of traffic is substantial enough to be inclusive of at least two to three thousand unique, new shoppers. Generate the Enhanced Conversions – Products report with the same filters applied. The visits metric should be substantially higher than 2,000, as only new shoppers are qualified into the Holdout Experiment and the overall traffic numbers may be reflecting a large percentage of returning shoppers.

If a low volume of traffic is observed, manipulate the applied filters in the Enhanced Conversions - Products report until a healthy number of shoppers are observed. This may include adding more sites, products, and URLs and/or expanding the timeframe. Return to the Lift report and apply the same filters to bring more data in to the calculation of the KPIs. Additional adjustments to the filters may be required to capture as much qualified Holdout Experiment data as possible.

A low volume of traffic to the desired URLs may indicate that the URLs do not actually have enhanced content live, or it may point to the need to perform an analysis of digital shelf analytics and refine SEO and other marketing efforts to drive traffic to the URLs.

• Utilize the Enhanced Coverage report to understand when, if ever, enhanced content was most recently viewed on the URLs. Investigate the URL and enhanced content statuses where the Coverage report does not reflect recent views of the content.
• Syndigo’s digital shelf analytics solutions includes reporting to help identify where product stock status, share of search, and other factors may be influencing traffic to the URLs.

If the traffic indicates there were enough unique shoppers, then the minimum effect size criteria is most likely not being met. Perform the following steps to gain a deeper understanding of shopper conversion behavior for these URLs.

• Gauge whether enough conversions were observed for these URLs. Generate the Enhanced Conversions – Products report with the same filters applied. If the conversions metric observed is zero, this is a clear indication that no Holdout Experiment shoppers added to cart or ordered the products. Otherwise, when the visits reflected are greater than two to three thousand, a healthy conversion rate is greater than 2%.
  • If no conversions are reported, this may indicate an inability to purchase due to the product being out of stock, or a potentially suboptimal user experience on a website due to a retailer-driven design change. Navigate to each of the URLs and confirm that enhanced content is loading as expected, and that the add to cart button is available to click. Contact Syndigo support for further information if the enhanced content is not displayed.
  • If some conversions are observed but the overall conversion rate is lower than expected, a deeper analysis is required targeting a wide array of influencing factors. The overall efficacy of the organization’s marketing initiatives should be investigated and measured, to better identify if the eCommerce strategy can be further optimized with more effective assets and messaging.

In either case, it is worthwhile to manipulate the applied filters in the Enhanced Conversions - Products report until a healthy number of conversions are observed. This may include adding more sites, products, and URLs and/or expanding the timeframe. Return to the Lift report and apply the same filters to bring more data in to the calculation of the KPIs. Additional adjustments to the filters may be required to capture as much qualified Holdout Experiment data as possible.

• If a healthy number of conversions are associated with the URLs, but the KPIs are not calculated at one or more levels of aggregation in the Lift report, this may indicate that a conversion has not yet been observed in one or both Groups targeted by the Holdout Experiment. It may also mean that there is no story of conversion rate lift to tell for this set of products and URLs, as the enhanced content does not appear to be influencing the Treatment Shoppers in a consistent and reproduceable trend. In either case, generate the Enhanced Conversion – Lift report to introduce more data into the view, followed by subsequent generations of the report to drill down into smaller sets of data. This is achieved by expanding and contracting the filters applied to bring more and less data into the calculations.
  • By expanding the timeframe, site, and product filters, more data is being added to the overall data set available to meet the minimum requirements for inclusion in the report. Returning to the example, expanding the January timeframe to encompass January through March should effectively triple the amount of data that was originally not meeting the inclusion criteria. Begin with at least 30 days of data (if the traffic reporting indicates enough visits were observed), and expand to 90 days, 180 days, and one year to observe how KPI calculations change as more data is introduced. Also, experiment with expanding the number of sites from 3 to 5 and the number of products to capture all 100 URLs as opposed to 50.
    • Note: On occasion, expanding the filters results in lower Conversion Rate Lift and Incremental Revenue than a prior view of a smaller data set, or even finding that the KPIs can no longer be calculated. The KPIs may be withheld across any and all visualizations. See the section Understanding low or zero Conversion Rate Lift for more information about this scenario.
  • While recording the results of the Lift report with larger data sets, make note of the sites in the Retailer Summary and the individual product + page records in the Product Summary visualizations that qualify for KPI calculations. Regenerate the Lift report filtered down to each individual site, followed by each individual product. By performing these “drill-downs” into more specific data sets, highly informative insights can be derived about the sites that are driving the most attribution to enhanced content and the products for which enhanced content is most effectively influencing conversions. The information obtained from these analyses should drive enhanced content design and coverage optimization efforts.

Understanding Low or Zero Conversion Rate Lift

After confirming that traffic and conversion data is healthy across the population of shoppers, the scenario may be encountered where Conversion Rate Lift KPIs are lower than expected or withheld entirely due to being a true zero value. This may impact one or many visualizations in the Enhanced Conversion – Lift report.

This result may be interpreted as follows: The Holdout Experiment did not observe a consistent and reproduceable difference in the conversion rate of shoppers from whom enhanced content was withheld and shoppers who had the opportunity to view enhanced content during their visits. Identify opportunities to improve the likelihood that enhanced content will have a positive impact on shopper behavior.

This is not indicative of invalid data. Rather, this result should serve as a starting point to begin investigating the effectiveness of the enhanced content itself. This may indicate that the enhanced content is not as compelling as intended, or in the least desirable case, the enhanced content potentially contains visuals and information that negatively impacts conversion rates.

Follow the guidance in the article Measuring Enhanced Content Performance, which covers best practices for utilizing the Enhanced Content Reporting Suite to effectively measure historical performance, discover potential gaps in coverage across retailer sites, and understand how to improve content design.