Lift Product Insights Report: In-depth Review of Results

[CommunityMod_WM]

Article Contents:

• Where to Start: Define the Analysis Plan
• An In-Depth Review of Summary & Outcome Statements
• Key Takeaways for Insight Categories & Syndigo’s Recommendations for Next Steps

The Lift Product Insights report tells the story of conversion rate lift - the greater frequency by which visitors who view enhanced content add-to-cart or purchase products as compared to those who do not view enhanced content. The addition of enhanced content to the eCommerce shopper experience is always considered beneficial, but every URL may be associated with its own unique story that is worth a deeper look.

To derive the best value from Lift Product Insights, it is important to analyze each insight category in detail. This is especially the case when the report summary provides a non-standard outcome statement. This article combines Syndigo’s recommendations for how to derive the most value from Lift Product Insights and explains how various results may be interpreted.

Where to Start: Define the Analysis Plan

Before attempting to perform a more in-depth analysis of Lift Product Insights, identify the “right size” of the report.

Every Brand is associated with its own unique volume and diversity of data over time, by product, and by website. It is crucial to assess overall traffic patterns and identify potential opportunities to segment product data further to optimize analysis activities.

To accomplish this, the following two initial steps are recommended:

Step 1: Modify the filters to determine the timeframe and segmentation that provides the right level of granularity for the analysis to be conducted.

Methodically regenerate the report several times with various filter adjustments to gain a general sense of how the story of lift changes with more and less data. The story of lift may change dramatically when comparing different timeframes or sets of products/URLs. This is to be expected, as any filters applied to the report when it is generated increases or decreases the number of data points used to calculate Lift. For this reason, the first recommended step is always to expand or contract the filters applied and regenerate the report to analyze how the story of lift changes with these alterations to the underlying data set. This is especially the case when Lift Product Insights is not presenting outcomes in alignment with expectations.

The ideal timeframe or breakdown may be the one that tells the most positive story (more URLs labeled as best performers).

Step 2: Choose a threshold for the minimum count of visitors required to consider a record “statistically significant”.

Unlike the general Enhanced Conversion – Lift report, which automatically withholds records that do not meet the strictest degree of criteria to be considered statistically significant, Lift Product Insights provides 100% of all product page data where the Holdout Experiment was conducted. For this reason, it is incumbent on analysts to set their own standards for the data that is qualified into the story of lift. The easiest method to accomplish this is by referencing the count of visitors in either the Content Viewers cohort or Non-Viewers cohort. After generating Lift Product Insights, sort in descending order each Insight Category table by the count of visitors in one of these cohorts. Draw an invisible line where there are too few visitors in the cohort, and only analyze the records above that line.

For the highest traffic pages, Syndigo recommends anywhere from a minimum of 3,000 to 5,000 visitors in either cohort.

At the end of this activity, the analysis plan should include a sense of timeframe or range of dates to be reviewed. Ideas around segmentation may also be identified – Perhaps viewing the data from the lens of each individual website or distinct sets of products provides the level of detail needed to extract actionable insights.

An In-Depth Review of Summary & Outcome Statements

When the Lift Product Insights report is generated, the first visualization displayed is the Summary. The Summary generally contains Conversion Rate Lift (displayed as a range) paired with a statement that speaks to what this KPI means in the context of visitor behavior. On occasion, no KPI is displayed, and a different set of messaging is available. Every potential Summary result is explained in greater detail below:

Summary Result #1: A range or single Conversion Rate Lift KPI is displayed, paired with the outcome statement.

This is the standard Summary that will be presented in most cases when there is at least one product page URL that qualified as a best performer. The KPI presented here is not an average or aggregation of any kind. The range represents the highest and lowest Conversion Rate Lift observed across all URLs deemed best performers. If a single value is displayed instead of a range, this may indicate there is only a single URL that qualified as a best performer, or that all best performing URLs have the same calculated KPI value.

Utilize this Summary result to quickly measure the positive impact of the enhanced content program overall – The outcome statements is the story of lift.

This may be the only insight that an analyst seeks to derive from Lift Product Insights, but the next section of this article further explains what important questions a Brand should ask when diving deeper into the individual product page KPIs.

Summary Result #2: “Too few visitors observed enhanced content to calculate an outcome.”

When the Summary does not contain any Conversion Rate Lift and instead presents this messaging around too few visitors, this means that no URLs received enough visits to calculate Conversion Rate Lift.

The minimum amount of data required for any URL to be qualified for one of the other insight categories is:

• At least one URL is associated with enhanced content that was available (published and in an Available status, without any errors in Syndigo).
• At least one Content Viewer and one Non-Viewer must have been observed visiting the URL.
• If one visitor in either cohort converted (add-to-cart or purchased the product), then at least one conversion must also be associated with the other cohort.* For example, if a Content Viewer converted but none of the Non-Viewers converted, then this URL is disqualified. *When zero conversions are associated with both cohorts for any given URL, the URL is not disqualified – rather, it is qualified for the Review Recommended insight category.

Scroll down in the report to the Unqualified Products table – If it is empty, this indicates there were no visits to any pages where enhanced content was present.

Utilize the other Enhanced Content Reports to first validate that enhanced content was successfully published (Enhanced Coverage). If Last Visit Dates in the Coverage report indicate that content is live, then generate Enhanced Conversion – Products or Widget Insights to ensure traffic is observed. If many visits were observed during the same time period, it is important to note that the Holdout Experiment may not be running on all retailer websites at all times. Gaps in coverage should be addressed by publishing enhanced content until it is in an approved status and adding enhanced content-enabled URLs where possible.

Summary Result #3: “Products and pages may not be performing to expectations: We recommend performing an additional analysis to identify potential opportunities to optimize enhanced content to better promote conversions.”

This result is presented when no URLs are categorized as having a measurable story of lift. Most if not all URLs may be sorted into the “Review Recommended” and/or “Inconclusive Results” categories. There may also be many URLs in the “Unqualified Products” table, as well. The “Best Performers” table contains no URLs.

This Summary result is not to be interpreted as enhanced content negatively impacting the likelihood of conversions, nor does it indicate inaccuracy in data collection. As this article goes on to explain in-depth, there are many factors that drive the add-to-cart or purchase behavior of visitors to retailer websites.

As mentioned previously, when this Summary is observed, the first step should be to regenerate the Lift Product Insights report with an expanded set of filters applied – Increase the timeframe, remove any site or product-specific filters to see how the story of lift may merge with more data points. If Lift Product Insights was generated with a very wide set of filters initially, then the recommendation is to contract these filters to decrease the number of data points and observe the differences in the visitor behavior from a more limited lens.

For more information about how to resolve this messaging, please proceed to the next section of this article.

Key Takeaways for Insight Categories & Syndigo’s Recommendations for Next Steps

Insight Category #1: Best Performing Products & Pages

Conversion Rate Lift is only available for the URLs that are sorted into the category of Best Performers, the first table in the Lift Product Insights report. Altogether, the full range of Conversion Rate Lift values across all best performing URLs represent the story of lift for the enhanced content program overall.

Beyond this overarching positive story, however, there is a key question to answer:

How does the enhanced content for these best performing product pages differ from the others? Identify the trends and commonalities amongst these as compared to URLs found in the “Review Recommended” or “Inconclusive Results” categories. This may inspire opportunities to optimize enhanced content - Consider A/B testing to vet out hypotheses that emerge from this analysis.

Insight Category #2: Review Recommended

Syndigo recommends reviewing the products and URLs listed in this second category available in the Lift Product Insights report. Records are categorized here when no visitors added the product to cart or purchased, or the Non-Viewers are associated with a greater frequency of conversions as compared to the Content Viewers.

The goal of highlighting these records for review is to drive action – Syndigo informs Brand where there may be opportunities to invest in digital and traditional marketing techniques, optimize creative assets and design, and coverage across the addressable market to promote product sales. Success may be measured as executing an improvement plan and observing the URL later be classified among the Best Performing Products & Pages.

It is crucial to remember that this result is not to be interpreted as enhanced content negatively impacting the likelihood of conversions, nor does it indicate inaccuracy in data collection. The data, as viewed only from the context of Lift Product Insights, may not speak to the success of a product overall.

Utilize the data provided throughout Syndigo’s Report Center in conjunction with Brand-internal information to delve deeper into these records and investigate the questions:

What are the forces that may have influenced conversions overall? Why didn’t the Content Viewers add-to-cart or purchase at a higher rate than the Non-Viewers?

There are many factors that drive the add-to-cart or purchase behavior of visitors. On a product-by-product basis, consider all perspectives to develop a hypothesis about what may have occurred and build a plan to refine the analysis or apply corrective actions.

Reviewing these records may help develop a better understanding of shopper segmentation and customer profiles.

If a higher conversion rate is frequently associated with the Non-Viewers for any specific products or websites, identify possible correlations and commonalities between them. The data may shed light on how customer behavior may differ across websites or when shopping for a certain product type.

Ideally, the count of Content Viewers should be much greater than that of Non-Viewers. If Non-Viewers outnumber Content Viewers by a large margin, this may mean enhanced content is not readily accessible to the average site visitor.

Validate that the presentation of the enhanced content is designed to promote product purchases. Follow the journey of a prospective shopper to the product details page. Assess how enhanced content is rendered on the page. Is enhanced content available immediately upon loading the product page, or does it require the visitor to scroll down? Is the content hidden or obscured behind a click? The real estate dedicated to Brand-provided content varies from retailer to retailer and greatly impacts the likelihood of enhanced content views. There may be opportunities to expand coverage to sites that will likely drive the count of Content Viewers higher, and as more content-viewing visitors are qualified into the Holdout Experiment, the story of lift may emerge.

If the count of Content Viewers is equivalent to or greater than the number of Non-Viewers, another story about behavior emerges – Identify the reasons why conversions did not occur when enhanced content was viewed. This segment of visitors are navigating to the product details page during their shopping journey and actively seek out key benefits, detailed imagery, or perhaps technical specifications.

Visitors who spend the greatest amount of time and effort to scroll through all product information available may simply not be ready to purchase. This pattern of behavior indicates they are still in the research or exploratory phase of their shopping journey or waiting for the right time to make the purchase. In any case, there is a chance they will return on a different day and add-to-cart. Certain products and retailers may also be associated with a higher percentage of visitors who are conducting research online in advance of shopping at a brick-and-mortar location.

The Non-Viewers who are adding to cart or purchasing a product may consist of visitors returning to a product details page at a future date, or they represent a different cohort of visitors who aren’t as motivated to consume the rich media available and rather base their decision to purchase on other factors.

Another possibility is that Content Viewers are ending their visits without adding to cart due to not finding the critical information they seek or determining that the product doesn’t align with their needs. Studies show that 50% of shoppers abandon a purchase when they cannot find sufficient information about it online.

Validate that the enhanced content itself contains the compelling imagery and critical information required to persuade a visitor of a known shopper profile to buy the product. Is the rich media designed to promote purchases of this specific product? Or does the content contain information that may be used by visitors to disqualify the product or redirect them elsewhere? Rely on expert judgement and marketing studies to determine if the assets and layouts utilized are providing the information a visitor needs to feel confident about purchasing. Utilize the other Enhanced Content Reports such as Widget Insights to assess the viewability and interaction rates of visitors on specific content types. Then perform A/B testing to incrementally optimize the effectiveness of enhanced content.

If conversions are waning or not meeting expectations overall, it is crucial to perform a competitive analysis. Enhanced content-viewing visitors are likely to be comparing across competitive products at the same time. If competitors are running promotions, backed by viral marketing, or have published particularly attractive creative assets, this may drive down owned-product conversions across the board. Pricing, stock status, and other market forces may be influencing further. Syndigo’s Enhanced Benchmarks report provides valuable insights about how owned products compare to competitive products in the same categories and on the same retailer websites, granting the ability to see where in the timeline product performance may have been impacted by competitors’ marketing activities.

Insight Category #3: Unqualified Products

The URLs in this category did not receive enough visits to calculate Conversion Rate Lift.
The minimum amount of data required for any URL to be qualified for one of the other insight categories is:

• At least one Content Viewer and one Non-Viewer must have been observed visiting the URL.
• If one visitor in either cohort converted (add-to-cart or purchased the product), then at least one conversion must also be associated with the other cohort.* For example, if a Content Viewer converted but none of the Non-Viewers converted, then this URL is disqualified.
  • *When zero conversions are associated with both cohorts for any given URL, the URL is not disqualified – rather, it is qualified for the Review Recommended insight category.

When investigating these unqualified records, the first recommended step is to expand the timeframe and other filters applied in the Filter Panel of Lift Product Insights. As this messaging is caused by low volumes of data, expansion of the filters should bring in more data points and increase the likelihood these URLs will qualify for inclusion in another category.

If expansion of the filters does not result in a different result, please continue to the next recommended step: Analyze each record in the Unqualified Products insight category.

1. Navigate to the Enhanced Conversions - Products report and filter down to the specific product pages of interest, for the same time period.
2. Validate that visits were observed for these URLs. The count of visits in Enhanced Conversion – Products should be similar to the total count of Content Viewers and Non-Viewers in Lift Product Insights, though it may not always be identical for a number of reasons.

If very few visits are captured in both reports, this indicates a need to increase the volume of traffic through effective digital marketing techniques such as SEO. There may be opportunities to collaborate with a retailer to promote products on other locations of the website, or optimize search terms. Syndigo’s Digital Shelf Analytics provide more crucial data points to supplement a plan to drive more traffic to product pages.

Insight Category #4: Inconclusive Results

In the last section of the Lift Product Insights report, Syndigo categorizes together the product page records for which the outcome of the Holdout Experiment is inconclusive. These are the cases where the impact of enhanced content on conversion rates remains incalculable or unclear due to an inconsistency in visitor behavior found in the collected analytics. The presence of URLs in this table should not be interpreted as enhanced content negatively impacting the likelihood of conversions, nor does it indicate inaccuracy in data collection.

URLs are categorized into the Inconclusive Results section in the following cases:

• The majority of visitors were adding the product to carts before viewing the enhanced content, or
• Visitors spent too little time active in their sessions to meaningfully consume and engage with enhanced content.

These behaviors prevent a definitive conclusion about whether the enhanced content influenced their purchase decisions.

Several factors contribute to this pattern. Many visitors arrive at the product details page with a strong pre-established intent to purchase, making them less reliant on the enhanced content to make a decision. Brand loyalists, for instance, may already trust the product and add it to their cart without further evaluation. Additionally, visitors driven by promotions or effective marketing campaigns may land on the page ready to buy, treating it as a transactional step rather than a point of product discovery. In other cases, visitors may be returning customers who have previously researched the product and do not need to review the enhanced content again. This leads to insufficient exposure to the enhanced content for the Holdout Experiment to yield conclusive results.

The first step in conducting a deep-dive analysis of these results is to expand and contract the filters applied to the Lift Product Insights report, to assess how the story of lift may emerge when adding more data points or breaking up the data into more refined and strategically sized sets. If a URL remains fixed in the Inconclusive Results category throughout multiple report generations covering different date ranges and timeframes, perform further investigation into the external forces that are influencing visitor behavior.

While the story of lift cannot yet be derived for these URLs, the results of the Holdout Experiment may tell a different but equally meaningful story: The size of the opportunity to influence visitors using enhanced content for these products and pages. This varies drastically based on the Brand, product type, other considerations. A very popular product with many loyal repeat customers or visits driven by viral marketing is generally going to be associated with a lower or incalculable Conversion Rate Lift – in other words, a smaller-sized opportunity to influence visitors using enhanced content. This points to the value of performing a more holistic investigation with a broader view of all market forces that come together to determine the overall success of a product.