Introduction to Enhanced Content A/B Testing

[CommunityMod_WM]

This article provides general information about Enhanced Content A/B Testing and definitions of the key terms you will encounter in related articles.

 

Article Contents:

• Introduction to Enhanced Content A/B Testing
• Key Terms and Definitions

Introduction to Enhanced Content A/B Testing

A/B testing is a scientific method used in the eCommerce industry to compare two versions of a product page on a retailer website to see which one performs better. The goal is to improve the user experience and increase conversions (add-to-carts or sales). This is done by running an experiment to randomly show different versions of Enhanced Content on the same product page to different visitors, and then analyzing the data to see which version resulted in more conversions. Enhanced Content A/B testing can be used to test a wide range of elements, including the layout, widget types, imagery, and text.

The process of conducting A/B Test experiments in the context of Syndigo Enhanced Content is as follows:

1. Question: The decision makers at a company first identify the problem to be solved. Listed below are some of the key questions that Syndigo’s A/B Testing solution will help answer:
   • Is this creative asset type worth investing in?
   • Which is the better asset for this widget?
   • Which layout and/or set of assets performs better?
2. Background: The context of the experiment is shaped by all the information that is already available to the company’s decision makers, such as customer demographics and the competitive landscape.
3. Hypothesis: The company makes a prediction about the outcomes or impact of a change to their Enhanced Content based on limited evidence as a starting point for further investigation. The goal of an A/B Test is to prove or disprove the hypothesis.
4. Identify the Variable: In this crucial decision-making step, the company identifies the single item to be tested in the experiment - The element that makes one version of the Enhanced Content different than the other. This step concludes with the preparation of the asset(s) and the configuration of the A/B Test in the Syndigo platform.
5. Run the Experiment: Once the Enhanced Content A/B test is made available on retailer websites, the system tracks a “control” visitor cohort that will see one version of the Enhanced Content (Content A) and a “treatment” visitor cohort that will see the other version of the Enhanced Content (Content B). All the same empirical data is collected for both cohorts. Syndigo’s algorithm processes this data continuously for the duration of the experiment to calculate an important component of A/B testing known as statistical significance.
6. Results: The output of the experiment is the declaration about the algorithm’s calculated confidence level that one version performs better than the other, based on conversion rate. Other valuable metrics Syndigo tracks for all Enhanced Content today are also available at the conclusion of the experiment, enabling additional analysis to compare how they differ between Content A versus Content B. All results are purely data-driven.
7. Conclusion: After the company reviews how the results support or reject the experiment’s hypothesis, they will provide confirmation of the Enhanced Content version to be published and confirm the experiment is complete.

Let’s apply the above process to tell an A/B testing story about a hypothetical Enhanced Content customer.

Question: A company specializing in outdoor gear and equipment, Outdoor Adventures, Inc., recently introduced a new camping tent and wants to know if the product demonstration video they created will increase the likelihood of customers making a purchase. Should Outdoor Adventures invest in producing a series of product demonstration videos for more products in this product line?

Background: Outdoor Adventures is getting ready to propose their marketing budget for the upcoming fiscal year and they are being challenged by executive leadership to include more data to support their eCommerce strategy. Analysis of the Enhanced Benchmarks report reveals that more than half of the competitors’ outdoor gear and equipment products deliver higher Enhanced Content interaction rates and conversion rates than Outdoor Adventures’ products in the same category. Combining customer research with an analysis of the Enhanced Content on the most popular competitive product pages, the marketing team feels strongly that the product demonstration videos are the key to higher visitor engagement and conversion. Videos are among the most expensive creative assets to produce, which is why they have been hesitant to invest in them.

Hypothesis: The marketing team at Outdoor Adventures believes that by providing customers with a clear and detailed video demonstration of the tent's features and benefits, customers will have a better understanding of the product and be more likely to make a purchase.

Identify the Variable: The variable in this case is the video asset. The agency has delivered the demonstration video of the new tent product in the required formats and the asset is ready to upload to Syndigo. The plan is to run an A/B test where the control group of visitors will see Content A - a product description, image gallery, and feature set widgets in the In-Line layout. The treatment group will see Content B, which has identical widgets as Content A, except the product demonstration video appears at the top of the In-Line layout. This way, Outdoor Adventures will be able to measure the exact impact to revenue that can be attributed fully to the presence of the video widget.

Running the Experiment: The A/B test is configured to run on several of the biggest retailer websites for Outdoor Adventures. After analyzing the Enhanced Content reports in Syndigo, the team has determined that two months should be enough time to collect at least 1,000 unique visits on all the URLs where the A/B test will be published. The test runs until the end date specified, at which point the results are made available to review.

Results: At the end of the two-month test period, Syndigo’s algorithm declares Content B to be the winner of the A/B test. The statistical confidence level is calculated to be 95%, which indicates:

The results of the A/B test are statistically significant. In other words, the experiment was configured effectively to collect an optimal amount of data.

There is a 95% probability that Content B will always deliver a higher conversion rate than Content A, even if the experiment is conducted with a larger sample population.

Conclusion: With the evidence provided by this experiment, Outdoor Adventures concludes that the product demonstration video has a quantifiable, positive impact on customer purchase behavior. They design and budget for a plan to create similar product demonstration videos for all new product launches planned in the upcoming fiscal year.

Key Terms and Definitions

A/B Testing: A method of comparing two versions of the Enhanced Content on a product page to determine which one performs better, by randomly exposing them to a portion of users.

Control Group: A group of users who receive the original version of the Enhanced Content being tested (Content A or Version A).

Conversion Rate: Visitor traffic that gets monetized is known as the conversion rate. Only some retailer websites offer us the data to calculate true conversion - for others, Syndigo tracks add-to-cart actions.

Metrics: A specific measurement of user behavior or experience, such as clicks, conversions, or time on site; used to measure the performance of Enhanced Content on retailer websites.

Treatment Group: A group of users who receive a variation of Enhanced Content being tested (Content B or Version B).

Sample, or Sample Size: The number of unique visitors who are included in the A/B test. Syndigo’s Enhanced Content solution excludes visitors from the sample if the system detects that they saw Enhanced Content on the URLs up to one year prior to start of the test. Also, the visitors who are qualified for inclusion in the experiment’s sample population are always presented the same version of the Enhanced Content during the course of the A/B test. These mechanisms decrease the risk of experiment results being skewed by customers who were potentially exposed to multiple versions of Enhanced Content for the same product.

Statistical Confidence Level: This is a measurement of the system’s confidence that the experiment results will be the same if the experiment is expanded to encompass a larger sample size. For example, a 95% confidence level that Content B is the winner means that if the experiment were to run 100 additional times, the outcome would be that Content B will win 95 out of 100 times. Syndigo targets a 95% confidence level to make a declaration about whether there is a winning version of Enhanced Content or that there is no measurable difference between Contents A and B to be ascertained.

Statistical Significance: A statistical measurement that determines if the results of an A/B test are likely due to chance or if they are a true reflection of the performance of the two versions being tested.