This article explains the concept of statistical significance in the context of Syndigo’s Enhanced Content A/B testing feature.
Statistical significance refers to the likelihood that the difference in results between the two Enhanced Content versions being tested (A and B) is not due to chance. In other words, it is a measure of how confident we can be that the results we see will be the same if the experiment is repeated over and over again. This determination is made by calculating a confidence level based on the results of each experiment. The greater the statistical significance, the higher the confidence level that the findings reflect reality. If the confidence level is low, this means any difference observed between A and B is most likely due only to random chance.
Syndigo’s Enhanced Content A/B Testing feature requires a confidence level of 95% or greater to be calculated for each experiment to determine statistical significance has been reached and make a declaration about whether there is a winning content version. The algorithm consumes the conversion rate data we track for both versions, A and B, over the course of an experiment to calculate the confidence level on your behalf.
To further understand how statistical significance is utilized in the context of A/B testing Enhanced Content, consider the following list of results that may be presented at the conclusion of any experiment:
Scenario 1: 95% or greater confidence level that B results in more conversions than A
Shown in the interface as: “There is a 95% probability that Content B is better”, where the percentage displayed ranges from 95% - 100%.
A confidence level of 95% or greater states that if the experiment is repeated over and over again and expanded out to the general population of visitors, there is a 95%+ probability that the results will be the same as observed in this experiment. Essentially, this means it is extremely likely that the higher conversion rates associated with Enhanced Content version B can be attributed to the variable – In the case of an “Add a Widget” A/B Test, the higher conversion rate is directly attributable to the video asset added in Content B.
Scenario 2: Relatively low confidence that B and A will result in different outcomes
Shown in the interface as: “There is low confidence that these collections are different”.
A percentage is not displayed in the declaration statement, as the exact confidence level calculated by the algorithm is not a meaningful measure of what has occurred. It can be assumed that the confidence level is not entirely unsubstantial but it clearly does not meet the criteria for declaring statistical significance. The most common reason for a low confidence level is that more data is required. Either the experiment did not run for a long enough period of time or there were not enough visitors to the product pages.
Scenario 3: No confidence that B is different than A
Shown in the interface as: “There is a 95% probability that these collections will not produce different results”, where the percentage displayed ranges from 95% - 100%.
This result indicates that experiment was found to be statistically significant, but there is a high confidence level that there is no difference in outcomes between Content A or B. The data indicates that the experiment variable does not impact the conversion rate in any discernible way – In an “Add a Widget” A/B test, for instance, adding a single text or image asset to the In-Line layout of Content B may not be noticed by most visitors. Often times, when the difference in conversions between Contents A and B is small but statistically significant, it is due to a very large sample size; in a sample of a smaller size, the differences would not be enough to be statistically significant.
In closing, regardless of the confidence level calculated by Syndigo’s algorithm, keep in mind that we can never be truly 100% certain that the variable is the only reason a difference was observed between Contents A and B, since eCommerce experiments never happen in isolation. There are countless external sources that influence results such as competitor landscape, trends, stock availability, and technical difficulties on the retailer website. But we are estimating the probability that we are wrong – and if the probability of being wrong is small, then we can say that our observation is a statistically significant finding.
