1WorldSync Customer Community

CommunityMod_WM · 4 weeks ago

Please refer to the steps in this article to perform troubleshooting related to the following concerns:

How to confirm the A/B test is available and live
Low data volume: Very few unique shoppers are observed during an A/B test
No data is collected during an A/B test

How to Confirm the A/B Test is Available and Live

An A/B test status can be effectively be considered live when the test status is "Active", the associated product URLs are valid, and Enhanced Content is present on page load.

Step 1: Validate that the A/B test status is currently "Active".

The status of "Active" as a general indication that the Syndigo system has made this test available to be loaded on the targeted retailer websites.

Find the status of the A/B test in the Syndigo platform. This information can be found in the following interfaces:
- On the Enhanced Content Dashboard, the A/B test tile presents a list of tests that have not yet concluded. The status is presented by each.
- Within the Enhanced Content Index, filters may be applied to present all A/B test status by each collection.
- Navigate to the product details page where the A/B test was created.
Validate that the A/B test status is "Active".
- If there is any status other than "Active" displayed, please refer to the information below to proceed with troubleshooting.
  - A/B test status is "Unpublished": Please navigate to the test details page and step forward in the workflow to Review the experiment details. Click the Publish button when ready to submit the content for review by the Syndigo Quality Assurance team.
  - A/B test status is "In Moderation": This status indicates the content is still in the queue to be moderated by the Syndigo Quality Assurance team. There is no further action required at this time. Within the next 24 – 48 hours, the A/B test shall be moderated.
  - A/B test status is "Rejected": This terminated status is applied when the content of the A/B test fails to pass the Syndigo Quality Assurance team's guidelines. A new A/B test must be created after the moderation feedback is addressed. Details about the reasons for rejection may be found in the Collection Status information, which can be accessed through the Enhanced Content section of the Product Details Page.
  - A/B test status is "Canceled": The test was either canceled by a user in the account, or a change in the account configuration rendered the test invalid. If the test was canceled as the result of an account configuration change, please perform the following checks to ensure the validity of the test configuration:
    - Validate that all Enhanced Content websites by the A/B test remain connected within the Syndigo Website Connections interface. Removal of Website Connections may result in automatic cancellation of impacted A/B tests.
    - Reach out to a Syndigo representative to confirm A/B testing feature access remains activated on the account. Feature deactivation results in the automatic cancellation of all A/B tests in the account.
  - A/B test status is "Unconfirmed" or "Needs Review": This status indicates the test is currently paused while awaiting review of the outcome and results. The A/B test met its criteria to conclude, whether that is a scheduled end date or the system automatically determined the outcome of the experiment. An A/B test in this status may be extended to return to "Active" state if desired.
  - A/B test status is "Complete": The A/B test was marked as confirmed and complete by an account user. The test is formally concluded and can not be returned to "Active" status.
- If the A/B test status is "Active", please proceed to the next step.

Step 2: Validate there is at least 1 valid URL targeted by the A/B test.

Navigate to the A/B Test Details Page or find the A/B Testing section within the Product Details Page to view test information.
In the test information displayed, locate the "URLs" counter.
The count of URLs displayed should be 1 or more. If the URLs count is zero:
- Please make note of the Websites targeted by the A/B test.
- From within the Product Details Page interface, navigate to the URLs section of the product.
- Review the listed URLs within "URLs and Features". Click the + URL button on this page to add relevant URLs for this product on the retailer websites targeted by the A/B test.
- If relevant URLs are already present, navigate to the "Enhanced Content" subtab next.
- Assess the "Status" column by each Enhanced Content-enabled URL for this product. "Error" and "Pending" statuses indicate the system has identified a conflict with a duplicative product or URL. Please reach out to a Syndigo representative or submit a Support request to obtain more information about steps to resolve these statuses.

If there is at least one available URL targeted by the A/B test, please proceed to the next step.

Step 3: Navigate to the product pages on the retailer websites to confirm the Enhanced Content loads.

While it is very rare, individual retailers may make modifications to their website technology that have a downstream impact on Syndigo Enhanced Content. The last step in confirming the A/B test is active involves checking that Enhanced Content is loading as expected on the retailer websites.

From the URLs section of the Product Details Page, click on each URL targeted by the A/B test to launch a new browser tab for each.
Validate that each URL to loads successfully.
- If a 404 error is observed, or redirects to another location on the retailer website, this indicates that the retailer has made a modification to the format of the URL since the URL was originally added to the Syndigo platform.
- Obtain the new URL of the product on the retailer website and add this new URL to the product in the Syndigo platform.
If the URL loads successfully, review the web content carefully. Validate that all of the Syndigo Enhanced Content loads as expected, in both the above-the-fold and below-the-fold sections of the page as applicable.
- If specific layouts or widgets are not present on the URL, this may indicate that the retailer is enforcing a restriction on certain content types at this time. Please reach out to Syndigo support to receive more information about the content generation restrictions for each retailer. However, the A/B test may still be considered live in this case.
- If no content loads on the URL, the content may still be moving through the retailer's pipeline. Published content may go live immediately after Syndigo Quality Assurance approval on certain websites, but for others, the retailer enforces a unique moderation pipeline of their own. If after several business days content is still not observed on the URL, please reach out to Syndigo support to receive detailed information about specific retailers' content pipelines.

Optional Step 4: How to view both the A and the B content versions of an A/B test

It is important to note that the A/B testing solution is designed to qualify only net-new shoppers into the experiment, while shoppers returning to the URL after having visited in the past 12 months are automatically disqualified. Also, once qualified into the experiment, information is stored in the local browser application memory to ensure shoppers see the same content version according to their assigned cohort upon each visit to the same URL during the experiment.

Both the A and B contents may be viewed on the same URL by utilizing one of the following workarounds:

Clear the local browser cache and reload the URL. Perform this step as many times as needed until both the A and the B versions are observed.
Launch the browser in Incognito mode and load the URL. Perform this step, launching a new Incognito browser each time, until both the A and B are observed.
Switch devices or have many team members visit the same URL. A seemingly random distribution of browsers will observe content A while the remainder will see content B.

Diagnosis and Recommendations: Gap in configurations. Perform a general check that the product, URLs, A/B test, and retailer websites are all configured for success.

Low Data Volume: Very Few Unique Shoppers Observed During an A/B Test

Ideally, all A/B tests should be configured to collect at least 3,000 unique shoppers (as indicated by the Unique Visits metric available in the experiment data) within 30 – 90 calendar days. One can determine if an A/B test is associated with "low data volume" if it is trending to collect less than 3,000 Unique Visits within 30 – 90 days after the test start date.

Please follow the below steps to troubleshoot the scenario where a low data volume is observed for any A/B Test:

Step 1: Use the Enhanced Conversion Products report to measure the traffic volume.

One of the leading causes of low data volume is low volumes of shopper traffic to the product pages on the retailer websites. The first step in troubleshooting is to check the amount of traffic observed, both across all Enhanced Content products on the targeted retailer websites and for the specific product pages where the A/B test is published. A low volume of traffic explains the low volume of experiment data collected.

Make note of the Unique Visits counts displayed in the A/B Test Details Page.
Navigate to Report Center.
Generate the Enhanced Conversion – Products report, with the following filter options applied:
- Set the date range in the filter panel to 30 days before the beginning of the A/B Test to present day.
- Filter down to one website targeted by the A/B test.
- Utilize the Unique Identifier filter option to filter down to the specific product(s) where the A/B test is published.
Analyze the Visits metric in each section of the report.
Compare the Unique Visits displayed on the A/B Test Details Page and the Visits displayed in the Enhanced Conversion – Products report.
- The number of Unique Visits counted towards the A/B test should be equal to or less than the count of Visits displayed in the report.
- Repeat the above steps in the Enhanced Conversion – Products report by filtering to each individual website targeted by the A/B test to compare the Visits metrics.

If the number of Unique Visits in the A/B test is very low despite there being a notably larger number of Visits reported, this may indicate that many shoppers are returning to view the Enhanced Content on the URLs who were observed prior to the start of the experiment. If shoppers have seen the Enhanced Content before the A/B test began, there is a high likelihood they intend to purchase. Return shoppers are automatically excluded from A/B tests in order to reduce the risk of experiment data becoming skewed by this segment of visitors. This reduces the number of shoppers qualified into the experiment.

An unexpectedly low number of Visits may be observed in the Enhanced Conversion – Products report, as well. In either case, we recommend proceeding forward to the next step in this troubleshooting scenario to validate that the products and URLs are configured in the Syndigo platform to promote optimal coverage of Enhanced Content.

Step 2: Validate that the URLs for the targeted retailer websites are associated with the Product (or added to the Child Product records) and EC is enabled for each.

Navigate to the Product Details Page for the product where the A/B test was created. In the case where many products are linked to inherit Enhanced Content, please navigate to the parent product record to perform the following steps.
In the Enhanced Content section, select the collection in the collection picker that is associated with the A/B Test.
Click the subtab A/B Testing to view the A/B tests associated with the selected collection.
Identify the correct A/B test tile and make note of the retailer sites listed by "Websites:" in this tile. The websites that the A/B test was configured to target shall be listed here.
If there are many child products inheriting the Enhanced Content, please launch each child Product Details Page in a new tab.
Navigate to the URLs section of each Product Details Page – whether viewing a single unlinked product or a parent product with many linked child products.
Count the URLs listed in the “URLs and Features” page and refer to the list of websites targeted by the A/B test.
- Across all product records inheriting Enhanced Content, is there at least 1 URL for each website targeted by the test? Click the + URL button on this page to add relevant URLs for this product on the retailer websites targeted by the A/B test.
- Is Enhanced Content enabled for each relevant URL? If not, toggle the control to the enabled position under the “Enhanced Content” column.

Step 3: Check the Enhanced Content status of the URLs for errors.

Remaining on the URLs section of the Product Details Page(s), click the Enhanced Content subtab next. All Enhanced Content-enabled URLs from the previous page are listed on this page.
Assess the "Status" column by each Enhanced Content-enabled URL for this product. "Error" and "Pending" statuses indicate the system has identified a conflict with a duplicative product or URL. Please reach out to a Syndigo representative or submit a Support request to obtain more information about steps to resolve these statuses.

Step 4: Check that the URLs are loading as expected.

If all relevant URLs were added and enabled for Enhanced Content across all linked products, then:

From the URLs section of the Product Details Page, click on each URL targeted by the A/B test to launch a new browser tab for each.
Validate that each URL to loads successfully.
- If a 404 error is observed, or redirects to another location on the retailer website, this indicates that the retailer has made a modification to the format of the URL since the URL was originally added to the Syndigo platform.
- Obtain the new URL of the product on the retailer website and add this new URL to the product in the Syndigo platform.
If the URL loads successfully, review the web content carefully. Validate that all of the Syndigo Enhanced Content loads as expected, in both the above-the-fold and below-the-fold sections of the page as applicable.
- If specific layouts or widgets are not present on the URL, this may indicate that the retailer is enforcing a restriction on certain content types at this time. Please reach out to Syndigo support to receive more information about the content generation restrictions for each retailer. However, the A/B test may still be considered live in this case.
- If no content loads on the URL, the content may still be moving through the retailer's pipeline. Published content may go live immediately after Syndigo Quality Assurance approval on certain websites, but for others, the retailer enforces a unique moderation pipeline of their own. If after several business days content is still not observed on the URL, please reach out to Syndigo support to receive detailed information about specific retailers' content pipelines.

Diagnosis and Recommendation: There are too few new shoppers visiting the product pages. Utilize industry best practices, such as SEO and SEM, to drive more traffic to the URLs targeted by the A/B test. Secondly, ensure that all URLs are added, enabled for Enhanced Content, and valid.

No Data is Collected During an A/B Test

Whether the experiment is currently active or if it has already concluded, zero Unique Visits may be observed. Email notifications are sent when no data has been observed for seven calendar days or more.

Please follow the below steps to troubleshoot this scenario.

Step 1: Use the Enhanced Conversion Products report to validate general shopper traffic is being observed on the URLs targeted by the A/B test.

The first step in troubleshooting is to check if any amount of traffic observed, both across all Enhanced Content products on the targeted retailer websites and for the specific product pages where the A/B test is published.

Navigate to Report Center.
Generate the Enhanced Conversion – Products report, with the following filter options applied:
- Set the date range in the filter panel to 30 days before the beginning of the A/B Test to present day.
- Filter down to one website targeted by the A/B test.
- Utilize the Unique Identifier filter option to filter down to the specific product(s) where the A/B test is published.
Make note of the Visits displayed across the report.
Repeat the above steps in the Enhanced Conversion – Products report by filtering to each individual website targeted by the A/B test to compare the Visits metrics.
Assess the line graph under “Performance Over Time”, focusing on the Visits line (represented in orange by default).

Attempt to identify whether a steady volume of traffic observed at any point before or after the start of the experiment, or whether the general traffic has consistently been extremely low or perhaps not observed at all (Visits being zero).

If the number of Unique Visits in the A/B test is zero despite there being a notably larger number of Visits reported, this may indicate that many shoppers are returning to view the Enhanced Content on the URLs who were observed prior to the start of the experiment. If shoppers have seen the Enhanced Content before the A/B test began, there is a high likelihood they intend to purchase. Return shoppers are automatically excluded from A/B tests in order to reduce the risk of experiment data becoming skewed by this segment of visitors. This reduces the number of shoppers qualified into the experiment.

An unexpectedly low number of Visits may be observed in the Enhanced Conversion – Products report, as well. In either case, we recommend proceeding forward to the next step in this troubleshooting scenario to validate that the products and URLs are configured in the Syndigo platforme to promote optimal coverage of Enhanced Content.

Step 2: Check for changes to linked Enhanced Content Recipients or for news about retailer organizational changes.

The next step in troubleshooting is to validate that the Syndigo account is still configured to publish Enhanced Content to all retailer websites targeted by the A/B test. At any time, an administrator on the Syndication account may remove (unlink) the Enhanced Content Recipients, which immediately removes all Enhanced Content on the retailer websites associated with them.

Log in to the Syndigo platform.
Navigate to the Connections tab.
Go to the Recipient Connections section, then under the “Recipient Type” dropdown menu, select Enhanced Content.
Analyze the list of Enhanced Content Recipients presented.
- Identify whether key retailers are missing from this list, where they may have been present at an earlier date. This indicates an administrator on the account may have unlinked the Connections associated with the websites at some point after the A/B test started, resulting in the A/B test no longer being live on all of the intended URLs.
- Check with the administrator users on the Syndigo account to understand whether the Connections may be restored. If all desired Connections are restored, the A/B test will automatically be live again on the associated retailer websites. Reference the Help Center article Publish to Enhanced Content Recipient Connections for more information about this step.
It is also recommended that a general online search is performed for the latest news about the specific retailer companies in question. While it is extremely rare, retailers may be acquired by others or close for business with little warning. The downstream impacts of these sudden retailer organizational changes may result in unexpected disruptions to Enhanced Content.

Step 3: Validate that the URLs for the targeted retailer websites are associated with the Product (or added to the Child Product records) and EC is enabled for each.

Navigate to the Product Details Page for the product where the A/B test was created. In the case where many products are linked to inherit Enhanced Content, please navigate to the parent product record to perform the following steps.
In the Enhanced Content section, select the collection in the collection picker that is associated with the A/B Test.
Click the subtab A/B Testing to view the A/B tests associated with the selected collection.
Identify the correct A/B test tile and make note of the retailer sites listed by “Websites:” in this tile. The websites that the A/B test was configured to target shall be listed here.
If there are many child products inheriting the Enhanced Content, please launch each child Product Details Page in a new tab.
Navigate to the URLs section of each Product Details Page – whether viewing a single unlinked product or a parent product with many linked child products.
Count the URLs listed in the “URLs and Features” page and refer to the list of websites targeted by the A/B test.
- Across all product records inheriting Enhanced Content, is there at least 1 URL for each website targeted by the test? Click the + URL button on this page to add relevant URLs for this product on the retailer websites targeted by the A/B test.
- Is Enhanced Content enabled for each relevant URL? If not, toggle the control to the enabled position under the “Enhanced Content” column.

Step 4: Check the Enhanced Content status of the URLs for errors.

Remaining on the URLs section of the Product Details Page(s), click the Enhanced Content subtab next. All Enhanced Content-enabled URLs from the previous page are listed on this page.
Assess the "Status" column by each Enhanced Content-enabled URL for this product. "Error" and "Pending" statuses indicate the system has identified a conflict with a duplicative product or URL. Please reach out to a Syndigo representative or submit a Support request to obtain more information about steps to resolve these statuses.

Step 5: Check that the URLs are loading as expected.