1WorldSync Critical Incident - March 30, 2021

[CommunityJedi22]

Last evening, the 1WorldSync Application team identified an issue with the slowness of message transaction processing in Item Management.  A handful of suppliers sent some very large message transactions that slowed our systems and created a backlog.  The Application team was monitoring the incident through the evening.

This morning, we are looking to determine when the backlog will be resolved.  We also reached out to the account manager for the large recipient that sent 100k messages yesterday to ascertain why and offer any assistance we can. 

The Service Desk updated the IVR to make any customers that contact us aware of the backlog. 

[CommunityJedi22]

Team, we received an update from our technology teams. We are holding all IM and M2M messages until the backlog is resolved to allow time for it to resolve more quickly. We estimate the backlog to resolve in the next 2.5 hours.  

All messages that came in since 9 am this morning are being held until the above backlog is cleared

[CommunityJedi22]

Team, just to wrap up the issue, the backlog was resolved yesterday and we appreciate the efforts of our technology teams who worked to close this as expeditiously as possible.  We are working with the large supplier who triggered hundreds of thousands of messages that slowed the system processing and created the backlog.