Incident Summary
From approximately 23:45 UTC on Tuesday, January 5, 2021 to 17:30 UTC on Thursday, January 14, 2021, some Xandr users were unable to access the Grapeshot Keywords app due to an authentication issue.
Scope of Impact
During the incident window, customers trying to use the Grapeshot Keywords app were shown an error message that cookies need to be enabled in the Apps Marketplace UI. This was caused by the underlying authentication failures.
Timeline (UTC)
2021-01-05 23:45: incident started after the release of an upgrade to our authentication API service
2021-01-14 11:00: issue was reported and escalated by customers
2021-01-14 12:03: issue was escalated to our engineers after initial checks
2021-01-14 13:35: customers reported that switching API authentication calls to HTTPS fixed the issue for existing users, but not new ones
2021-01-14 17:30: incident resolved - we rolled back the release that caused the problem and clients confirmed this resolved the issue for all users
Cause Analysis
The incident was caused by a release to our authentication API service. This caused failures for third-party app marketplace authentication. Regular UI/API authentications were unaffected.
Resolution Steps
Our engineers resolved the issue by rolling back the problematic release. Separately we are working to fix the root issue before releasing that update again.
Next Steps
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.