Summary

Starting on Monday, October 11th, 2021, at approximately 10:00PM ET, Streams Endpoints experienced a delay in processing updates to data and endpoint configuration. As a result of this delay, the data returned when fetching from these endpoints via Streams API did not return updates made during this time period. Some Answers experiences that have been migrated to use Streams will also have been affected. This was mitigated on Tuesday, October 12th, at 10:47AM ET (which removed the delay effect, but increased request latency for consumers in some regions), and fully resolved at 11:25AM ET.

Live API v2 requests were unaffected, and no data was lost.

Root Cause

An error in configuration when updating a backend system delivering updates from our source of truth database to cloud regions resulted in Streams services in the Eastern US to stop receiving updates. The issue was mitigated by failing over to our European cloud regions, and resolved by correcting the configuration.

Remediation

We will be revising our monitoring to allow us to detect delayed updates for Streams more quickly, and exploring automated failover to apply the mitigation as soon as a problem is detected.

Posted Oct 18, 2021 - 15:14 EDT

Resolved

This incident has been resolved.

Posted Oct 12, 2021 - 12:15 EDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Oct 12, 2021 - 11:26 EDT

Identified

The issue has been identified and a fix is being implemented.

Posted Oct 12, 2021 - 10:57 EDT

Update

A mitigation has been put in place to ensure that updates are reflected in a timely manner. While this mitigation is in place, consumers in some regions may see increased query latency from Streams endpoints.

Posted Oct 12, 2021 - 10:54 EDT

Investigating

We are currently investigating a delay in processing of updates to data and new endpoints in the Streams APIs.

Posted Oct 12, 2021 - 10:41 EDT

This incident affected: Content (Content API).