Elevated Error Rates for Log Queries and Monitors

Incident Report for Datadog EU

Resolved

Fix has been fully rolled out.
Posted 2 years ago. Oct 04, 2023 - 10:39 EDT

Update

The fix rollout is currently ongoing. Once completed we will confirm resolution.
Posted 2 years ago. Oct 04, 2023 - 09:54 EDT

Update

We have successfully tested a fix for this issue and are currently deploying it to resolve this incident.
Posted 2 years ago. Oct 04, 2023 - 09:05 EDT

Update

We're still working on a fix for historical data impacted by this incident.
Posted 2 years ago. Oct 04, 2023 - 08:18 EDT

Update

We're still working on a fix for historical data impacted by this incident.
Posted 2 years ago. Oct 04, 2023 - 07:41 EDT

Update

We're still working on a fix for historical data impacted by this incident.
Posted 2 years ago. Oct 04, 2023 - 07:06 EDT

Update

We're still working on a fix for historical data impacted by this incident.
Posted 2 years ago. Oct 04, 2023 - 06:26 EDT

Update

We're still working on a fix for historical data impacted by this incident.
Posted 2 years ago. Oct 04, 2023 - 05:20 EDT

Monitoring

We have deployed a fix and we are monitoring the results. We will provide another update once the issue is fully resolved.

At this time, newly ingested data is properly queryable, and monitors targeting Logs sent from 2023-10-03 20:40 UTC onwards are valid.

Queries targeting logs between 2023-10-02 11:40 UTC and 2023-10-03 20:40 UTC may return erroneous data. We are evaluating a fix that will restore query correctness for this time-window.
Posted 2 years ago. Oct 03, 2023 - 16:48 EDT

Identified

We have identified the underlying issue and are working on a fix.
Posted 2 years ago. Oct 03, 2023 - 15:32 EDT

Update

We are continuing to investigate these issues, and will provide an update as soon as possible.
Posted 2 years ago. Oct 03, 2023 - 14:50 EDT

Investigating

We are actively investigating issues with Log Queries returning unexpected results.
As a result of this issue, some users may experience issues querying logs on the web application or API, and with Logs based Monitors.
Posted 2 years ago. Oct 03, 2023 - 13:32 EDT
This incident affected: Log Management and Monitors.