Read Errors on loki-prod3 Cluster

Incident Report for Grafana Cloud

Resolved

This incident has been resolved.

Posted Apr 04, 2023 - 13:49 UTC

Monitoring

As of 1545 UTC, we have observed marked improvement to the loki-prod3 cluster and are currently monitoring for reoccurrence. At this time, customers should no longer experience 504 read errors when querying Loki.

We will continue to monitor and provide updates.

Posted Mar 31, 2023 - 15:57 UTC

Investigating

As of 1400 UTC, we were alerted to a series of read errors stemming from the loki-prod3 cluster. At this time, some users may encounter 504 read errors when attempting to query for logs making use of the mentioned cluster.

Engineering is aware and actively investigating. We will continue to provide updates as information is shared.

Posted Mar 31, 2023 - 15:18 UTC

This incident affected: Grafana Cloud: Loki (GCP US Central - prod-us-central-0).