We still see increased latency on writes over many of our internal clusters and services; however, in most cases you need to look at 1 in 10,000 requests to see the behavior. Latency has been narrowed to our internal gateways and the traffic upstream. We have not pinpointed the source but it does appear most likely upstream of our systems. We have relaxed some timeouts to reduce the frequency of errors and continue to monitor the increased latency
Posted Jun 14, 2022 - 23:28 UTC
Update
We are continuing to investigate this issue.
Posted Jun 14, 2022 - 22:49 UTC
Update
We are still investigating the issue. Billing dashboard may also be impacted at this time, as well. We are currently seeing degradation in US Central for Prometheus ingestion and Billing dashboards.
Posted Jun 14, 2022 - 21:59 UTC
Investigating
We are seeing some outages for writes to us-central1. Mimir Team is currently working on this.
Posted Jun 14, 2022 - 19:46 UTC
This incident affected: Grafana Cloud: Prometheus (GCP US Central - prod-us-central-0: Ingestion).