As of 17:01 UTC, Engineering has applied a remediation and we continue to observe normal behavior for metric ingestion and querying. At this time, we are considering this incident resolved.
No further updates.
Posted Sep 11, 2023 - 17:51 UTC
Update
As of 15:00 UTC, we have observed significant improvement in error rates and users should begin experiencing normal behavior when writing metrics and querying metrics. Engineering will continue efforts to ensure long term nominal behavior and we will continue to provide updates as we learn more.
Posted Sep 11, 2023 - 15:11 UTC
Investigating
As of 14:33 UTC, we became aware of an issue affecting Hosted Metrics in the eu-west-2 cluster only. Users encountering this issue will experience 5xx errors when attempting to either write metrics or query metrics in their Hosted instances. Engineering is aware and actively investigating. We will continue to provide updates as information is shared.