Engineering has mitigated and completed all actions towards resolution on the Alert State History issue for Grafana Managed Alerts. Users should now be able to view their Alert State Histories without issue. If adverse experiences persist, please reach out to Grafana Support.
Considering this incident resolved. No further updates.
Posted Feb 21, 2024 - 20:57 UTC
Update
Engineering has mitigated the immediate issue and all previously affected users should now be able to view their Alert State History for Grafana Managed Alerts. We are still working to complete the final releases to fully populate the necessary code changes into Grafana Cloud.
We will continue to monitor and provide updates.
Posted Feb 21, 2024 - 13:24 UTC
Identified
Engineering has identified the issue surrounding the Alert State History error for Grafana Managed Alerts and have developed a code change to correct the problem. Engineering is currently working through the release process to populate the code changes in all instances.
As of this time, the Alert State History UI will continue to display "An unexpected error occurred" for all instances running Grafana Cloud version 10.4.0-67274. Grafana Managed Alerts continue to evaluate as expected. We will provide additional updates as the release process progresses.
Posted Feb 20, 2024 - 21:00 UTC
Investigating
We have been alerted to an issue whereby the Alert State History for Grafana Managed Alerts is returning "An unexpected error happened" and does not load properly. This issue is present for any Grafana Cloud instance running version 10.4.0-67274, regardless of cluster. Please note this only impacts the Alert State History view; alerts themselves continue to evaluate as expected.
Engineering is aware and actively working towards remediation. We will provide updates as more information is shared.
Posted Feb 20, 2024 - 19:40 UTC
This incident affected: Grafana Cloud: Hosted Grafana (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS Germany - prod-eu-west-4, AWS India - prod-ap-south-1, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US East - prod-us-east-2, AWS US West - prod-us-west-0, Azure Netherlands - prod-eu-west-3, Azure US Central - us-central2, GCP Australia - prod-au-southeast-0, GCP Belgium - prod-eu-west-0, GCP Brazil - prod-sa-east-0, GCP India - prod-ap-south-0, GCP Singapore - prod-ap-southeast-0, GCP UK - prod-gb-south-0, GCP US Central - prod-us-central-0, GCP US Central - prod-us-central-3, GCP US Central - prod-us-central-4, GCP US East - prod-us-east-1, play.grafana.org).