Query results degraded
Incident Report for Honeycomb
Resolved
We are unable to recover some of the data from between 2021-04-03 00:31UTC and 2021-04-03 01:51UTC for the 8% of datasets affected. You may notice a dip in total event throughput of up to 33% of expected traffic. Please contact support if you would like to confirm whether a specific dataset is impacted and by how much.
Posted Apr 03, 2021 - 00:49 PDT
Update
We are continuing repair the underlying services that were impacted.
Posted Apr 02, 2021 - 22:54 PDT
Update
Ingestion and querying continues to work correctly for all data since 01:51UTC; some data during the preceding 80 minutes is still unavailable for 8% of datasets. We are continuing repair the underlying services that were impacted.
Posted Apr 02, 2021 - 21:40 PDT
Monitoring
We have taken the impacted partition out of service and queries are degraded for roughly 8% of datasets. Some data is missing from queries for the time window from 2021-04-03 00:31UTC to 2021-04-03 01:51UTC. Data before and after that time range is unaffected.
We are continuing our investigation.
Posted Apr 02, 2021 - 20:29 PDT
Update
We are continuing to work on restoring access to the small subset of data that has been unavailable since 0037 UTC
Posted Apr 02, 2021 - 19:04 PDT
Identified
Between 0037 UTC and 0054 UTC incoming API requests to store telemetry experienced an elevated error rate. Additionally, some users may see incomplete data in their query results or delayed data starting from 0037 UTC.
Posted Apr 02, 2021 - 18:31 PDT
This incident affected: api.honeycomb.io - US1 Event Ingest and ui.honeycomb.io - US1 Querying.