Metrics Ingestion and Schema Checks Delayed

Incident Report for Apollo Graph, Inc.

Resolved

We have observed no issues while monitoring, and we believe this incident to be resolved. If you have any questions, please reach out to our support team.

As always, we appreciate your patience.
Posted 2 years ago. Dec 15, 2022 - 23:16 UTC

Update

Our metrics ingestion lag has recovered, and our operations check data storage system latency is no longer elevated. We are continuing to monitor our systems for further discrepancies.
Posted 2 years ago. Dec 15, 2022 - 22:16 UTC

Update

We've determined that while the storage system service was non-functional, our metrics ingestion lag also increased. This lag has been recovering since we implemented the fix, and continues to recover.
Posted 2 years ago. Dec 15, 2022 - 21:28 UTC

Monitoring

We believe the delay is due to increased latency in our operations check data storage system, due to an issue in one of the constituent services. We have fixed the issue with the constituent service, and the storage system is recovering.
Posted 2 years ago. Dec 15, 2022 - 21:19 UTC

Investigating

We are currently investigating an issue with schema checks being delayed. We are currently investigating to understand the scope of the degradation.
Posted 2 years ago. Dec 15, 2022 - 21:01 UTC
This incident affected: Metrics Ingestion and Schema Checks.