Splunk APM Trace Data Delayed

Incident Report for Splunk Observability Cloud US2

Resolved

This incident has been resolved.
Posted 2 years ago. Mar 16, 2023 - 11:48 PDT

Update

The major outage is over at 11:44 (PDT). We will continue to monitor the systems.
Posted 2 years ago. Mar 16, 2023 - 11:44 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted 2 years ago. Mar 16, 2023 - 11:40 PDT

Investigating

A degradation in the performance of the Splunk APM data ingestion pipeline is causing the processing and storage of raw trace data to be delayed by more than five minutes. No data is being lost at this time and MetricSets are not impacted but the most recent data may not be available in trace search results.
Posted 2 years ago. Mar 16, 2023 - 11:38 PDT
This incident affected: Splunk APM (Splunk APM Troubleshooting MetricSets, Splunk APM Trace Data, Splunk APM Tag Spotlight, Splunk APM Business Workflows).