Investigating Issues with VMWare Carbon Black Response/Carbon Black Cloud Telemetry Ingestion and CrowdStrike Falcon Telemetry/Alert Ingestion

Incident Report for Red Canary

Resolved

This incident has been resolved.
Posted 2 years ago. Jun 13, 2023 - 22:36 UTC

Monitoring

Telemetry is now flowing normally and we are monitoring the system as we process the backlog of data.
Posted 2 years ago. Jun 13, 2023 - 21:11 UTC

Update

Carbon Black Cloud ENDPOINT STANDARD and Enterprise EDR users may experience degraded performance with event processing and Dashboard.

Security Impact: Prevention capabilities are unaffected. Visibility of events and alerts related to Prevention and Detection may be temporarily delayed until normal operation of Event Processing is restored.
Posted 2 years ago. Jun 13, 2023 - 19:58 UTC

Identified

We have identified the source of the outage in Amazon Web Services around increased error rates and latencies in the US-EAST-1 Region.
Posted 2 years ago. Jun 13, 2023 - 19:49 UTC

Investigating

We are investigating issues with VMware Carbon Black Response and CrowdStrike Falcon services hosted by Red Canary. Sensor telemetry, events, and detections may be delayed in our Web Portal, APIs, and systems that leverage those APIs such as Canary Exporter. New sensor connections and installations may be impacted.
Posted 2 years ago. Jun 13, 2023 - 19:46 UTC
This incident affected: Web Portal, API Requests, Detections, Endpoint Telemetry Ingestion (VMware Carbon Black Response hosted by Red Canary, VMware Carbon Black Cloud, CrowdStrike Falcon), and Alert Ingestion and Correlation (Crowdstrike Falcon).