Processing data in EU2 disturbed
Incident Report for Auvik Networks Inc.
Postmortem

Service Disruption - TrafficInsights and Syslog data flow interrupted

Root Cause Analysis

Duration of incident

Discovered: Nov 2, 2021 9:50 - UTC
Resolved: Nov 2, 2021 10:20 - UTC

Cause

The backend service was shutdown as it reached the disk threshold emergency limit for free space available.

Effect

This impeded the disc availability to collect TrafficInsights and Syslog data for performance clients on the EU2 cluster. Data could not be recovered during this period. Auvik Performance clients on other clusters as well as Auvik Essentials clients were not affected by this service disruption.

Action taken

11/03/2021 - All times in UTC

09:50 UTC Auvik receives the information that the backend service for the TrafficInsights and Syslog located on the EU2 cluster has been shut down due to lack of free space.
09:52 UTC Standard disc scale up is prepared by the Auvik engineering team. The team identifies the affected services.
10:02 UTC Auvik team implements the fix..
10:20 UTC Auvik team validates the fix and verifies the data flow.

Future consideration(s)

Auvik will demonstrate full review of current disk space alerts to validate proper thresholds are in place
Auvik team will improve the automation process to prevent disk thresholds from reaching a shutdown limit.

Posted Nov 24, 2021 - 14:46 EST

Resolved
No issues detected, resolving.
Posted Nov 03, 2021 - 06:51 EDT
Monitoring
The fix was applied data is processed again, monitoring.
Posted Nov 03, 2021 - 06:23 EDT
Identified
We are experiencing problems in EU2 with processing data, we identified the issue and working on a fix.
Posted Nov 03, 2021 - 06:05 EDT
This incident affected: Network Mgmt (eu2.my.auvik.com).