Service Disruption - Syslog Data Delayed on the US4 Cluster

Root Cause Analysis

Duration of incident

Discovered: Jan, 10, 2023 - 14:21 UTC
Resolved: Jan 11, 2023 - 03:00 - UTC

Cause

The data framework related to a previous restore created an unforeseen data acceptance limitation.

Effect

Syslog data was delayed and unable to be refreshed

Action taken

All times UTC

01/10/2023
14:21 - First alert comes through for Syslog data being delayed to the UI.
15:33 - Syslog alerting reports significant reduction of insertions.
15:35 - Auvik Engineering is engaged to the issue.
16:00 - Engineering finds it can restore services but cannot remedy the underlying cause of the syslog data delay long term.
16:30 - Engineering engages a third party for support with the hosted services that are in distress and have caused the data delay.
17:00 - Engineering devises a plan to restore delayed data.
17:10 - Engineering begins restore of data.
17:22-22:13 - Engineering continues work on restoring data properly
22:15 - Engineering begins data validation process.

01/12/2023
3:00 - Engineering validates restored data and framework. Closes incident

Future consideration(s)

● Auvik will review and improve procedures with issues involving the underlying services affected in the incident.
● Auvik will implement in-house vendor training for specific services affected in this incident.
● Auvik will implement additional database monitoring.

Posted Jan 19, 2023 - 15:00 EST

Resolved

The fix for the syslog data delay has been deployed. The source of the disruption has been resolved, and services have been fully restored.

Posted Jan 10, 2023 - 22:01 EST

Update

We’ve identified the source of the service disruption with Syslog information for clients on cluster US4 and continue to work toward a resolution. Recent Syslog data is still not available. Engineering is continuing the effort to restore service fully. Updates will be provided when there is a change to status.

Posted Jan 10, 2023 - 16:29 EST

Update

We’ve identified the source of the service disruption with Syslog information for clients on cluster US4 and continue to monitor the situation. Work toward a resolution continues. Recent Syslog data is still not available. We’ll keep you posted on a resolution.

Posted Jan 10, 2023 - 15:30 EST

Update

Posted Jan 10, 2023 - 14:31 EST

Update

We’ve identified the source of the service disruption with Syslog information for clients on cluster US4 and are monitoring the situation. We continue to work toward a resolution. Recent Syslog data is still not available. We’ll keep you posted on a resolution.

Posted Jan 10, 2023 - 13:33 EST

Monitoring

We’ve identified the source of the service disruption to Syslog information for clients on cluster US4 and are monitoring the situation. We have begun to implement a fix. Syslog information is delayed to the UI. We’ll keep you posted on a resolution.

Posted Jan 10, 2023 - 12:37 EST

Update

We’ve identified the source of the service disruption with Syslog information for clients on cluster US4. Syslog information is delayed to the UI. We are continuing to work to restore service as quickly as possible.

Posted Jan 10, 2023 - 12:27 EST

Identified

We’ve identified the source of the service disruption with Syslog information for clients on cluster US4. Syslog information is delayed to the UI. We are working to restore service as quickly as possible.

Posted Jan 10, 2023 - 11:47 EST

Investigating

We’re experiencing disruption to Syslog information for clients on cluster US4. Syslog information is delayed to the UI. We will continue to provide updates as they become available.

Posted Jan 10, 2023 - 11:26 EST

This incident affected: Network Mgmt (us4.my.auvik.com).