Infrastructure/hardware instability Incident Report  

‌

Date of Incident:05/01/2023  

Time/Date Incident Started: 05/01/2023, 5:00 pm EDT  

Time/Date Stability Restored:05/01/2023, 11:48 pm EDT  

Time/Date Incident Resolved:05/01/2023, 11:48pm EDT  

Users Impacted: All 

Frequency: Intermittent 

Impact: Major  

Incident description:  

Third party vendor infrastructure/hardware instability 

Root Cause Analysis:  

A third party vendor infrastructure issue affected performance and system availability for the underlying data storage layer servicing platform resources. 

Actions Taken:  

1. Investigated system-generated alerts and identified affected platform functionality  
2. SRE and DBA teams initiated a platform infrastructure redeployment, forcing the new infrastructure to spun up on unaffected infrastructure/hardware 

Mitigation Measures:     

1. Continue the ongoing investigation into root causes of the infrastructure issue within our cloud hosting provider.  
2. Continue to implement high availability improvements to prepare the platform to respond better to unexpected hardware issues that are beyond our control.