Squiz internal monitoring detected intermittent timeouts for customers hosted in our Sacramento Data Centre (DC). This may have impacted all Squiz hosted applications in the DC, resulting in an intermittent downtime. Our network and platform teams were engaged and ran multiple checks on the Data Centre (DC) switch logs to identify a potential workaround solution. In parallel, our network team worked in conjunction with the 3rd party vendor to restore connectivity after a power failure to our core switches was identified. Multiple recovery actions were performed from our end that have successfully restored the service. At this stage, performed investigations have identified the root cause to be an undocumented change in behaviour of a redundancy feature after a software upgrade, which caused our config to no longer work. This incident has been recovered and impacted service(s) are fully restored.
Posted Aug 10, 2021 - 07:03 AEST
Update
Squiz network and platform teams are running checks on the Data Centre (DC) switch logs to identify a potential workaround solution. We are seeing intermittent recovery responses, but our teams are continuing to access the situation and ensure overall environment stability. Additionally, investigations are also ongoing with the third party vendor to restore connectivity.
Posted Aug 10, 2021 - 06:26 AEST
Update
We are continuing to monitor for any further issues.
Posted Aug 10, 2021 - 06:05 AEST
Update
Our network team is continuing to work with 3rd party vendors to restore connectivity. Issues may be intermittent as the network is being restored. Thank you for bearing with us.
Posted Aug 10, 2021 - 05:02 AEST
Monitoring
Our monitoring has detected improvements in network stability, however intermittent issues still persist. Our support and network teams remain engaged to resolve the issue.
Posted Aug 10, 2021 - 04:45 AEST
Update
Dear Customers,
This network issue remains under urgent investigation and all required teams are engaged in providing a resolution. Our network teams also remain engaged with our 3rd party providers.
Posted Aug 10, 2021 - 04:10 AEST
Update
Dear Customer,
Network troubleshooting continues and customers may still experience intermittent connect issues.
Posted Aug 10, 2021 - 03:46 AEST
Update
At this stage network connectivity may appear intermittent, however are still persistent. Our Support and Network teams remain engaged with 3rd Party providers.
Posted Aug 10, 2021 - 03:28 AEST
Update
Dear Customers,
Our Network teams are currently engaged with 3rd party providers investigating the issue.
No ETR is currently available.
Posted Aug 10, 2021 - 03:14 AEST
Update
Dear Customer,
Our network and support teams are engaged in identifying the issues.
Posted Aug 10, 2021 - 02:51 AEST
Investigating
Dear Customers,
Squiz Support has detected issues that may result in some customers experiencing network timeouts and connectivity.
Squiz Teams are working to restore service for all customers, at this time ETR is unknown. Next update to follow in 15 minutes.
If you have any questions, please contact Squiz Support