Cornerstone Status - Minor Issue - Latency

Minor Issue - Latency - LHR SL1/LHR SL2/LHR SL3

Incident Report for Cornerstone

Postmortem

On January 1st, 2022 Cornerstone engineers were alerted by internal monitoring tools for network errors. Clients hosted in the LHR datacenter may have identified portal latency and exceptions intermittently.
Cornerstone engineers were engaged for investigation and troubleshooting pointed towards issues with SQL & Network. After collaborative deep dive sessions, the issue was identified with the hardware component of one of the switches. This was fixed by disabling the port to allow traffic to route through other ports.

To mitigate this situation from occurring in the future and to reduce the risk of a future impact Cornerstone is actively working internally to review the incident with the hardware component. Internal teams are working with our third-party vendors to establish the issue and conducting a thorough investigation and creating remediation plans to avoid the recurrence of this incident. Additional layers of monitoring/alerts were implemented.

Posted Jan 05, 2022 - 22:59 PST

Resolved

The CSOD Technology Team observed a performance degradation affecting this swimlane. The problem was resolved at 3/1/22 10:15 AM Pacific Time. During this time, clients with portals on this swimlane may have experienced slow performance or intermittent errors while accessing the application.
RCA will be published here in 7-10 business days.

Posted Jan 03, 2022 - 11:34 PST

Monitoring

This issue has been identified and fixed.
We continue to monitor while we recover and will update again once we deem this issue fully resolved.

Posted Jan 03, 2022 - 10:45 PST

Update

We are continuing to work on and investigate this matter with our 3rd party vendor and internal teams as a top priority - we will provide further updates once available.

Posted Jan 03, 2022 - 10:08 PST

Update

We are continuing to work on and investigate this matter with our 3rd party vendor and internal teams - we will continue to provide updates once available.

Posted Jan 03, 2022 - 06:58 PST

Update

We are continuing to work on and investigate this matter with our 3rd party vendor and internal teams - we will provide further updates once available. We apologise for the inconvenience to you and your users in the interim.

Posted Jan 03, 2022 - 04:51 PST

Identified

Cornerstone is working with our 3rd Party vendor in order to further investigate this matter - we will provide updates once available and keep this incident update on Trust.
Please note this incident may include impacts to feed runs and integrations.

Posted Jan 03, 2022 - 01:37 PST

Investigating

This swimlane is experiencing some latency issues. Clients with portals in this swimlane may experience delays or issues accessing certain pages in the application. This is our top priority and we are working to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Posted Jan 02, 2022 - 22:29 PST

This incident affected: LHR-SL1 (Response Time), LHR-SL2 (Response Time), and LHR-SL3 (Response Time).