Summary

Between Thursday, June 15, 2022, 13:33 UTC to 14:08 UTC, all users of the Infinity Portal (EU and US regions) suffered an outage- All users couldn't login, and those logged in were disconnected.

The event was triggered by a networking issue with our site-to-site communication which unexpectedly brought down our production system. Our internal alerting and client reports were clear to point on a major issue.

Once the site-to-site was recovered, the system then became stable again.

Incident Timeline

Thursday, June 16, 2022, 13:25 UTC – Reports start to gather on issues with using the portal.checkpoint.com. Also an alert is triggered. A war room is created to diagnose the issues.

Thursday, June 16, 2022, 13:33 UTC – It’s clear an incident has been started. The status page is updated.

Thursday, June 16, 2022, 13:40 UTC – We identify the site-to-site is offline and that it’s the cause for the incident.

Thursday, June 16, 2022, 14:08 UTC – site-to-site is recovered. It’s clear the system is back to being fully operational. Closing the incident.

Root Cause Analysis

The Infinity Portal uses a secured AWS VPC. In addition we have internal services working in an on-premise location which should impact our production system. A shared networking resource that was unrecognized until the incident affected our production system following the site-to-site outage and caused the Infinity Portal to be out of service.

Next Steps

1. Create detailed network diagrams, site-to-site monitoring and recovery plan in case of future issues and for the sake of planning the disconnection of the dependency of our production system from the site-to-site network. ETA: 3 weeks.