AWS EC2 incident affecting CircleCI

Incident Report for CircleCI

Resolved

This incident has been resolved.

Posted Dec 22, 2021 - 14:22 UTC

Update

We are no longer seeing errors in our services, so we are marking this incident as resolved on our side.

However, AWS hasn't yet declared their incident as fully resolved, so due to our integration with the AWS status page, you will still see a "Partially Degraded Service" or "Minor Service Outage" banner on our status page, as well, as a "Degraded" Status widget in the CircleCI UI.

Nonetheless, we recommend monitoring the AWS status page (https://status.aws.amazon.com/) for further updates.

Posted Dec 22, 2021 - 14:22 UTC

Update

We are continuing to monitor for any further issues.

Posted Dec 22, 2021 - 14:06 UTC

Update

We are continuing to monitor for any further issues.

Posted Dec 22, 2021 - 14:06 UTC

Update

AWS has implemented a solution to restore service in the impacted Availability Zone (USE1-AZ4) in the US-EAST-1 region. (https://status.aws.amazon.com/)

We are continuing to monitor to ensure full recovery.

Posted Dec 22, 2021 - 13:44 UTC

Update

We are continuing to monitor the AWS incident (https://status.aws.amazon.com/), as well as, further assessing the impact on our services.

Posted Dec 22, 2021 - 13:24 UTC

Update

We are continuing to monitor for any further issues.

Posted Dec 22, 2021 - 13:03 UTC

Monitoring

AWS is currently experiencing an incident affecting EC2 instances in a single Availability Zone (USE1-AZ4) in the US-EAST-1 region.

The AWS status page reporting this incident is located here: https://status.aws.amazon.com

This can cause CircleCI builds failures with "Infrastructure Fail" for a subset of our customers.

We are currently monitoring the AWS incident and its impact on our system.

Posted Dec 22, 2021 - 13:00 UTC

This incident affected: Docker Jobs, Machine Jobs, Windows Jobs, Pipelines & Workflows, CircleCI UI, and Runner.