Console Login Situation

Incident Report for ion interactive

Postmortem

On November 1st, a few customers experienced a brief login and/or downtime situation, due to a communication failure between servers and consoles, caused by a large amount of requests that generated slowness and errors, which made servers temporarily stop services as a protective measurement.

Between 12:04 and 2:02pm (GTM-3), many users were unable to log in and some consoles were offline for a few minutes at a time, varying among them. To resolve the situation, it was necessary to clear the queue of requests. After that, some consoles were normalized and others required additional recycling of failures to reestablish the communication.

Our servers are now configured for automatic cleaning in order to avoid similar events affecting consoles in the future.

Posted Nov 04, 2022 - 14:12 EDT

Resolved

This situation has been resolved for a few days.

Posted Nov 04, 2022 - 10:22 EDT

Monitoring

The situation seems mostly normalized now and the team will keep monitoring. A report should be provided soon.

Posted Nov 01, 2022 - 12:55 EDT

Update

We are continuing to investigate this issue.

Posted Nov 01, 2022 - 12:35 EDT

Investigating

We have identified a behavior in which some customers may have been experiencing difficulties logging in to their console and it is being investigated by our team. The Status Page will be updated once fixed. Thank you for understanding.

Posted Nov 01, 2022 - 11:47 EDT

This incident affected: ion interactive Platform.