Our investigation indicated memory usage was spiking in response to particular traffic patterns from the mobile apps. This traffic has stopped naturally, which allowed the system to stabilize. Error rates returned to normal at 07:35 UTC.
We've applied a mitigation to prevent this specific traffic from causing a re-occurrence and will resume our normal monitoring.
Posted Feb 20, 2022 - 08:49 UTC
Update
We have stabilized the system, and at this time we believe we have found the root cause. We are working to put a fix in place.
Posted Feb 20, 2022 - 08:16 UTC
Update
The stability problems appear to be related to memory usage. We've pulled in more of the team to help determine the cause and apply a fix.
Posted Feb 20, 2022 - 07:45 UTC
Monitoring
With the mitigation efforts in place we are continuing to monitor the error rates of the Platform.
Posted Feb 20, 2022 - 07:15 UTC
Update
We have applied some mitigating measures to bring down the error rate across the Eleos Platform, but we are continuing to investigate the root cause of the errors and come to a complete resolution of the issue.
We will post another update here in 30 minutes.
Posted Feb 20, 2022 - 06:38 UTC
Investigating
We are currently experiencing elevated error rates within the Eleos Platform when fetching up-to-date data, and when sending messages. Messages sent during this time will be stored on device, and Eleos Platform apps will continue to retry sending the messages until they are successfully sent.
We will post an update here in 30 minutes or sooner if new information is available.
Posted Feb 20, 2022 - 06:02 UTC
This incident affected: Eleos Platform (API, Mobile Apps).