Newly scanned documents are now processing at our normal turnaround time of less than 2 minutes, and we've worked through the backlog of documents that accumulated during the incident. All affected documents are now available in the document hub, FTP, and document API where applicable.
We appreciate your patience and understanding as we worked through this severe infrastructure outage.
Posted Dec 08, 2021 - 00:09 UTC
Monitoring
We've made a change to remove a bottleneck when generating document integration files so the rate we are processing documents that should appear in FTP gateway and document API has increased.
We'll mark this incident as resolved once we have fully worked through our document backlog.
Posted Dec 07, 2021 - 23:47 UTC
Update
We are continuing to work through document processing backlog, and have now processed over half of the backlog. Meanwhile, we are continuing to optimize our document processing backend to work against this backlog even faster.
Note: documents will appear in the document hub before they are available in the document FTP gateway and document API as we work to process documents with custom document integration formats.
Posted Dec 07, 2021 - 23:40 UTC
Update
We are currently running 3 times our normal peak capacity to process the current backlog of documents that was accumulated throughout the day during the outage. So far we've processed about 20% of the document backlog, and we are expecting to have completed processing our document backlog in about 2 hours.
We are working to speed up document processing to work through this backlog even quicker.
Posted Dec 07, 2021 - 23:07 UTC
Update
Our document image processing system is starting to successfully process documents. You should see documents captured throughout the day to start to appear in the Drive Axle Hub, document FTP, and document API.
The Eleos Platform Dashboard is back up and is functioning normally.
Posted Dec 07, 2021 - 22:40 UTC
Update
We are starting to see some errors when trying to load the Eleos Platform Dashboard. We will be monitoring the situation and will post updates here.
Our focus continues to be to getting document processing back up by deploying our systems to a different infrastructure region that is unaffected by the provider outage.
Posted Dec 07, 2021 - 22:28 UTC
Update
Our infrastructure provider is still working to resolve their issue within the region we operate.
We are continuing to work to deploy our document processing system in a region where we'll be able to process document images.
Posted Dec 07, 2021 - 22:02 UTC
Update
Our infrastructure provider is still working to resolve their issue within the region we operate.
We are continuing to work towards deploying our document imaging processing systems in a different region to resolve issues surrounding delivering documents.
Posted Dec 07, 2021 - 21:27 UTC
Update
Our infrastructure provider is still working to resolve the issue and cannot provide an ETA of resolution. Access to the Drive Axle Hub appears to be resolved.
In parallel we are working on bringing up our document processing system in a different provider region to hopefully begin processing document uploads.
Posted Dec 07, 2021 - 20:53 UTC
Update
Our infrastructure provider at this time cannot provide an ETA on the resolution to this issue. As soon as they do or our own monitoring indicates resolution to the outage we will post an update here.
We are continuing to monitor our systems to ensure the outage doesn't spread to effect other aspects of the Eleos Platform.
Posted Dec 07, 2021 - 20:17 UTC
Update
Our infrastructure provider has identified issues within their core networking system within the region we utilize. They are currently working to resolve the issue.
Meanwhile we are working to mitigate the impact to Drive Axle Hub where possible.
Posted Dec 07, 2021 - 19:34 UTC
Identified
We've confirmed that no documents have successfully been processed through Drive Axle or the Platform mobile apps since 15:45 UTC due to the upstream infrastructure provider issues. With this in view, we're upgrading the "Document Hub and Drive Axle" portion of this incident to a major outage.
While our infrastructure provider works to resolve the underlying issue, we are attempting to find workarounds or mitigations that can get documents flowing again. Documents scanned during this time will be queued either in the mobile app or in durable storage in our backend systems until they can be processed normally after the root cause is resolved.
At this time, no components of the Eleos Platform or mobile apps besides document processing the Document Hub and API, and FTP are affected. With an eye to transparency, we're keeping the Mobile Apps listed as "Degraded performance," but this is limited in scope to document scanning. Please investigate and raise any issues not related to these components as you would during normal operations.
One note: Platform Dashboard users may have trouble getting logged in, since the login page is served via the Document Hub's infrastructure. About 50% of requests are succeeding, so try a few times if you are not able to log in.
We'll provide another update in 30 minutes, at around 19:30 UTC.
Posted Dec 07, 2021 - 18:55 UTC
Update
Issues uploading documents from the Drive Axle app, Eleos Platforms apps, and accessing the Drive Axle Hub continue to persist. During this period, scanned documents captured within the app will queue to be uploaded at a later time.
Our infrastructure provider is continuing to investigate the issue, and we are continuing to monitor our systems to ensure the impact of this outage doesn't spread to cause more disruptions.
Posted Dec 07, 2021 - 18:09 UTC
Update
Drive Axle Hub and Eleos Platform apps document uploads continue to experience delays.
Our infrastructure provider is still investigating resolving the root cause. Meanwhile we are investigating possible mediation steps we can take to lessen the impact of the issue.
Posted Dec 07, 2021 - 17:29 UTC
Update
We now believe that the Eleos Platform app will experience issues uploading documents. During this period, scanned document captured within the app will queue to be uploaded at a later time.
Posted Dec 07, 2021 - 17:08 UTC
Investigating
We are currently investigating an issue within our infrastructure that may cause issues related to the Drive Axle Hub which will manifest as timeouts when trying to access the Drive Axle Hub. Our upstream provide has indicated they have identified the problem and are working to resolve the issue.
At this time we don't believe that the Eleos Platform is impacted. If our monitoring begin to indicate that it is we will update this page.
Posted Dec 07, 2021 - 16:59 UTC
This incident affected: Eleos Platform (Dashboard, Mobile Apps) and Document Hub and Drive Axle.