Managed Cloud production deployment was terminated successfully. The issue with Secure (Modeling) workers is mitigated. Deployments Stats on the DataRobot GUI are catching up as well. We expect them to be fully up-to-date within the next 1 hour.
The incident is contained. Please reach out to support@datarobot.com if there are any questions.
Posted Sep 13, 2021 - 10:56 UTC
Identified
The engineering team is terminating the Managed Cloud production deployment to mitigate the performance degradation. Once it is completed, the issue with Secure (Modeling) Workers should be eliminated. However, we will execute additional tests to confirm.
At the same time, there are other issues with Deploymnets Dashboard statistics. V1 dedicated predictions are not affected, however, customers will see statistics updated on the GUI with a delay. Current the delay is 2-3 hours, however, it is already catching up. We will provide another update once the statistics are up to date.
The next update will be provided once the issue with Secure (Modeling) Workers is eliminated.
Posted Sep 13, 2021 - 10:38 UTC
Investigating
DataRobot Managed AI Cloud is experiencing issues with Secure (Modeling) Workers. Any jobs submissions on the Modeling Workers might be affected.
The engineering team is evaluating the scope of the issue and potential mitigation. The next update will be provided within 1 hour.
Posted Sep 13, 2021 - 10:09 UTC
This incident affected: Managed AI Cloud (Website, AutoML).