We will continue to monitor the environment over the weekend. We will also keep this incident open till Monday morning as long as no further issues are detected.
Posted Nov 04, 2022 - 14:39 PDT
Monitoring
Health checks have been successful and technical teams have not observed any errors or issues in the logs since the change was deployed to production yesterday. We will continue to monitor the environment till the EOD tomorrow.
Posted Nov 03, 2022 - 11:20 PDT
Update
The fix has been deployed to Production. We are currently monitoring the change.
Posted Nov 02, 2022 - 12:42 PDT
Identified
The fix is ready to be deployed to the non-production environment today. We will be monitoring the change over the next day few days before deploying it to production.
Posted Oct 31, 2022 - 09:28 PDT
Update
Teams are actively working on a code change which will be later deployed to Staging for further testing. Due to the intermittent nature of the issue, the plan is to let it run for a few days in staging to ensure we do not encounter the same errors before production deployment; and perform additional testing as needed.
Meanwhile, as a temporary workaround, we will continue to manually re-run the failed tasks to remediate any customer impact.
Posted Oct 28, 2022 - 13:08 PDT
Update
We are continuing to actively work on a fix for this issue.
Posted Oct 28, 2022 - 08:42 PDT
Update
The investigation has revealed that there have been only a few occurrences since Saturday, October 22nd, impacting a very small customer base. We have also identified the cause of the failures and are currently working on a fix. We are continuing to monitor the environment and rectify any problems requiring manual intervention.
Posted Oct 27, 2022 - 13:17 PDT
Investigating
Around 1:37 AM PDT today, integration tasks failed again with 'signing failed "500 Internal Server Error". The outage may have lasted for approximately 10 mins. Services have been stable since, and teams are investigating the error logs.
Posted Oct 27, 2022 - 08:34 PDT
Update
We have not observed any issues and tasks have been running successfully since the additional resources were deployed. Technical teams will continue to monitor the tasks for another day to ensure a successful run.
Posted Oct 26, 2022 - 12:54 PDT
Update
We are continuing to monitor for any further issues.
Posted Oct 26, 2022 - 08:03 PDT
Update
We have added additional resources in the environment to avoid any memory contention issues. Technical teams are monitoring further.
Posted Oct 25, 2022 - 13:02 PDT
Monitoring
There have been no reoccurrences of this incident since Sunday, October 23rd. Integration tasks have been running successfully without errors. However, the root cause is still unidentified. We are continuing to investigate the root cause to avoid any reoccurrences.
Posted Oct 25, 2022 - 08:36 PDT
Investigating
Incident Description: Customers using SaaS Manager may receive intermittent errors while trying to retrieve daily Integration tasks data via Managed Applications. These tasks run only once every day. If the task gets stopped by the error one day, it will attempt again on its regular cadence the following day. As a result, the availability of new information could be delayed by a day.
Priority: 2
Restoration activity: Technical teams have been engaged and are currently investigating
Posted Oct 24, 2022 - 12:06 PDT
This incident affected: Flexera One - IT Asset Management - North America (IT Asset Management - US SaaS Manager).