All jobs are queued for a long period of time
Incident Report for Channable
Resolved
The incident has been resolved, Google Text and Shopping Ads are now running at normal capacity again.
Posted Dec 18, 2021 - 12:41 CET
Monitoring
We have received confirmation from Google that they have resolved the issue that caused the degrated performance that we were experiencing. This means that you as a user should now see normal processing times again within your Channable account.

Any updates that could not have been sent due to the issue on Google's side, will be sent on the next scheduled synchronization.

We will monitor your accounts to confirm that the issue has indeed been resolved.
Posted Dec 17, 2021 - 23:15 CET
Identified
The problem has been identified and caused by a combination between Google related jobs taking longer and our scheduling methods being ill-equipped to handle this case. Google has been made aware about our observed issues and are investigating. Meanwhile, we are adjusting our job scheduling system to be prevented this from causing issues in other parts of the tool.

The performance of Google Shopping Ads and Google Text Ads tasks remains degraded for now.
Posted Dec 17, 2021 - 17:59 CET
Update
We found that the issue to originates from our Google Ads related jobs, we have limited the amount of jobs that can run for this feature. This resolves the throughput problem for Feeds and Marketplaces.
Posted Dec 17, 2021 - 17:02 CET
Investigating
We are investigating degraded performance in our job scheduling system, jobs are queued for longer than expected.
Posted Dec 17, 2021 - 16:43 CET
This incident affected: Application, Feeds, Ads, and Marketplaces.