Control Is Currently Unreachable

Incident Report for Workspot

Postmortem

We have been provided an RCA from our PaaS provider.

”Between August 23, 2022 17:54UTC and August 24, 2022 03:33UTC, our customers experienced DNS resolution failures in all regions of Private Spaces and Common Runtime. We sincerely apologize for the negative effects our customers experienced

A failure originating with our upstream DNS provider began at 17:54 UTC. This failure initially caused DNS propagation delays for new apps and newly added custom domains, deteriorating at 23:30 UTC, when existing applications also could not complete DNS resolution.

The failure was detected automatically by our monitoring systems and engineers were paged in, immediately engaging the provider. During this time, we learned that a migration that was performed on our account was directly responsible for the system degradation. The DNS provider’s engineers started working on mitigation at 20:18 UTC, resolving the issue at 00:56 UTC.”

‌

Workspot Control service was impacted by this DNS resolution failure/outage from 23:15:44 UTC on 8/23 to 00:54:44 AM on 8/24.

Posted Aug 31, 2022 - 17:29 UTC

Resolved

Workspot Control service is reachable, and we have not observed the issue in the last couple of hours.
Our Service provider has confirmed that the issue with the upstream DNS provider is resolved now. We will update the incident with an RCA once we have it.
Thank you for your patience.

Posted Aug 24, 2022 - 03:47 UTC

Monitoring

Workspot Control Service is once again reachable.
We are waiting on an update/RCA from our PaaS provider that experienced the DNS issue.
We are continuing to monitor the situation.
We will update the incident with a post mortem/RCA from our PaaS provider once we have it.
Thank you for your patience.

Posted Aug 24, 2022 - 00:59 UTC

Update

We are receiving reports that some customers are able to access Control. It appears to be location dependent.
All end users should be able to access their VMs, the only impact felt currently is Control is unreachable.
Our PaaS is experiencing DNS issues, is aware of the issue and is investigating.
We will continue to update status as warranted.

Posted Aug 24, 2022 - 00:07 UTC

Investigating

As of 4:45PM we are aware that Control is currently unreachable.
Our service provider is experiencing DNS issues and is aware of the issue and investigating it.
We will keep you updated as we get more information.

Posted Aug 23, 2022 - 23:53 UTC

This incident affected: Workspot Control.