GitLab Integration Errors
Incident Report for Gatsby Cloud
Postmortem

Summary

On May 17, 2022 at 08:16 AM ET, Gatsby Support was made aware that some customers were unable to authenticate to Gatsby Cloud because of a change in Gitlab authentication. This resulted in users unable to build their site.

The issue was resolved with a code fix in 10 hours.

Services Impacted

Gitlab Authentication

Timeline of Events

Identified: 2022-05-17 08:16 AM ET

Fix Verified: 2022-05-17 12:35 PM ET

Resolved: 2022-05-17 06:17 PM ET

Details

Gitlab introduced a breaking change in their OAuth APIs without informing their partners and customers. The Gitlab auth endpoint would error out when receiving the old shape of the token data prior to version 15 which is what Gatsby and many other customers were using. This prevented us from pulling the necessary information needed to pull the user code, and build the site.

To fix our OAuth flow we added logic that pulled down a new auth token. For tokens where this was not the fix, we added a simple button to the Gatsby Cloud dashboard to refresh the tokens manually.

Unfortunately, builds would stop processing on the first step during the repository clone process when using Gitlab.

Resolved

The authentication issue is now confirmed to be fully resolved.

Monitoring

Gatsby’s Engineering Team has successfully changed the logic of authentication with Gitlab and is monitoring to ensure this doesn’t occur again.

Identified

We’ve identified the authentication issue that occurred with Gitlab’s change and have applied logic to prevent this incident from happening again.

Posted May 24, 2022 - 07:29 PDT

Resolved
This incident has been resolved.
Posted May 17, 2022 - 15:18 PDT
Monitoring
GitLab moved to 15.0:

https://about.gitlab.com/blog/2022/04/18/gitlab-releases-15-breaking-changes/#oauth-tokens-without-expiration which set our tokens to expire after 2 hours. For companies who use a single workspace with Gatsby Cloud we have implemented a fix that will refresh tokens upon next build. For multiple workspaces attached to the same Gitlab Organization, we need the owners of these workspaces to reauthenticate with Gitlab(Sign out from Gatsby cloud and relogin or add a new site using GitLab). If you continue to have issues after reauthenticating with Gitlab, please open a support ticket
Posted May 17, 2022 - 13:59 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted May 17, 2022 - 08:50 PDT
Update
We are currently investigating this issue.
Posted May 17, 2022 - 05:23 PDT
Investigating
We are continuing to investigate this issue which is related to GitLab integrations.
Posted May 17, 2022 - 05:22 PDT
This incident affected: API.