Dates:

Friday September 27th 2022, 17:59 - 20:08 UTC

What happened:

During the duration of the incident, users were unable to start Real Device tests on devices below iOS 13.2. If a user attempted to start a session the device would appear to be busy and eventually would time out. 

Why it happened:

A new feature called WebRTC was deployed for iOS devices which required us to install a service application on our real iOS based devices. This service app was to be installed during our device cleaning process.This new feature was intended to go out for only devices running iOS 13 and higher. This was intended to be controlled by setting an iOS TARGET version in Xcode but due to a bug in that control it went out to all devices. 

When the service application was installed on devices running iOS versions lower than 13.2 it failed. During device cleaning if any step should fail the device is marked “dirty” and isn’t available for tests.

How we fixed it:

We tried to rollback the rdc-pool version which contains this WebRTC service app.We ran into two issues:

• We were not able to rollback, because this component is configured to prevent multiple deployments within 24 hours.
• We had to SSH into every single individual instance of this component, stop the queued deployment image, kill the service, and enable the new rolled back version.

What we are doing to prevent it from happening again:

We identified two areas to improve:

• Enable a more expedient deployment mechanism for this component.
• Fixing the bug that prevented us from targeting specific iOS versions.