System Status

Zoom user license issue
Incident Report for Rippling
Postmortem

Overview

The Zoom app in Rippling caused a provisioning error for license type assignment at 2:00 AM PST and was restored at 11:08 AM PST. This resulted in employees with paid license types in Zoom to be downgraded to Basic, and upon restoration to their paid license, their add-on licenses types to be lost.

Root Cause

While adjusting the configuration of licensed Zoom users based on new Zoom license types, there was a misconfiguration that caused Rippling’s system to think Licensed and On-Prem types were not available in Zoom. This caused all users to be moved to the Basic license type.

Resolution Path

‌We were able to restore the Supergroups associated with license types, and then restore all users' license types in Zoom for all customers by 11:08 AM PST.

In the rare case of a Zoom Pro or On-Prem user with support for Large, Concurrent or Webinars, please read the section below as we are unable to restore supplemental license options.

Customers with Zoom add-ons and concurrent meetings

Zoom Licensed or On-Prem users that have add-on functionality need to be restored in Zoom directly by their Zoom admin. These add-ons are only available for certain Zoom plans

You can view and edit these options by using Zoom’s User Management.

Action Items

  • Rippling will update its synchronization logic to ensure that empty Supergroups sets are not sent to third-party applications.
  • Rippling will update its test suite to ensure that Supergroups tied to license types across all third-party apps will not send empty Supergroup sets
  • Rippling will store additional license information for future mitigation, if needed.
  • Rippling is updating automated testing infrastructure to ensure that existing Supergroups maintain expected behavior with updates to App configurations

Rippling’s mission is to Free Smart People to Work on Hard Problems, and we can only achieve that by earning the trust of our customers. When we fail to achieve our mission, we strive to be transparent with customers, and to outline our commitment to preventing future occurrences of similar incidents.

We apologize for any impact this incident has had on your company operations.

Posted Mar 17, 2022 - 21:55 UTC

Resolved
This incident has been resolved.
Posted Mar 15, 2022 - 19:28 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 15, 2022 - 18:06 UTC
Update
We are currently rolling out a fix to all affected accounts and will provide an update at 11:00 AM PST
Posted Mar 15, 2022 - 17:35 UTC
Update
We are currently deploying a fix and will provide an update at 10:30 AM PST.
Posted Mar 15, 2022 - 17:13 UTC
Update
We are continuing to work on a fix for this issue.
Posted Mar 15, 2022 - 16:50 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Mar 15, 2022 - 15:50 UTC
Investigating
Employees with Zoom licenses have lost access to their assigned license. We are currently investigating the issue.
Posted Mar 15, 2022 - 15:48 UTC
This incident affected: Rippling App.