We would like to share more details about the events that occurred with Phrase between 10:36 AM CEST and 3:48 PM CEST on March 7, 2024 which led to inability to log in to Phrase Suite for a limited group of users and what Phrase engineers are doing to prevent these issues from happening again.
March 7, 10:36 AM CEST: Problematic code deployed. Soon after, an issue was reported by a user → not possible to login using 2FA authorization. Commenced Initial analysis.
March 7, 12:56 PM CEST: Phrase Suite team notified about the 2FA issue and working on the 2FA fix.
March 8, 10:41 AM CEST: Internal testing of the fix reveals the source of the problem; not 2FA, but special characters in passwords (see Root Cause section).
March 8, 11:58 AM CEST: Fix deployed to production.
March 8, 1:13 PM CEST: Discovered another case with some UTF-8 characters that disallowed some users to log in to the Phrase Suite.
March 8, 16:01 PM CEST: Complete fix deployed to production.
Root Cause
Issue tracing activity was confused by the 2FA information. Note that there has never been an issue with TFA in the Phrase Suite.
A solution was broken with an upgrade of the NUXT-security package (XSS plugin) plugin that filtered out any content containing “<” or “>” characters (buggy behavior in the library, fixed in the latest version). We overcame this limitation in all places where passwords were handled and this solution broke 2FA and UTF-8 characters handling in passwords.
It was not possible to upgrade the library to the latest (and fixed) version at the time of the incident (it is now - e.g. April 4). We temporarily downgraded to the original version to hot-fix the issue.
To prevent such issues in the future, the NUXT-security package was upgraded to the latest version (which was possible a few days later after this incident) and extended our test suites to use UTF-8 and “<”, “>” characters on all incident-impacted places and increased the coverage of 2FA functionality.