Summary:

New Load Balancers with Auto certificate provisioning were rate limited on Let’s Encrypt API.

Root cause:

F5 Distributed Cloud provides automated certificates for HTTP Load Balancers using Let’s Encrypt API. The majority of new Load Balancers with auto certificates were not creating around 6pm UTC on Dec 12th 2022. During certificate processing, F5XC received an  unknown error from Let’s Encrypt, as they were performing  emergency maintenance https://letsencrypt.status.io/pages/maintenance/55957a99e800baa4470002da/63978ca9eec28305808da195 .

Due to a higher than normal rate of failing pending authorisations against Let’s Encrypt, F5 account was rate limited and it took a few hours to process the remaining pending authorisations to resolve all auto certificates requests for new Load Balancers. This caused significant delay in HTTP Load Balancers creation. Around 4am UTC on Jan 13th 2023 everything got back to normal state.