On 2020/09/01 15:30 UTC Hologram detected an isolated, unplanned outage affecting a small proportion of Hologram’s live SIMs due to a misconfiguration by one of our upstream providers. As a result, cellular connectivity was denied on these SIMs and those that were affected were unable to connect and pass data. The root cause of the isolated outage was resolved by the upstream provider as of 2020/09/01 22:30 UTC. After the misconfiguration was corrected, 90% of affected SIMs recovered without issue. A subset of the affected SIMs that had not recovered as of 2020/09/02 23:51 UTC were monitored by Hologram and specific remediation paths have been identified.
On 2020/09/01 10:43 UTC a small proportion of Hologram’s live SIMs had their roaming profile misconfigured. Thereafter, devices attempting to connect during this time began receiving “roaming not allowed” messages effectively resulting in an isolated, unplanned outage for those devices.
Additionally, while a large majority of devices recovered without issue after the misconfiguration was corrected, <1% of SIMs failed to reconnect as of 2020/09/08.
At approximately 2020/09/02 01:30 UTC configuration was rolled back to each SIMs original state. This process completed at approximately 2020/09/02 03:30 UTC.
For the remaining affected SIMs, Hologram’s continued monitoring shows some SIMs are coming back online on their own. Hologram is working with customers for any required intervention.