Block Sync Degraded
Incident Report for Helium
Resolved
Proof-of-Coverage seems to be returning to a normal rate since this outage and the core developers are closing this incident as resolved.
Posted Aug 25, 2021 - 01:19 UTC
Monitoring
The core developers are monitoring the network as hotspots start to come online with the new firmware. It will take a few hours before Proof of Coverage gets back to normal. We'll keep this incident open until we start to see more activity.
Posted Aug 24, 2021 - 17:36 UTC
Update
The core developers have released version 2021.08.24.1 to GA and have started the OTA process for updating the firmware on Original Hotspots and RAK hardware. Manufacturers should update to this version as soon as the docker images are available.
Posted Aug 24, 2021 - 16:47 UTC
Identified
The core developers have identified an issue that is affecting Hotspots syncing and an emergency fix for Hotspots is being prepared that will help Hotspots sync again and bring block gossip back to normal.

Hotspots that are stuck at block 979920 are the ones that are affected by this issue.
Posted Aug 24, 2021 - 15:42 UTC
Investigating
The core developers have noticed that, as of 1:15am UTC, block sync has been degraded. We are looking into the issue.

Transactions may take a long time to clear (including payments, onboarding hotspots, transferring hotspots, etc.) and hotspots may fail to perform Proof-of-Coverage activities because they are not in sync.

The Consensus Group is producing blocks but they are not syncing to all network participants. Elections may not succeed as new proposed groups may not be in sync with current blocks.
Posted Aug 24, 2021 - 15:18 UTC
This incident affected: User Apps (Blockchain) and Consensus Groups, Proof-of-Coverage.