ETL Services have re-synced fully and is now starting to process a majority of the backlogged transactions. You may see some transactions fail due users retrying them with the same nonce. If you submitted transactions more than once, verify your Add Gateway, Assert Location, Payment, and Transfer Hotspot transactions.
Posted Aug 14, 2021 - 17:20 UTC
Monitoring
The changes made should help the API return to normal after a few hours. We'll continue to monitor the situation and release the status from Degraded when the pending transaction count reaches a more normal state and/or syncing in the ETL matches block production.
Posted Aug 14, 2021 - 07:24 UTC
Identified
The team has released a change to ETL that improves its gossip connection to the blockchain. This has allowed blocks to start syncing more consistently and transaction processing to resume. There is a backlog of about 4600 pending transactions that will be processed so it may take a few hours for these to clear.
Posted Aug 14, 2021 - 05:27 UTC
Investigating
The core developers are actively investigating issues with the the main API services (api.helium.io). We've identified and confirmed reports that the API is falling behind in block sync, has stale Hotspot data, and is not clearing pending transactions at a normal rate. Users of the network may see payments and other transactions submitted via the API (through the Rust CLI or Hotspot App) delayed.
The blockchain itself is operating normally, Validators continue to produce blocks, and other submitted transactions (directly from Hotspots for PoC, through blockchain-node, and through Validators) continue to process normally.
We will continue to investigate the root cause and provide more details as we have it.
Posted Aug 14, 2021 - 03:32 UTC
This incident affected: User Apps (API) and Explorer, Helium App.