Core developers, validators, and other members of the community are waiting for more data on oracle performance and load shedding efforts that have been made. Block production resumed but election times are slightly longer than expected. We will keep monitoring.
Posted Feb 05, 2023 - 20:38 UTC
Monitoring
After a few elections, block production and elections have resumed and Validator operators have reported that block times are more in line with expectations.
Proof-of-Coverage has resumed and the IoT Oracle has been brought back online with the version that we planned to deploy early this week. It has been in devnet testing for the last 12 hours and we feel that it resolves some of the issues that have been pending since the mainnet launch of Oracled PoC. We will acknowledge the fixed issues and address any pending bugs.
We will continue to monitor the chain for issues and look at increasing the transaction rate above 900 Proof-of-Coverage transactions per block.
Stay tuned and thanks for being patient over the weekend. Next update should be in the next two hours..
Posted Feb 05, 2023 - 18:34 UTC
Identified
This is placeholder comms for a decision made by the core devs and the Foundation a few minutes ago. We will come back with more specific communications on what we did, why it was necessary in an urgent manner, and what's next.
Core developers, validator operators, wallet and exchange operators, and others in the community have noticed significantly slower block times in the last 12+ hours due to a large backlog of Oracled Proof-of-Coverage.
No action is required by Hotspot Manufacturers, Validator operators, Data users, or Hotspot owners. We expect to resolve after we clear the backlog.
This could be due to a rate limiter that hasn't been implemented yet that needs to be addressed but in order to continue processing state channels, payments, burns, Hotspot onboards, location changes, etc., we believe the correct response is to halt the oracle that is pushing transactions on chain and let it recover. We will provide more details as we execute the steps defined in the blockchain channel on Discord.
Stay tuned and thanks for being patient over the weekend. We'll come back with a proper post-mortem and further steps during the week.
Thanks!
Posted Feb 05, 2023 - 17:04 UTC
This incident affected: Consensus Groups and Proof-of-Coverage.