Solana Network faced a significant slowdown in block production following a planned upgrade, negatively impacting transactions and forcing validators to downgrade their software to restore performance.
The network freeze continued into the weekend as validators prepared a second network reset.
Solana’s Last Interruption
The Solana Network faced yet another network slowdown after an upgrade initiated in the validator software. The incident, which occurred on February 25th, resulted in a significant disruption to transactions on the network. As a result, validators struggled to downgrade the validator software in an attempt to restore performance on the network. The problem started at 6:00 UTC when the network updated the validator software to 1.14. With the network outage, validators were forced to downgrade the network to version 1.13 to restore network performance. Solana’s Compass website noted,
“The network experienced a significant slowdown in block production that coincided with a validator software update. Engineers are still conducting a root cause analysis.”
However, the downgrade failed to restore normal operations on the Solana network, forcing validators to try to restart the network in 1.13.6. The network announced plans for a reboot on Twitter, stating,
“The Solana network is currently restarting after an issue during the upgrade from 1.13 to 1.14 which delayed the completion of the block. Once validators with 80% share are restarted, the network will resume.”
The root of the problem
According to available information, the problem is related to an update from version 1.13 to version 1.14, leading to a slowdown in block validation. To restart the network, 80% of active validators must resume operations.
“As more validators complete their reboot, this number will increase with the amount of participation they delegate: this means that larger validators like CEX have an outsized impact on reboot times.”
Solana validators remained in discussions over the incident over the weekend, with infrastructure provider Chorus One noting that the incident demonstrated the level of decentralization in the network.
“Without all these debates, we would be back in an hour. But, all the decisions along the way – whether to downgrade, to restart, when to switch from the downgrade approach to the restart approach – are up for debate. Voting takes place. We ended up taking 8-10 hours to recover instead of 1.”
The network freeze in Solana spread over the weekend as it continued into Saturday, with validators preparing a second restart attempt to restore services to users. At night, the validators concluded that the best way forward would be to sync a reboot and fork the chain. However, the first attempt had to be abandoned as the validators discovered that they had chosen the wrong starting point for the restart, further prolonging the outage.
An almost complete shutdown
The problem, which started as a simple slowdown in transaction processing, turned into an almost complete shutdown of any kind of activity on Solana, with block production stopping, causing transactions not to be processed or validated. As a result, on-chain crypto assets were rendered immobile until the network came back online. Major players in the Solana ecosystem were still looking to pinpoint the issue over the weekend, with one theory suggesting that a “fat block” messed up the mechanics of the blockchain.
Not the first network outage
This isn’t the first time Solana has had issues with network outages. In September 2021, the network was hit by intermittent instability due to what the protocol’s Twitter handle described as “resource exhaustion”. O network returned to the news in December 2021 after it emerged that technical difficulties and network congestion had led to another network slowdown. Just one month later, in January 2022, a 48-hour outage wreaked havoc on the solana network, forcing some users who took loans to liquidate their holdings.
Disclaimer: This article is provided for informational purposes only. It is not offered nor is it intended to be used as legal, tax, investment, financial or other advice.