뉴스레터 구독하기

블로그

Post / 2021-09-29

9/14 Devnet Bug Hotfix Retro

Find out more about recent upgrades to Mina’s network in our latest release retrospective.

By: Mina Foundation

Summary
  • On September 14th, a bug was discovered on Mina’s devnet testing environment via the core protocol engineers’ standard testing practice. The bug was not discovered on Mina’s mainnet or exploited, and no funds were compromised.
  • As a precaution, an emergency upgrade was released and also strongly recommended block producers on Mina’s mainnet to upgrade to the new network. More than 60% of the Mina network smoothly upgraded within 24 hours.
  • Additionally, to ensure there are no other bugs of this type, the core protocol engineers added more aggressive tests to that area of the code.  Additional edge cases that could lead to similar transaction logic issues were discovered and a follow-up upgrade with a larger, comprehensive set of fixes, was shared on September 23. More than 50% of the Mina network was upgraded in just over 24 hours.
  • Improvements will be made to the transaction pool to reduce the likelihood of future bugs.
Bug Discovery

At around 5pm PST on September 14th, a bug was discovered on Mina’s devnet environment. The bug was triggered by an automatic transaction-sending script that O(1) Labs – one of Mina’s ecosystem partners – runs as a standard testing practice on the devnet. Engineers discovered the bug after being notified by the devnet health alerting system — the same system used to monitor mainnet. The bug was identified promptly within a couple of hours of the alert going off, and although the bug had not been exploited, emergency hotfix discussions started the same day in an abundance of caution.

Technical Details

A block producer on devnet run by O(1) Labs experienced a glitch in its transaction pool, mistakenly choosing an invalid transaction to include in a block. This transaction moved more MINA than the amount that was available in the account. Other nodes confirmed this transaction. The receiving account did not exist, and the transferred amount was insufficient to pay to create that account, so nodes disregarded the full amount and only verified that the account contained enough to pay the fee. The transaction SNARK correctly rejected the transaction for using more funds than were held in the account, and prevented SNARK workers from proving the transaction. 

As a result of this invalid transaction, the devnet network is no longer processing new transactions and will need to be relaunched once this has been fully addressed. All of the caution and urgency in the incident response was intended to ensure that the mainnet did not confirm a similar bad transaction. We would like to highlight that this is not a trivial condition to recreate and is more likely to happen by accident than on purpose by an adversary.

Network Upgrade Implementation

Engineers implemented a fix shortly after the initial hotfix discussion, and a phased upgrade rollout was agreed upon. 

Mina participants were notified about the upgrade via a phased approach, starting with block producers and gradually expanding to the wider community, to prioritize the safety of the network and reduce the likelihood of the bug being exploited. 

More than 60% of the Mina network was upgraded within the first 24 hours. On September 23, a follow-up upgrade was shared with the whole community which encompassed a larger, more comprehensive set of bug fixes. More than 50% of the Mina network upgraded to this follow-up in just over 24 hours.

Network Improvements

Engineers have since fixed the transaction application logic to handle this case. They have also added tests to check that such transactions cannot be included in verified blocks, and if they are included (either maliciously or by accident) those blocks get rejected. The team at O(1) Labs is also working to add more checks to the transaction pool, to ensure that such a glitch is even less likely in the future.

Nodes still running earlier versions should upgrade to the newest releases as soon as possible. For participants currently on 1.1.5 or 1.1.7, please follow the release notes here. If you are currently running any of the 1.2.0 betas, please follow the release notes here. 

This is the first major hotfix on Mina’s mainnet since its launch six months ago and we’d like to thank our dedicated Block Producer community for their swift response, which ensured the situation remained controlled throughout. We are currently working on formalizing a new process for handling these types of situations and we welcome community members and developers’ continued contributions to Mina’s network security and ecosystem.

추가 블로그 글

블로그 / 2021-10-28 / Mina Foundation
Mina Protocol Product Priorities & Mina Foundation Mission
Sharing Mina Protocol’s near-term product priorities, as well as Mina Foundation’s role and focus within the Mina ecosystem
추가 자료
블로그 / 2021-09-30 / Mina Foundation
New Community Programs: Grants, Community Spark Leaderboard, Mina Academy, and Mina Achievements Program
추가 자료
블로그 / 2021-09-28
Least Authority Concludes Security Audit on StakingPower Wallet
추가 자료
블로그 / 2021-09-22
Update on Mina’s Supercharged Rewards Schedule
추가 자료

About the Tech

AboutTechCta

Mina uses advanced cryptography and recursive zk-SNARKs to deliver true decentralization at scale.

Get Started

GetStartedCta

Mina makes it simple to run a node, build and join the community.