Testworld Mission 2.0: Track 3 Retrospective

Highlights

• The Testworld Mission 2.0: Protocol Performance Testing program received more than 3,305 applications from over 70 countries.
• 250+ experienced node operators were selected to provide the network backbone for Testworld Track 3.
• Operators ran more than 670 individual nodes and processed more than 29k blocks, 176k transactions, and 85k zkApp transactions over 10 rounds of load testing.
• Track 3 allowed for the testing of various loads and helped uncover issues which have since been resolved. As a result, an optimal configuration was identified, and the release candidate for the Mainnet Upgrade is ready.
• Next, there will be testing of all the processes and tooling for the upgrade mechanism as the final preparation step before the major Berkeley Upgrade.

Overview 

There are three key features of the Berkeley mainnet Upgrade which were proposed and voted on by the Mina community through on-chain governance (MIP1, MIP3, MIP4). 

• ZK programmability with o1js, previously “SnarkyJS,” which is a TypeScript library for writing zkApps. 
• Enhanced security and efficiency from Kimchi, Mina’s marketing-leading proof system that facilitates the creation of recursive ZK proofs.
• Removal of supercharged rewards, a temporary incentive to increase staking adoption at the earliest stages of mainnet.

Mina’s Testworld Mission 2.0: Track 3 (Protocol Performance Testing) was launched to stress test the protocol and push the network to its limits, with the ultimate goal of optimizing protocol performance ahead of this major protocol upgrade. The program brought together more than 250 experienced node operators to perform various testing tasks, including block production, load testing,SNARK work production, and more. Since the beginning of the program in October, more than 670 individual nodes have processed 29,000 blocks, 176,000 transactions, and 85,000 zkApp transactions across 10 rounds of load testing.

Community support was paramount to the success of the pre-upgrade testnet. Aside from active participation in Testworld Mission 2.0 Track 3, community members also contributed through Q&A, guides, scripts, wallets, and explorers, and showed remarkable adaptability. 

Additionally, testing efforts throughout the program have allowed the team to address and resolve critical issues relating to block production, node catch up performance, and memory usage, among others. Overall, strengthening network infrastructure and paving the way for the upcoming Berkeley Upgrade.

Fixes & Improvements 

Initially set to last three epochs, a series of critical issues found in epoch 3 of the testing period needed immediate attention. This required the schedule to be extended by a further two epochs, carrying it through to January 22, 2024, for additional testing and to observe network behavior once fixes were implemented.

A total of 4 node versions were released over the period, contributing to efficiency, performance, and stability gains that the team is excited to share with you. 

GRAPHQL, CLI, UX, AND LOGGING 

• CLI command to dump libp2p keys #14169
• Fix parsing of URLs #14809
• add timing log to check_database #14667
• Remove some oversized logs #14054
• Hardcode number of oversized logs #14286
• New protocol versioning #14227

LEDGER IMPROVEMENTS

Ledger performance has been optimized to address an issue with block production performance. The optimizations show more than a 10x performance improvement to transaction application time, with additional performance optimizations queued up.

• More memory-efficient scan state and staged ledger hashing #14524
• Fix genesis ledger directory growing in size #14819
• Batch account location lookups for sparse ledger #14522
• Batch merkle_path lookups in Sparse_ledger #14528
• Replace tables with maps in merkle masks #14595
• Use ‘wide merkle paths’ to optimize Sparse_ledger.of_ledger_subset_exn #14594
• Fix block producer long async cycle #13654

OPTIMIZATIONS

• Avoid duplicating proofs during txn verification #14525
• Fetch merkle paths from masks instead of disk #14570
• Node performance improvement by batching transactions when writing to persistent frontier #13557
• Remove constructors for account precondition #14117
• Remove duplicate account inclusion checks #13422
• Reduce delay in broadcasting blocks #13551
• Avoid slowdown when using multiple masks #14617
• Field: More efficient conversion into Bignum_bigint.t #14599
• Fix duplicated rows in zkapp_account_precondition table #14785
• Field: More efficient conversion into Bignum_bigint.t #14599
• Fix snark pool long async cycles #13556
• Fix bootstrap long async cycle #13885

ADDITIONAL IMPROVEMENTS 

• Batch account lookups for sparse ledger creation #14527
• Stop printing accounts from runtime config #1407
• Fix non-determinism in scan state representation #13435
• Base58Check for receipt chain hashes in account preconditions #14418
• Format last_vrf_output as Base64 #14462
• Function to preserve sign for 0 #14459
• Generate script to patch receipt chain hashes and last VRF output #14419
• Use generated field name for subchain queries in extract_blocks #14422
• support ci on forks #14596
• Porting CI fixes from develop and berkeley #14638
• Avoid ledger copy and mutation in Sparse_ledger.of_ledger_subset_exn #14587
• Allow merkle masks to handle empty accounts directly #14585
• Preload accounts into merkle path for staged ledger diff application #14571
• Fix/archive db add indexes #14792
• Adding the ability to soft-cap #14644
• batch insertion events in archive database #14779
• Handle inability to contact metrics server on forwarding #14505
• Support setting zkapp limit in orchestrator #14818
• Child_processes: ensure stderr/stdout get flushed #14696
• Disable rate limiting for snark works #14174

A forthcoming release candidate will address outstanding issues related to adding blocks to the archive database and the Rosetta API.

Findings 

The third track of Testworld Mission 2.0, protocol and performance testing, completed what it set out to do: stress test the network, ensuring the robustness and efficiency of the upcoming Berkeley upgrade. 

A new testing framework allowed the team to work closely with the community, and send vast amounts of transactions to the network in a controlled way, providing lots of high-quality data. Overall, engineering teams were able to iterate quickly, running tens of multi-day tests with varying configurations, contributing to the identification and fixing of critical performance issues. 

Early tests helped to identify a performance regression when creating large blocks, for which a series of optimizations have been pushed out, reducing time spent by approximately 100x. Other important issues that gained focus thanks to this Testworld track are the genesis ledger growing in size, which is now fixed, and archive node performance, which has improved significantly. Solutions for increased memory usage are still being explored and will be applied post-Berkeley through minor upgrades. 

Finally, the team is happy to report that the network performed well under high load, and can handle spikes of transaction activity. A special thank you to both the community testers and the teams at o1Labs who made this testnet successful. The Berkeley Upgrade is now one step closer, with only Track 4, the final stage of testing the upgrade mechanism, pending completion.

Next Steps 

The testnet for Mina’s Berkeley Upgrade is split into four tracks, with the first three tracks including zkApp End-to-End (E2E) testing, external security auditing, and protocol performance testing now complete. The fourth, which will begin in February 2024, will test the new features in a replicated network and engineer the upgrade mechanism, the final stage before the Mina mainnet upgrade. Private testing with exchanges and custodians is already underway, and will continue to run throughout this track.

You can find more information, and keep track of upgrade related updates at the following links: 

X: @MinaProtocol

Discord: https://discord.com/invite/minaprotocol

Newsletter: https://minaprotocol.com/newsletter