Testworld Mission 2.0 Protocol Performance Testing – Program Details

Program Details

Last Revised: October 9, 2023

The Testworld Mission 2.0: Protocol Performance Testing program is here. The goal of this program is to stress test the protocol and network with Mina community members to have a high level of confidence for Mina’s upcoming mainnet upgrade that will enable easier zkApps on Mina Mainnet.

The Program gathers experienced node operators to provide the network backbone for the Testworld 2.0 testnet. Participants will perform various node operation testing tasks for different grants. Participants can perform multiple node operation tasks.

For more details, please carefully read the rest of this document.

Contents

1- NODE OPERATOR RESPONSIBILITIES

2- TECHNICAL REQUIREMENTS

3- TESTWORLD 2.0 TEST PLAN

4- TIMELINES

5- INCENTIVES

6- FEEDBACK AND QUESTIONS

7- HOW TO APPLY

8- PROGRAM TERMS & CONDITIONS

1- Node Operators Responsibilities

Node operators participating in the Protocol Performance Testing program can perform one or more of the following tasks:

1. Block Production
2. Load Testing
3. SNARK Work
4. Archive Node

Block Production

HIGH LEVEL RESPONSIBILITIES

• Run 2 Mina nodes on a cloud provider or hosted solution of your choice from the start until the end of the protocol performance testing
• Run the latest Mina Node version (the Latest Mina Node version will be shared on Discord before the protocol performance testing starts) with all the required flags
• Upgrade to a new Mina Node version within 24 hours if required
• Ensure a high uptime % during the testing (a minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
• The Mina Nodes should be configured to restart automatically on crash and to persist the configuration directory across restarts
• The Block Producer is expected to raise any abnormal behaviour during the protocol performance testing on Github, using the labels “Testworld-2-0-protocol-performance-testing” and “Testworld-2-0-block-producer”
• Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

CONFIGURATION SETUP:

• The private keys will be shared before the protocol performance testing starts
• The release notes with the latest software baseline and the configuration instructions will be provided to Block Producers in the dedicated Discord channel before the start of the program
• The Block Producer runs 2 Mina nodes during the baseline load testing
• The Block Producer node joins the P2P network using seeds for bootstrapping

LOGS COLLECTION:

• The Block Producer will be notified about configuring Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
• The Block Producer may be requested to send these logs (to be able to debug abnormal behaviour)

UPDATE PROCESS:

• The Block Producer will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes 
• The Block Producer is required to upgrade within 24 hours of the announcement during the testing

SECURITY:

• The Performance Testing Toolset is organized as a separate GraphQL interface that is included only in testnet builds (not available for mainnet build) and is activated with certain CLI flags (–itn-graphql-port, –itn-keys) provided to the node.
• CLI flag –itn-keys specifies Ed25519 public keys that are allowed access to the Control GraphQL interface. An authentication mechanism will ensure that only requests signed by specified public keys are allowed and protect against a range of attacks (such as request replay attacks). This way, it will ensure that access to the GraphQL Control interface of a node will be granted only to engineers from the Mina ecosystem facilitating the testnet. Control GraphQL provides a way to:
  • Send transactions from a Mina node to the network.
    • Secret keys for transaction sending will be provided by the caller (wallet or Block Producer keys associated with the Mina node will not be used)
  • Configure a Mina node’s networking
    • E.g. ban communication to certain peers in the Mina network
  • Access a Mina node’s information (such as slot allocation)

NOTES:

• GraphQL Control will only use the capabilities of Mina nodes and only in a limited way. It grants the caller no access to the file system, firewall configuration or any other system configuration. 
• Any changes made or actions launched to a Mina node via GraphQL Control will not persist over a Mina node’s restart. The GraphQL Control interface allows the execution of large-scale experiments involving hundreds of Mina nodes without coordination by node operators.
• There will be random node restarts via the orchestrator once every 6 hours.

 Load Testing

HIGH LEVEL RESPONSIBILITIES

• Run 10 Mina nodes on a cloud provider or hosted solution of your choice – you will be asked to start and stop the load testing node in Discord  channel.
• Run the latest Mina Node version (the Latest Mina Node version will be shared on Discord)
• Upgrade to a new Mina Node version within 24 hours if required
• Ensure a high uptime % during the testing (a minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
• The Mina Nodes should be configured to restart automatically on crash and to persist the configuration directory across restarts
• The Load Tester expected to raise any abnormal behaviour during the protocol performance testing on Github, using the labels “Testworld-2-0-protocol-performance-testing” and “Testworld-2-0-load-tester”
• Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

STRESS TESTING:

• During the testing (on epochs 2 and 3), there will be stress tests of the network for 96h hours each time
• The Load Testers will get a notification 24 hours in advance in the dedicated Discord channel
• The Load Testing Block Producer spins up 10 extra nodes each during stress testing 
• The 10 extra nodes must meet or exceed the minimum hardware requirements
• The Load Tester runs the extra nodes in the same way as during the baseline load testing (same configuration, uptime, SW baseline, etc.) but without running consensus (no block production)

CONFIGURATION SETUP:

• The release notes with the latest software baseline and the configuration instructions will be provided to Load Testers in the dedicated Discord channel before the start of the program
• The Load Tester runs 10 non-consensus Mina nodes during the stress  testing windows, in addition to their 2 Block Producer nodes
• The Load Tester spins down the 10 load testing nodes when asked to in Discord
• The Load Testers nodes join the P2P network using seeds for bootstrapping

LOGS COLLECTION:

• The Load Testers will be notified about configuring Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
• The Load Testers may be requested to send these logs (to be able to debug abnormal behaviour)

UPDATE PROCESS:

• The Load Testers will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes 
• The Load Tester is required to upgrade within 24 hours of the announcement during the testing

SECURITY:

• The Performance Testing Toolset is organized as a separate GraphQL interface that is included only in testnet builds (not available for mainnet build) and is activated with certain CLI flags (–itn-graphql-port, –itn-keys) provided to the node.
• CLI flag –itn-keys specifies Ed25519 public keys that are allowed access to the Control GraphQL interface. An authentication mechanism will ensure that only requests signed by specified public keys are allowed, as well as protect against a range of attacks (such as request replay attacks). This way, it will ensure that access to the GraphQL Control interface of a node will be granted only to engineers from the Mina ecosystem facilitating the testnet. Control GraphQL provides a way to:
  • Send transactions from a Mina node to the network
    • Secret keys for transaction sending will be provided by the caller (wallet or Block Producer keys associated with the Mina node will not be used)
  • Configure a Mina node’s networking
    • E.g. ban communication to certain peers in Mina network
  • Access a Mina node’s information (such as slot allocation)
  • Control block production

NOTES:

• GraphQL Control will only use capabilities of Mina nodes and only in a limited way. It grants the caller no access to file system, firewall configuration or any other system configuration. 
• Any changes made or actions launched to a Mina node via GraphQL Control will not persist over a Mina node’s restart. The GraphQL Control interface allows the execution of large scale experiments involving hundreds of Mina nodes without coordination by node operators.
• There will be random node restarts via the orchestrator once every 6 hours.

SNARK Work

HIGH LEVEL RESPONSIBILITIES

• Run a pool of SNARK workers and 1 SNARK Coordinator on a cloud provider or hosted solution of your choice from the start until the end of the protocol performance testing 
• Run the latest Mina Node version (the Latest Mina Node version will be shared on Discord, before the protocol performance testing starts) with all the required flags
• Upgrade to a new Mina Node version within 24 hours if required
• Ensure a high uptime % during the testing (a minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
• The Mina Nodes should be configured to restart automatically on crash and to persist the configuration directory across restarts
• The SNARK worker Operator is expected to raise any abnormal behaviour during the protocol performance testing on Github, using the labels “Testworld-2-0-protocol-performance-testing” and “Testworld-2-0-snark-worker”
• Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

CONFIGURATION SETUP:

• Each SNARK worker process should be allocated with 4 cores / 8 threads, bringing the number of SNARK worker processes to 4 per SNARK work server
• All SNARK worker processes should be connected to one SNARK coordinator
• Private keys will be shared before the protocol performance testing starts
• The release notes with the latest software baseline and the configuration instructions will be provided to SNARK worker Operator in the dedicated Discord channel before the start of the program

LOGS COLLECTION:

• The SNARK worker Operator will be notified about how to configure Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
• The SNARK worker Operator may be requested to send these logs (to be able to debug abnormal behaviour)

UPDATE PROCESS:

• The SNARK worker Operator will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes
• The SNARK worker Operator is required to upgrade within 24 hours of the announcement during the testing

Archive Node

HIGH LEVEL RESPONSIBILITIES

• Run the latest Mina Node and Archive Processor version (the Latest Mina Node and Archive Processor version will be shared on Discord, before the protocol performance testing starts)
• Run a PostgreSQL database connected to the Archive Processor
• Upgrade to a new Mina Node and Archive Processor version within 24 hours if required
• Ensure a high uptime % during the testing (a minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
• The Mina Node should be configured to restart automatically on crash and to persist the configuration directory across restarts
• The Archive Node is expected to raise any abnormal behaviour during the protocol performance testing on Github, using the labels “Testworld-2-0-protocol-performance-testing” and “Testworld-2-0-archive-node”
• Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

CONFIGURATION SETUP:

• The release notes with the latest software baseline and the configuration instructions will be provided to Archive Node operators in the dedicated Discord channel before the start of the program
• The Archive Node joins the P2P network using seeds for bootstrapping.

LOGS COLLECTION:

• The Archive Node will be notified about how to configure Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
• The Archive Node may be requested to send these logs (to be able to debug abnormal behaviour)

UPDATE PROCESS:

• The Archive Node will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes 
• The Archive Node is required to upgrade within 24 hours of the announcement during the testing

2- Technical Requirements

3- Testworld 2.0 test plan

Your role as node operators is paramount in ensuring the success of Testworld 2.0. Below are the revised instructions and test plan details for your reference.

• Network Structure: The Testworld 2.0 network ledger mirrors the mainnet structure with approximately 200k accounts.
• Participants: We have 250 community members managing various aspects of the network.

REPORTING OF BUGS

All participants are expected to raise any abnormal behaviour during the protocol performance testing on Github, using the label “Testworld-2-0-protocol-performance-testing“

ESCALATING OTHER ISSUES

In dedicated Discord channel #testworld-2

QUESTIONS

In dedicated Discord channel #testworld-2

4- Timelines

The total duration of the protocol performance testing will last for about 3 epochs with a tentative start date of October 17, 2023 (we will notify successful applicants 1 week before the testnet launch to provide adequate time for renting servers). The Mina Foundation and its ecosystem partners will be conducting internal testing prior to the incentivized testnet start. It’s possible the October 17 start could move in the event of any security or stability related issues. The community will be informed with as much notice as possible if that occurs.

If testers uncover critical bugs and issues on the network, it may be necessary to pause the testing, fix the issues, and then restart the testing process. In that case, participants will be requested to pause the testing or participate for an extended testnet duration.

5- Incentives

We detail the incentives per category of participant. Please note that the incentives cover a period of 2 months of testing. Should the testing last longer, the incentives may be adapted to cover the operational costs (servers).

Please note:

If applicable, the Program might be paused or its duration might be extended upon participants uncovering critical bugs or issues on the network. Participants might be required to pause or extend their server rental in such circumstances.

6- Feedback and Questions

We have set up a dedicated Discord #testworld-2, where you can provide feedback and ask questions related to the Testworld Mission 2.0: Protocol Performance Testing program. We’d be happy to hear from you!

7- How to Apply

A big thank you to all who applied for Testworld 2.0! We’re thrilled by the enthusiasm and want to let you know that applications are now officially closed. 

8- Program Terms & Conditions

Please see the Program Terms & Conditions here.