Announcement,Community

Testworld Mission 2.0 Protocol Performance Testing — Program Details

Program details about Testworld Mission 2.0, an incentivized testnet to test zkApps and network resilience in preparation for a hard fork that will enable improved zkApp programmability on Mina Mainnet.

Program Details

Last Revised: October 9, 2023

The Testworld Mission 2.0: Protocol Performance Testing program is here. The goal of this program is to stress test the protocol and network with Mina community members to have a high level of confidence for Mina’s upcoming mainnet upgrade that will enable easier zkApps on Mina Mainnet.

The Program gathers experienced node operators to provide the network backbone for the Testworld 2.0 testnet. Participants will perform various node operation testing tasks for different grants. Participants can perform multiple node operation tasks.

For more details, please carefully read the rest of this document.

Contents

1- NODE OPERATOR RESPONSIBILITIES

2- TECHNICAL REQUIREMENTS

3- TESTWORLD 2.0 TEST PLAN

4- TIMELINES

5- INCENTIVES

6- FEEDBACK AND QUESTIONS

7- HOW TO APPLY

8- PROGRAM TERMS & CONDITIONS

1- Node Operators Responsibilities

Node operators participating in the Protocol Performance Testing program can perform one or more of the following tasks:

  1. Block Production
  2. Load Testing
  3. SNARK Work
  4. Archive Node

Block Production

HIGH LEVEL RESPONSIBILITIES

  • Run 2 Mina nodes on a cloud provider or hosted solution of your choice from the start until the end of the protocol performance testing
  • Run the latest Mina Node version (the Latest Mina Node version will be shared on Discord before the protocol performance testing starts) with all the required flags
  • Upgrade to a new Mina Node version within 24 hours if required
  • Ensure a high uptime % during the testing (a minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
  • The Mina Nodes should be configured to restart automatically on crash and to persist the configuration directory across restarts
  • The Block Producer is expected to raise any abnormal behaviour during the protocol performance testing on Github, using the labels “Testworld-2-0-protocol-performance-testing and “Testworld-2-0-block-producer
  • Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

CONFIGURATION SETUP:

  • The private keys will be shared before the protocol performance testing starts
  • The release notes with the latest software baseline and the configuration instructions will be provided to Block Producers in the dedicated Discord channel before the start of the program
  • The Block Producer runs 2 Mina nodes during the baseline load testing
  • The Block Producer node joins the P2P network using seeds for bootstrapping

LOGS COLLECTION:

  • The Block Producer will be notified about configuring Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
  • The Block Producer may be requested to send these logs (to be able to debug abnormal behaviour)

UPDATE PROCESS:

  • The Block Producer will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes 
  • The Block Producer is required to upgrade within 24 hours of the announcement during the testing

SECURITY:

  • The Performance Testing Toolset is organized as a separate GraphQL interface that is included only in testnet builds (not available for mainnet build) and is activated with certain CLI flags (—itn-graphql-port, —itn-keys) provided to the node.
  • CLI flag —itn-keys specifies Ed25519 public keys that are allowed access to the Control GraphQL interface. An authentication mechanism will ensure that only requests signed by specified public keys are allowed and protect against a range of attacks (such as request replay attacks). This way, it will ensure that access to the GraphQL Control interface of a node will be granted only to engineers from the Mina ecosystem facilitating the testnet. Control GraphQL provides a way to:
    • Send transactions from a Mina node to the network.
      • Secret keys for transaction sending will be provided by the caller (wallet or Block Producer keys associated with the Mina node will not be used)
    • Configure a Mina node’s networking
      • E.g. ban communication to certain peers in the Mina network
    • Access a Mina node’s information (such as slot allocation)

NOTES:

  • GraphQL Control will only use the capabilities of Mina nodes and only in a limited way. It grants the caller no access to the file system, firewall configuration or any other system configuration. 
  • Any changes made or actions launched to a Mina node via GraphQL Control will not persist over a Mina node’s restart. The GraphQL Control interface allows the execution of large-scale experiments involving hundreds of Mina nodes without coordination by node operators.
  • There will be random node restarts via the orchestrator once every 6 hours.

 Load Testing

HIGH LEVEL RESPONSIBILITIES

  • Run 10 Mina nodes on a cloud provider or hosted solution of your choice — you will be asked to start and stop the load testing node in Discord  channel.
  • Run the latest Mina Node version (the Latest Mina Node version will be shared on Discord)
  • Upgrade to a new Mina Node version within 24 hours if required
  • Ensure a high uptime % during the testing (a minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
  • The Mina Nodes should be configured to restart automatically on crash and to persist the configuration directory across restarts
  • The Load Tester expected to raise any abnormal behaviour during the protocol performance testing on Github, using the labels “Testworld-2-0-protocol-performance-testing and “Testworld-2-0-load-tester
  • Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

STRESS TESTING:

  • During the testing (on epochs 2 and 3), there will be stress tests of the network for 96h hours each time
  • The Load Testers will get a notification 24 hours in advance in the dedicated Discord channel
  • The Load Testing Block Producer spins up 10 extra nodes each during stress testing 
  • The 10 extra nodes must meet or exceed the minimum hardware requirements
  • The Load Tester runs the extra nodes in the same way as during the baseline load testing (same configuration, uptime, SW baseline, etc.) but without running consensus (no block production)

CONFIGURATION SETUP:

  • The release notes with the latest software baseline and the configuration instructions will be provided to Load Testers in the dedicated Discord channel before the start of the program
  • The Load Tester runs 10 non-consensus Mina nodes during the stress  testing windows, in addition to their 2 Block Producer nodes
  • The Load Tester spins down the 10 load testing nodes when asked to in Discord
  • The Load Testers nodes join the P2P network using seeds for bootstrapping

LOGS COLLECTION:

  • The Load Testers will be notified about configuring Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
  • The Load Testers may be requested to send these logs (to be able to debug abnormal behaviour)

UPDATE PROCESS:

  • The Load Testers will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes 
  • The Load Tester is required to upgrade within 24 hours of the announcement during the testing

SECURITY:

  • The Performance Testing Toolset is organized as a separate GraphQL interface that is included only in testnet builds (not available for mainnet build) and is activated with certain CLI flags (—itn-graphql-port, —itn-keys) provided to the node.
  • CLI flag —itn-keys specifies Ed25519 public keys that are allowed access to the Control GraphQL interface. An authentication mechanism will ensure that only requests signed by specified public keys are allowed, as well as protect against a range of attacks (such as request replay attacks). This way, it will ensure that access to the GraphQL Control interface of a node will be granted only to engineers from the Mina ecosystem facilitating the testnet. Control GraphQL provides a way to:
    • Send transactions from a Mina node to the network
      • Secret keys for transaction sending will be provided by the caller (wallet or Block Producer keys associated with the Mina node will not be used)
    • Configure a Mina node’s networking
      • E.g. ban communication to certain peers in Mina network
    • Access a Mina node’s information (such as slot allocation)
    • Control block production

NOTES:

  • GraphQL Control will only use capabilities of Mina nodes and only in a limited way. It grants the caller no access to file system, firewall configuration or any other system configuration. 
  • Any changes made or actions launched to a Mina node via GraphQL Control will not persist over a Mina node’s restart. The GraphQL Control interface allows the execution of large scale experiments involving hundreds of Mina nodes without coordination by node operators.
  • There will be random node restarts via the orchestrator once every 6 hours.

SNARK Work

HIGH LEVEL RESPONSIBILITIES

  • Run a pool of SNARK workers and 1 SNARK Coordinator on a cloud provider or hosted solution of your choice from the start until the end of the protocol performance testing 
  • Run the latest Mina Node version (the Latest Mina Node version will be shared on Discord, before the protocol performance testing starts) with all the required flags
  • Upgrade to a new Mina Node version within 24 hours if required
  • Ensure a high uptime % during the testing (a minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
  • The Mina Nodes should be configured to restart automatically on crash and to persist the configuration directory across restarts
  • The SNARK worker Operator is expected to raise any abnormal behaviour during the protocol performance testing on Github, using the labels “Testworld-2-0-protocol-performance-testing and “Testworld-2-0-snark-worker
  • Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

CONFIGURATION SETUP:

  • Each SNARK worker process should be allocated with 4 cores / 8 threads, bringing the number of SNARK worker processes to 4 per SNARK work server
  • All SNARK worker processes should be connected to one SNARK coordinator
  • Private keys will be shared before the protocol performance testing starts
  • The release notes with the latest software baseline and the configuration instructions will be provided to SNARK worker Operator in the dedicated Discord channel before the start of the program

LOGS COLLECTION:

  • The SNARK worker Operator will be notified about how to configure Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
  • The SNARK worker Operator may be requested to send these logs (to be able to debug abnormal behaviour)

UPDATE PROCESS:

  • The SNARK worker Operator will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes
  • The SNARK worker Operator is required to upgrade within 24 hours of the announcement during the testing

Archive Node

HIGH LEVEL RESPONSIBILITIES

  • Run the latest Mina Node and Archive Processor version (the Latest Mina Node and Archive Processor version will be shared on Discord, before the protocol performance testing starts)
  • Run a PostgreSQL database connected to the Archive Processor
  • Upgrade to a new Mina Node and Archive Processor version within 24 hours if required
  • Ensure a high uptime % during the testing (a minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
  • The Mina Node should be configured to restart automatically on crash and to persist the configuration directory across restarts
  • The Archive Node is expected to raise any abnormal behaviour during the protocol performance testing on Github, using the labels “Testworld-2-0-protocol-performance-testing and “Testworld-2-0-archive-node
  • Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

CONFIGURATION SETUP:

  • The release notes with the latest software baseline and the configuration instructions will be provided to Archive Node operators in the dedicated Discord channel before the start of the program
  • The Archive Node joins the P2P network using seeds for bootstrapping.

LOGS COLLECTION:

  • The Archive Node will be notified about how to configure Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
  • The Archive Node may be requested to send these logs (to be able to debug abnormal behaviour)

UPDATE PROCESS:

  • The Archive Node will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes 
  • The Archive Node is required to upgrade within 24 hours of the announcement during the testing

2- Technical Requirements

Node type Minimum hardware requirements
Block Production
  • 8 core processor
  • 16 GB of RAM
  • 10 GB of free storage
  • 1 Mbps internet connection
Load Testing
  • 8 core processor
  • 16 GB of RAM
  • 10 GB of free storage
  • 1 Mbps internet connection
SNARK Work
  • 16 core/32 threads dedicated instance
  • 16 GB of RAM
  • 12 GB of free storage
  • 1 Mbps internet connection

Note: multiple SNARK Worker processes should run on the same SNARK Work server. 4 core/8 threads should be provisioned per SNARK worker process, giving a total of 4 workers per SNARK Work server.

SNARK Coordinator
  • 8 core processor
  • 16 GB of RAM
  • 10 GB of free storage
  • 1 Mbps internet connection
Archive Node
  • 8 core processor
  • 16 GB of RAM
  • 10 GB of free storage
  • 1 Mbps internet connection

3- Testworld 2.0 test plan

Your role as node operators is paramount in ensuring the success of Testworld 2.0. Below are the revised instructions and test plan details for your reference.

  • Network Structure: The Testworld 2.0 network ledger mirrors the mainnet structure with approximately 200k accounts.
  • Participants: We have 250 community members managing various aspects of the network.

Epoch 1: Ensuring smooth onboarding

Goals:

  • Verify tooling functionality.
  • Resolve node setup and connection issues.
  • Conduct small experiments for validation.

Runbook:

Day 1:

  • Block Producers operators run 2 node replicas each with the same staking key to connect their nodes.
  • 10 SNARK work operators set up SNARK infrastructure with a total of 10 SNARK coordinators, 50 SNARK Worker servers and 200 SNARK worker processes (4 SNARK worker processes per server) with the following breakdown:
    • 2 operators will run 1 SNARK coordinator and 10 SNARK worker servers
    • 1 operator will run 1 SNARK coordinator and 7 SNARK worker servers
    • 3 operators will run 1 SNARK coordinator and 5 SNARK worker servers
    • 4 operators will run 1 SNARK coordinator and 2 SNARK worker servers

Day 5:

  • Verify all nodes are connected.

Day 10:

  • Run the transaction generator at half the goal throughput for 24 hours.

Epoch 2: Testing higher loads and scalability

Goals:

  • Test higher transaction loads.
  • Increase the number of nodes.
  • Test zkApps contracts from Mina builders.

Runbook:

Day 7:

  • zkApp builders deploy their contracts for testing.

Day 8:

  • Load-testing BPs spin up an additional 200 non-consensus nodes (10 nodes each).

Day 10:

  • Run transaction generator at goal throughput for 24 hours with increased node count.

Day 12:

  • Load-testing BPs spin down extra nodes.

Epoch 3: Testing higher loads and scalability with extra epoch complexity

Goals:

  • Test higher transaction loads with increased nodes.
  • Test load increases from being in the 3rd epoch.

Runbook:

Day 8:

  • Load-testing BPs increase the number of nodes on the network.

Day 10:

  • Run transaction generator at goal throughput for 24 hours with increased node count.

Day 12:

  • Load-testing BPs spin down extra nodes.

REPORTING OF BUGS

All participants are expected to raise any abnormal behaviour during the protocol performance testing on Github, using the label «Testworld-2-0-protocol-performance-testing«

ESCALATING OTHER ISSUES

In dedicated Discord channel #testworld-2

QUESTIONS

In dedicated Discord channel #testworld-2

4- Timelines

The total duration of the protocol performance testing will last for about 3 epochs with a tentative start date of October 17, 2023 (we will notify successful applicants 1 week before the testnet launch to provide adequate time for renting servers). The Mina Foundation and its ecosystem partners will be conducting internal testing prior to the incentivized testnet start. It’s possible the October 17 start could move in the event of any security or stability related issues. The community will be informed with as much notice as possible if that occurs.

If testers uncover critical bugs and issues on the network, it may be necessary to pause the testing, fix the issues, and then restart the testing process. In that case, participants will be requested to pause the testing or participate for an extended testnet duration.

5- Incentives

We detail the incentives per category of participant. Please note that the incentives cover a period of 2 months of testing. Should the testing last longer, the incentives may be adapted to cover the operational costs (servers).

Node type

Grant and payout schedule

Block Production Grant: 850 USDC and 1,000 MINA tokens per Node Operator.

Payout schedule:

  • 425 USDC after confirmation of active participation on the testnet based on an uptime snapshot to determine nodes are online, which will be made around Oct 24, 2023. Payments are expected to be completed within 1 week after the snapshot.
  • 425 USDC after confirmation at the end of the testnet that the Protocol Performance Testing T&C have been met
  • 1,000 MINA tokens will be unlocked one year after Block Production’s successful completion of the testnet
  • Participants must fulfill KYC/AML requirements and remain in good-standing status during the testing to receive payments noted herein
Load Testing Grant: 1,600 USDC and 4,000 MINA tokens per Node Operator.

Payout schedule:

  • 800 USDC after confirmation of active participation on the testnet based on an uptime snapshot, to determine nodes are online, which will be made around Oct 24, 2023. Payments are expected to be completed within 1 week after the snapshot.
  • 800 USDC after confirmation at the end of the testnet that the Protocol Performance Testing T&C have been met
  • 4,000 MINA tokens will be unlocked one year after Load Testing’s successful  completion of the testnet
  • Participants must fulfill KYC/AML requirements and remain in good-standing status during the testing to receive payments noted herein
SNARK Work Grant: 

  1. 400 USDC for the single SNARK Work coordinator
  2. 600 USDC and 200 MINA tokens per SNARK worker

Note: there will be 50 SNARK workers in total, spread over 10 Node Operators who will additionally run 1 SNARK Coordinator each. The number of SNARK workers per Node Operator will vary.

Payout schedule:

  • 300 USDC per SNARK worker run + 200 USDC for the SNARK Work coordinator after confirmation of fulfilling KYC/AML requirements. The intention is for the grant to be disbursed before the launch of the testnet
  • 300 USDC per SNARK worker run + 200 USDC for the SNARK Work coordinator is intended to be disbursed prior to the second month of the testnet
  • 200 MINA tokens per SNARK worker run will be unlocked one year after SNARK Work’s successful completion of the testnet

Participants must fulfill KYC/AML requirements and remain in good-standing status during the testing to receive payments noted herein

Archive Node Grant : 500 USDC and 1,000 MINA tokens per Node Operator.

Payout schedule:

  • 250 USDC after confirmation of fulfilling KYC/AML requirements. The intention is for the grant to be disbursed before the launch of the testnet.
  • 250 USDC after confirmation at the end of the testnet that the Protocol Performance Testing T&C have been met.
  • 1,000 MINA tokens will be unlocked one year after Archive Node’s successful  completion of the testnet.
  • Participants must fulfill KYC/AML requirements and remain in good-standing status during the testing to receive payments noted herein

Please note:

If applicable, the Program might be paused or its duration might be extended upon participants uncovering critical bugs or issues on the network. Participants might be required to pause or extend their server rental in such circumstances.

6- Feedback and Questions

We have set up a dedicated Discord #testworld-2, where you can provide feedback and ask questions related to the Testworld Mission 2.0: Protocol Performance Testing program. We’d be happy to hear from you!

7- How to Apply

A big thank you to all who applied for Testworld 2.0! We’re thrilled by the enthusiasm and want to let you know that applications are now officially closed. 

8- Program Terms & Conditions

Please see the Program Terms & Conditions here.

 

More from our Blog

SEE ALL POSTS
Retro / 2024-03-21 / Vitor Silva
Upgrade Mechanism Testing Retrospective
Track 3 allowed for the testing of various loads and helped uncover issues which have since been resolved. As a result, an optimal configuration was identified, and the release candidate for the Mainnet Upgrade is ready.
Read more
Learn / 2024-03-15 / Will Cove
Introducing ‘httpz’: the internet you can trust
Read more
Community, Events / 2024-03-13 / Mina Protocol
BUIDL with Mina Protocol at ETH Seoul 2024
Read more
Announcement / 2024-03-12 / Mina Foundation
zkIgnite, Cohort 3 Funded Projects
Read more

О технологии

AboutTechCta

Mina использует передовую криптографию и рекурсивные zk-SNARKs для обеспечения полной децентрализации при масштабировании.

Начните работу

GetStartedCta

С Mina вы сможете легко подключить ноду, присоединиться к сообществу и участвовать в его развитии.