Testworld Mission 2.0 Protocol Performance Testing – Program Details

Program Details

The Testworld Mission 2.0: Protocol Performance Testing program is here. The goal of this program is to stress test the protocol and network with Mina community members, in order to have a high level of confidence for Mina’s upcoming hard fork that will further enable zkApps on Mina Mainnet.

The Program gathers experienced node operators to provide the network backbone for the Testworld 2.0 testnet. Participants are invited to apply to perform a variety of node operation testing tasks, for different grants. It is possible to apply and be selected for multiple node operation tasks.

For more details, please carefully read the rest of this document.

Contents

1- Node Operator Responsibilities

2- Technical Requirements

3- Timelines

4- Incentives

5- Feedback and Questions

6- How to Apply

7- Program Terms & Conditions

Node Operators Responsibilities

Node operators participating in the Protocol Performance Testing program can perform one or more of the following tasks:

1. Block Production
2. Load Testing
3. SNARK Work
4. Archive Node

Block Production

High level responsibilities

• Run 2 Mina nodes on a cloud provider or hosted solution of your choice from the start until the end of the protocol performance testing
• Run the latest Mina Node version (Latest Mina Node version will be shared on Discord, before the protocol performance testing starts)
• Maintain at least 72 consecutive hours of logs at any time (set storage limit to 72 hours of logs) 
• Upgrade to a new Mina Node version within 24 hours if required
• Ensure high uptime % during the testing (minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
• The Block Producer is expected to raise any abnormal behavior during the protocol performance testing on Github, using the label ITN- track 3- protocol performance testing
• Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts 

Performance Testing Toolset includes, amongst others:

• Graphql experimental control
• Libp2p port 
• Prometheus port 
• Uptime data 
• Node status collection 
• Tracing and monitoring service 

Configuration setup:

• The private keys will be shared before the protocol performance testing starts
• The release notes with the latest software baseline and the configuration instructions will be provided to Block Producers in the dedicated Discord channel before the start of the program
• The Block Producer runs 2 Mina nodes during the baseline load testing
• The Block Producer node joins the P2P network using seeds for bootstrapping 

Logs collection: 

• The Block Producer will be notified about how to configure Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
• The Block Producer may be requested to send these logs (to be able to debug abnormal behavior)

Update process:

• The Block Producer will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes 
• The Block Producer is required to upgrade within 24 hours of the announcement during the testing

Security:

• The Performance Testing Toolset is organized as a separate GraphQL interface that is included only into testnet builds (not available for mainnet build) and is activated with certain CLI flags (–itn-graphql-port, –itn-keys) provided to the node.
• CLI flag –itn-keys specifies Ed25519 public keys that are allowed access to the Control GraphQL interface. An authentication mechanism will ensure that only requests signed by specified public keys are allowed, as well as protecting against a range of attacks (such as request replay attack). This way it will ensure that access to the GraphQL Control interface of a node will be granted only to engineers from the Mina ecosystem facilitating the testnet. Control GraphQL provides a way to:

• • Send transactions from a Mina node to the network
    • Secret keys for transaction sending will be provided by the caller (wallet or Block Producer keys associated with the Mina node will not be used)
  • Configure a Mina node’s networking
    • E.g. ban communication to certain peers in Mina network
  • Access a Mina node’s information (such as slot allocation)
  • Control block production

Notes:

• GraphQL Control will only use capabilities of Mina nodes and only in a limited way. It grants the caller no access to file system, firewall configuration or any other system configuration. 
• Any changes made or actions launched to a Mina node via GraphQL Control will not persist over a Mina node’s restart. The GraphQL Control interface allows the execution of large scale experiments involving hundreds of Mina nodes without coordination by node operators

Load Testing

High level responsibilities

• Run 2 Mina nodes on a cloud provider or hosted solution of your choice from the start until the end of the protocol performance testing
• Run the latest Mina Node version (Latest Mina Node version will be shared on Discord, before the protocol performance testing starts)
• Maintain at least 72 consecutive hours of logs at any time (set storage limit to 72 hours of logs) 
• Upgrade to a new Mina Node version within 24 hours if required
• Ensure high uptime % during the testing (minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
• The Block Producer is expected to raise any abnormal behavior during the protocol performance testing on Github, using the label ITN- track 3- protocol performance testing
• Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

Performance Testing Toolset includes, amongst others:

• Graphql experimental control
• Libp2p port 
• Prometheus port 
• Uptime data 
• Node status collection 
• Tracing and monitoring service 

Stress testing:

• During the testing (roughly once every 2 weeks), there will be a stress test of the network for 48h hours each time
• The Block Producer will get a notification 72 hours in advance in the dedicated Discord channel
• The Load Testing Block Producer spins up 10 extra nodes each during stress testing 
• The 10 extra nodes must meet or exceed the minimum hardware requirements
• The Block Producer runs the extra nodes in the same way as during the baseline load testing (same configuration, uptime, SW baseline, etc.)

Configuration setup:

• The private keys will be shared before the protocol performance testing starts
• The release notes with the latest software baseline and the configuration instructions will be provided to Block Producers in the dedicated Discord channel before the start of the program
• The Block Producer runs 2 Mina nodes during the baseline load testing
• The Block Producer node joins the P2P network using seeds for bootstrapping 

Logs collection: 

• The Block Producer will be notified about how to configure Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
• The Block Producer may be requested to send these logs (to be able to debug abnormal behavior)

Update process:

• The Block Producer will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes 
• The Block Producer is required to upgrade within 24 hours of the announcement during the testing

Security:

• The Performance Testing Toolset is organized as a separate GraphQL interface that is included only into testnet builds (not available for mainnet build) and is activated with certain CLI flags (–itn-graphql-port, –itn-keys) provided to the node.
• CLI flag –itn-keys specifies Ed25519 public keys that are allowed access to the Control GraphQL interface. An authentication mechanism will ensure that only requests signed by specified public keys are allowed, as well as protecting against a range of attacks (such as request replay attack). This way it will ensure that access to the GraphQL Control interface of a node will be granted only to engineers from the Mina ecosystem facilitating the testnet. Control GraphQL provides a way to:

• • Send transactions from a Mina node to the network
    • Secret keys for transaction sending will be provided by the caller (wallet or Block Producer keys associated with the Mina node will not be used)
  • Configure a Mina node’s networking
    • E.g. ban communication to certain peers in Mina network
  • Access a Mina node’s information (such as slot allocation)
  • Control block production

Notes:

• GraphQL Control will only use capabilities of Mina nodes and only in a limited way. It grants the caller no access to file system, firewall configuration or any other system configuration. 
• Any changes made or actions launched to a Mina node via GraphQL Control will not persist over a Mina node’s restart. The GraphQL Control interface allows the execution of large scale experiments involving hundreds of Mina nodes without coordination by node operators.

SNARK Work

High level responsibilities

• Run a pool of snarkworkers and 1 SNARK Coordinator on a cloud provider or hosted solution of your choice from the start until the end of the protocol performance testing 
• Run the latest Mina Node version (Latest Mina Node version will be shared on Discord, before the protocol performance testing starts)
• Maintain at least 72 consecutive hours of logs at any time (set storage limit to 72 hours of logs) 
• Upgrade to a new Mina Node version within 24 hours if required
• Ensure high uptime % during the testing (minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
• The Snarkworker Operator is expected to raise any abnormal behavior during the protocol performance testing on Github, using the label ITN- track 3- protocol performance testing
• Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

Performance Testing Toolset includes, amongst others:

• Libp2p port 
• Prometheus port 
• Uptime data 
• Node status collection 
• Tracing and monitoring service

Configuration setup:  

• Private keys will be shared before the protocol performance testing starts
• The release notes with the latest software baseline and the configuration instructions will be provided to Snarkworker Operator in the dedicated Discord channel before the start of the program

Logs collection:

• The Snarkworker Operator will be notified about how to configure Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
• The Snarkworker Operator may be requested to send these logs (to be able to debug abnormal behavior)

Update process:

• The Snarkworker Operator will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes
• The Snarkworker Operator is required to upgrade within 24 hours of the announcement during the testing

Archive Node

High level responsibilities

• Run the latest Mina Node version (Latest Mina Node version will be shared on Discord, before the protocol performance testing starts)
• Maintain at least 72 consecutive hours of logs at any time (set storage limit to 72 hours of logs) 
• Upgrade to a new Mina Node version within 24 hours if required
• Ensure high uptime % during the testing (minimum of 90% uptime is required) as monitored by the snark-work-based uptime system
• The Archive Node is expected to raise any abnormal behavior during the protocol performance testing on Github, using the label ITN- track 3- protocol performance testing
• Configure the Mina Node correctly – configuration instructions will be shared before the protocol performance testing starts

Performance Testing Toolset includes, amongst others:

• Libp2p port 
• Prometheus port 
• Uptime data 
• Node status collection 
• Tracing and monitoring service 

Configuration setup:  

• The release notes with the latest software baseline and the configuration instructions will be provided to Block Producers in the dedicated Discord channel before the start of the program
• The Archive Node joins the P2P network using seeds for bootstrapping. 

Logs collection: 

• The Archive Node will be notified about how to configure Mina nodes for logging. Instructions about how to send the logs will be shared before the start of the program
• The Archive Node may be requested to send these logs (to be able to debug abnormal behavior)

Update process:

• The Archive Node will get a notification in advance in the dedicated Discord channel as to when a new version of the Mina Node will be available along with the Release Notes 
• The Archive Node is required to upgrade within 24 hours of the announcement during the testing

Technical Requirements

Block Production

Minimum hardware requirements of an 8-core processor, 16GB RAM and 10GB storage with at least 1 Mbps connection.

Load Testing

Minimum Hardware requirements of an 8-core processor, 16GB RAM and 10GB storage with at least 1 Mbps connection.

SNARK Work

Minimum hardware requirements of a 16 core/32 thread dedicated instance, 16GB RAM and 10GB storage with at least 1 Mbps connection.

Archive Node

Minimum hardware requirements of an 8-core processor, 16GB RAM and 10GB storage with at least 1 Mbps connection.

Timelines

The total duration of the protocol performance testing will last for about 2 months with a tentative start date of 26 April (we will notify successful applicants 1 week before the testnet launch to provide adequate time for renting servers). The Mina Foundation and its ecosystem partners will be conducting internal testing prior to the incentivized testnet start. It’s possible the 26 April start could move in the event of any security or stability related issues. The community will be informed with as much notice as possible if that occurs.

If testers uncover critical bugs and issues on the network, it may be necessary to pause the testing, fix the issues, and then restart the testing process. In that case, participants will be requested to pause the testing or participate for an extended testnet duration.

Reporting of bugs

All participants are  expected to raise any abnormal behavior during the protocol performance testing on Github, using the label ITN- track 3- protocol performance testing

Escalating other issues

In dedicated Discord channel #protocol-performance-testing

Questions

In dedicated Discord channel #protocol-performance-testing

Incentives

We detail the incentives per category of participant. Please note that the incentives cover a period of 2 months of testing. Should the testing last longer, the incentives will be adapted to cover the operational costs (servers).

Block Production

Grant: 850 USDC and 1,000 MINA tokens per Node Operator. 

Payout schedule:

• 425 USDC after confirmation of active participation on the testnet based on an uptime snapshot to determine nodes are online, that will be made around May 1, 2023. Payments are expected to be completed within 1 week after the snapshot.
• 425 USDC after confirmation at the end of the testnet that the Protocol Performance Testing T&C have been met 
• 1,000 MINA tokens will be unlocked one year after the completion of the testnet
• Participants must fulfill KYC/AML requirements to receive payments noted herein

Load Testing

Grant: 1,200 USDC and 3,000 MINA tokens per Node Operator. 

Payout schedule:

• 600 USDC after confirmation of active participation on the testnet based on an uptime snapshot, to determine nodes are online, that will be made around May 1, 2023. Payments are expected to be completed within 1 week after the snapshot.
• 600 USDC after confirmation the at the end of the testnet that the Protocol Performance Testing T&C have been met 
• 3,000 MINA tokens will be unlocked one year after the completion of the testnet
• Participants must fulfill KYC/AML requirements to receive payments noted herein

SNARK Work

Grant: 600 USDC and 200 MINA tokens per SNARK worker. 

Note: there will be 50 SNARK workers in total, spread over 10 Node Operators who will additionally run 1 Snark Coordinator each. The number of snarkworkers per Node Operator will vary.

Payout schedule:

• 300 USDC per SNARK worker run, after confirmation of fulfilling KYC/AML requirements. The intention is for the grant to be disbursed before the launch of the testnet
• 300 USDC per SNARK worker run is intended to be disbursed prior to the second month of the testnet
• 200 MINA tokens per SNARK worker run will be unlocked one year after the completion of the testnet
• Participants must fulfill KYC/AML requirements to receive payments noted herein

Archive Node

Grant : 500 USDC and 1,000 MINA tokens per Node Operator. 

Payout schedule:

• 250 USDC after confirmation of fulfilling KYC/AML requirements. The intention is for the grant to be disbursed before the launch of the testnet.
• 250 USDC after confirmation at the end of the testnet that the Protocol Performance Testing T&C have been met.
• 1,000 MINA tokens will be unlocked one year after the completion of the testnet.
• Participants must fulfill KYC/AML requirements to receive payments noted herein

Please note:

If applicable, the Program might be paused or its duration might be extended upon participants uncovering critical bugs or issues on the network. Participants might be required to pause or extend their server rental in such circumstances.

Feedback and Questions

We have set up a dedicated Discord channel, #protocol-performance-testing, where you can provide feedback and ask questions related to the Testworld Mission 2.0: Protocol Performance Testing program. We’d be happy to hear from you!

How to Apply

Please submit your applications through this form ASAP. The deadline for submissions is 18 April, 2023 23:59 UTC. It is important that you begin the application and KYC/AML process, described below, immediately.

KYC/AML Onboarding Process

In order to receive USDC and MINA grants, applicants must successfully complete the KYC/AML onboarding process. Please start this process immediately.

• Non-FastPass holders (find information about the FastPass system here) should complete the KYC/AML checks through Synaps, here, or CoinList, here, ASAP.
• FastPass holders must complete the acknowledgement form, here, ASAP.

After the selection process is completed, successful applicants will be contacted by email with further steps in the KYC/AML onboarding process.

Program Terms & Conditions

Please see the Program Terms & Conditions here.