Emergency Hardfork Test Details (APPENDIX)

Actor

What they run 

What they care about

Block Producer (BP)

Mina daemon with block-producer keys

Producing on the right branch post-fork; not double-producing across the rollback boundary; new coinbase / slot duration

zkApp Developer

o1js app

That their submitted transactions either land or are cleanly rolled back; that o1js bindings + verification key still validate post-fork

Snark worker operators

Mina internal snark-worker against a coordinator/daemon, or a standalone worker pool

Not wasting compute on work that will be rolled back; fees they earned actually surviving the rollback; their worker is using the right proving/verification keys post-fork; their bundles are being accepted into the new SNARK pool

Archive / Rosetta operator

Archive node + Rosetta + Daemon on top of Postgres Data

DB integrity across the rollback; replayer convergence; Rosetta serving the right network

STAGE 1: Mesa-based Test Network
(After successful mesa fork Emergency Hardfork Day, morning UTC)

Goals:

• Confirm healthy steady state on the pre-emergency fork (Mesa-era).

Runbook:

1. Block Producers 
   1. Keep nodes synced 
   2. Verify your producer key is delegating/staking correctly and that you are seeing your own produced blocks land (not just on best-tip).
2. Snark Workers Operators
   1. Confirm worker is connected to a synced daemon and successfully completing bundles 
   2. Capture your current snark-fee account balance and a small sample of recently completed-work bundle hashes — you’ll diff fees post-fork to confirm nothing landed-then-rolled-back silently.
3. zkApp developers
   1. Capture the deployed account addresses and verification key hashes — you’ll diff them post-fork.
4. Archive / Rosetta Operators
   1. Ensure no missing blocks

mina-missing-blocks-auditor --archive-uri "$PG_URI"    

# or as daemon

DB_USERNAME=.. \
PGPASSWORD=... \
DB_HOST=... \
DB_PORT=... \    
DB_NAME=.. \
MINA_NETWORK=... \  
PRECOMPUTED_BLOCKS_URL=.... \ TIMEOUT=600 \                 
mina-missing-blocks-guardian daemon

                  2. Ensure you are connected to correct network

# ROSETTA_URL env var points to base rosetta url (e.g. http://localhost:3087)

curl -s -X POST $ROSETTA_URL/network/list -H 'Content-Type: application/json' -d '{}' | jq .

# Expect: { "network_identifiers":[{"blockchain":"mina","network":"testnet"}] }     
                                                                                    
curl -s -X POST $ROSETTA_URL/network/status -H 'Content-Type: application/json' \  
    -d "{\"network_identifier\":{\"blockchain\":\"mina\",\"network\":\"testnet\"}}" | jq '.current_block_identifier, .genesis_block_identifier'

Find more detailed information in the docs and reach out in Discord if you experience any issues.

STAGE 2: EMERGENCY HF DRILL

Step 1 – Fork point announcement (Emergency Hardfork Day, ~ noon UTC)

Goals:

•  o1Labs simulates an unplanned halt and announces a fork-point block out-of-band on #mesa-upgrade-testnet-announcements 
  

FORK_HEIGHT=302442
FORK_STATE_HASH=3NKfDPqtvJyVLVYn2eabjxXspH336eqkjHDkPus5WzbJ8iczckKs
FORK_GLOBAL_SLOT=460817      # global_slot_since_genesis (slotSinceGenesis)
LATEST_STATE_HASH=3NKj4ir6iRTRmxnrbu92VkUCqnpJ2waatiTAEraZs97iJ7mAFLPz  #height 302455 - used for confirmation depth check
REQUIRED_CONFIRMATIONS=19  # k
MESA_GENESIS_TIMESTAMP=2026-06-017T14:00:00Z

The block must be pending (< `REQUIRED_CONFIRMATIONS` deep)  at announcement time — o1Labs only announces a fork point once it is confirmed in their archive  by more than 15 blocks.  BPs, zkApp devs, Archive ops –  must align on this tuple.

Runbook:

1. 1. Block Producers

• Stop producing-  As soon as the announcement lands, halt the BP role on the daemon (e.g. remove the block-producer key or stop the daemon) — you must not extend any branch above FORK_HEIGHT.

1. 1. Inventory blocks you produced since FORK_HEIGHT  so you can sanity-check coinbases lost in the rollback. Optional but useful for the post-mortem.
2. Snark Worker Operators
   1. Stop submitting new work. Halt the worker process (or set the coordinator’s snark-fee high enough that nothing is accepted). Any bundle you complete and submit above FORK_HEIGHT will be discarded with the rollback, and the fee will not survive — you’re spending CPU for nothing.
   2. Inventory in-flight bundles (work IDs / state hashes you submitted since the fork-point slot) so you can verify post-fork that the corresponding fees were either rolled back cleanly or are absent from your balance — useful for the post-mortem.                                                                                                                                                          
   3. Pause any auto-restart / supervisor that would relaunch the worker before the Mesa rollout.
3. zkApp developers
   1. Inventory your in-flight transactions  (deploys, method calls, account updates) submitted since the fork-point slot. Anything in blocks above FORK_HEIGHT, or in the mempool, will be discarded — record it for resubmission post-fork.
4. Archive/Rosetta Operator
   1. Verify the fork-point block is in your archive  using the toolbox (no raw SQL needed):
      

   mina-archive-hardfork-toolbox fork-candidate is-in-best-chain \
     --postgres-uri "$PG_URI" \
     --fork-state-hash "$FORK_STATE_HASH" \
     --fork-height   "$FORK_HEIGHT" \
     --fork-slot     "$FORK_GLOBAL_SLOT"

   mina-archive-hardfork-toolbox fork-candidate confirmations \
     --postgres-uri "$PG_URI" \
     --latest-state-hash "$LATEST_STATE_HASH" \
     --fork-slot "$FORK_GLOBAL_SLOT" \
     --required-confirmations "$REQUIRED_CONFIRMATIONS"

⚠ is-in-best-chain –  confirms the block is on your best chain; confirmation –  is the gate that proves it is canonical (k-deep) If is-in-best-chain  fails, your archive does not yet have this block on its best chain — wait or pull from the precomputed-blocks bucket. If confirmations fail, either your archive is behind on tip ingestion, or the announcement is wrong. Never proceed against a pending fork point.

                  b. Final missing block auditor pass before destructive changes

mina-missing-blocks-auditor --archive-uri "$PG_URI"; echo "exit=$?"

                  c. Snapshot the DB

pg_dump "$PG_URI" -Fc -f archive_pre_emergency_hf_$(date +%s).dump

                 d. Ensure replayer successful run

Check that replayer can verify correctness of your postgres database by replaying all transactions from archive db and validating that final ledger is correct

mina-replayer --input-file (last checkpoint) --archive-uri postgres://....

 ⚠ If you want to replay from beginning of network you can convert genesis ledger to replayer input file:

jq -c '{genesis_ledger: {accounts: .ledger.accounts, num_accounts: .ledger.num_accounts}}'  genesis.json > replayer_input_file.json

Find more detailed information in the docs and reach out in Discord if you experience any issues.

Goals:

• All nodes running legacy mode (manual upgrade similar to Berkeley HF) will transition over to the new fork

Runbook:

1. Block Producers
   1. Upgrade daemon  to the Mesa build distributed by o1Labs. Either:
      1. Install new dockers/debians
      2. Or Reconfigure deployments with the new Mesa runtime-config (rooted at the fork-point ledger) – same artifact the network agreed on.
   2. Re-enable BP keys
   3. Using explorer, verify your first produced block lands canonically – check protocol_versions.transaction = 4 and your producer pubkey on the new tip.

curl -s $GRAPHQL_URL -H 'Content-Type: application/json' \ -d '{"query":"{ bestChain(maxLength:5){ protocolState { previousStateHash consensusState blockHeight slotSinceGenesis blockchainLength } } stateHash creator } }"}' | jq .

      2. Snark Work Operator

1. 1. Upgrade the worker binary to the Mesa build distributed by o1Labs — the transaction-SNARK circuit and proving keys can change across protocol versions, so an old worker against a new daemon will produce rejected proofs.    
   2. Update the runtime-config / proving-key bundle if o1Labs ships a new one with the Mesa release.                                                                                                       
   3. Re-enable the worker only after the coordinator daemon reports syncStatus = SYNCED against Mesa genesis — submitting against an unsynced daemon risks pinning to the wrong branch.                    
   4. Verify your first post-fork bundle is accepted — look for a non-error log line on submission, and confirm the fee shows up in your account on a canonical post-fork block (global_slot_since_hard_fork  > 0).     

        3.Zkapp Developer

1. 1. 1. Upgrade o1js to the Post fork compatible release.
      2. Resubmit any transactions that were rolled back (from your Step 1 inventory).
      3. Re-deploy regression zkApps if verification-key formats changed; otherwise verify existing on-chain VKs still validate against locally-recompiled provers.

          4. Archive/Rosetta Operator

1. 1. 1. 1. Upgrade Archive, Rosetta and Daemon to new Mesa build. Ensure that new block on post fork network is canonical

Goals:

• Force the chosen branch to be `canonical` up to and including the fork point; effectively “roll back” by ensuring no descendants of the fork point are treated as canonical for the pre-fork protocol version.

Runbook:

1. Archive / Rosetta Operator

```bash
# Stop daemon + Rosetta first so they don't race new inserts.

mina-archive-hardfork-toolbox convert-chain-to-canonical \
  --postgres-uri "$PG_URI" \
  --target-block-hash "$FORK_STATE_HASH" \
  --protocol-version "3.0.0" \
  --stop-at-slot "$FORK_GLOBAL_SLOT" \
  | tee convert_to_canonical.log
```

Then re-run the toolbox to confirm the archive is in the expected shape:

```bash
mina-archive-hardfork-toolbox validate-fork \
  --postgres-uri "$PG_URI" \
  --fork-state-hash "$FORK_STATE_HASH" \
  --fork-slot "$FORK_GLOBAL_SLOT"

mina-archive-hardfork-toolbox fork-candidate last-filled-block \
  --postgres-uri "$PG_URI" | jq .
# Expect: height=$FORK_HEIGHT, state_hash=$FORK_STATE_HASH, slot_since_genesis=$FORK_GLOBAL_SLOT

mina-missing-blocks-auditor --archive-uri "$PG_URI"; echo "exit=$?"
```

⚠ Unlike the standard plan, there is **no `stop_transaction_slot` / `stop_network_slot`** to monitor. The “halt” is operator-driven via `convert-chain-to-canonical`.

Find more detailed information in the docs and reach out in Discord if you experience any issues.

Goals:

• Monitor Mesa network after hard fork transition

Runbook:

1.  Archive / Rosetta operators validations:
   1. Archive:
      1. Coinbase = 360 MINA on post-fork blocks (coinbase = 360000000000 nanomina; filter global_slot_since_hard_fork > 0 ).
      2. Check that Slot duration ≈ 90 s between consecutive canonical post-fork blocks.
      3. Missing blocks auditor run is clean 
      4. Checks for mina-archive-hardfork-toolbox validate-fork … still passes.

Run Replayer on post hardfork:

# Install config:
sudo apt-get install -y mina-mesa-config=4.0.0-preflight-3f038cb


# Produce new input replayer file 

post-fork-replayer-intput.json: {
 "start_slot_since_genesis":{global slot since genesis for first block on post fork network},
 "genesis_ledger": {
    "add_genesis_winner": false,
    "hash": "{ledger hash of first block on post fork network}",
    "s3_data_hash": "{s3 data hash}"
  }
}

# Run replayer:
mina-replayer --input-file post-fork-replayer-input.json --archive-uri postgres://postgres:....

1. 1. Rosetta sanity  against a known post-fork height/hash
      1. /network/list (expect `network: “devnet”` post-fork),
      2. /network/status, spot-check 
      3. /block
      4. /block/transaction.

       2. Block Producers

1. 1.  Confirm produced-block share is in line with stake; no orphaned blocks above FORK_HEIGHT from before the rollback.
   2. Watch for slot-time drift: 90 s slots, 2-min windowing. Sustained < 90 s → time skew or fork mismatch.
   3. Sanity-check coinbase amount on your own blocks (360 MINA).

       3. zkApp developers

1. 1.  Re-run regression suite end-to-end against Mesa: deploy → method call → account update → event emission → fetch.
   2. Verify account.update and zkApp commands appear correctly in the archive (zkapp_commands table) and in Rosetta /block/transaction.
   3. Diff verification-key hashes pre-fork vs post-fork for the same source app — they should match (no o1js semantic break) unless a Mesa-era binding bump was announced.
   4.  Confirm rolled-back transactions either re-landed or were re-submitted; nothing silently lost.

Find more detailed information in the docs and reach out in Discord if you experience any issues.