How to Hard Reset Your Orchestrator Using a State Snapshot (Without Burning Your RPC Quota)

Development
1

This is a walkthrough for performing a hard reset of your orchestrator by bootstrapping events/ and queues/ from a published snapshot, rather than rebuilding state from scratch via RPC. The from-zero rebuild path is not viable on free-tier RPC endpoints — you will hit rate limits long before the sync completes. The snapshot approach reduces the sync to just the delta from snapshot height to chain tip, which most free tiers can handle comfortably.

What this procedure does

  1. Stops the orchestrator service

  2. Moves existing events/ and queues/ into a timestamped backup directory

  3. Downloads pre-built snapshot zips from DigitalOcean Spaces

  4. Extracts them into ~/.orchestrator/

  5. Restarts the service and reports status

The orchestrator binary is not modified. This is a state reset, not an upgrade.

Prerequisites

  • Orchestrator running as a systemd service named orchestrator

  • Orchestrator data directory at ~/.orchestrator/

  • curl, wget, unzip, and sudo available

  • Sufficient disk space for: the backup of your current state + the downloaded zips + the extracted snapshot (roughly 3x your current events/ + queues/ size as a safe estimate)

  • You should run this as the user that owns ~/.orchestrator/, not as root

Snapshot locations

https://zenon.sfo2.digitaloceanspaces.com/orchestrator/events.zip
https://zenon.sfo2.digitaloceanspaces.com/orchestrator/queues.zip

Step 1 — Run the reset script

curl -sSL https://gist.githubusercontent.com/0x3639/61eff5d80a3099cb0e038d71bf688f00/raw/317a0fd70f4fdb1cea0c8e87fa9a2990a91a2a0d/gistfile1.txt | bash

If you want to read the script before executing it (recommended), download first:

curl -sSL https://gist.githubusercontent.com/0x3639/61eff5d80a3099cb0e038d71bf688f00/raw/317a0fd70f4fdb1cea0c8e87fa9a2990a91a2a0d/gistfile1.txt -o reset.sh
less reset.sh
bash reset.sh

Do not run with sudo bash — the script uses sudo internally where needed, and running the whole thing as root will resolve $HOME to /root/, which will break path resolution and create the backup in the wrong place.

Step 2 — Verify the service is running

The script ends with systemctl status, but tail the logs to confirm the orchestrator is actually processing from the snapshot height rather than starting at zero:

sudo journalctl -u orchestrator -f

You’re looking for log lines showing the orchestrator picked up at the snapshot’s block height and is syncing the delta forward. If you see it starting from block 0 or genesis, the extraction probably didn’t land in the right place — see troubleshooting below.

Step 3 — Confirm catch-up to tip

Let it run for a few minutes and check that the current block height in the logs is advancing toward the live chain tip. Once it’s caught up, the orchestrator should resume normal operation.

Step 4 — Clean up old backups

The script does not delete old backups. Once you’ve confirmed the reset worked and the orchestrator is healthy, you can remove the backup directory:

ls -lh ~/.orchestrator/backups/
rm -rf ~/.orchestrator/backups/<timestamp>

Keep at least one recent backup until you’re fully confident. Disk usage adds up if you run this procedure repeatedly.

Rollback

If something goes wrong and the orchestrator won’t start cleanly on the snapshot data, you can roll back to your pre-reset state:

sudo systemctl stop orchestrator
rm -rf ~/.orchestrator/events ~/.orchestrator/queues
mv ~/.orchestrator/backups/<timestamp>/events ~/.orchestrator/
mv ~/.orchestrator/backups/<timestamp>/queues ~/.orchestrator/
sudo systemctl start orchestrator

Troubleshooting

  • Script fails at extraction with “events or queues folder missing” — the zip layout may have changed. Manually inspect with unzip -l events.zip | head to see the structure and adjust the extraction target.

  • Orchestrator starts but appears to sync from zero — the snapshot likely extracted into a subdirectory rather than directly into ~/.orchestrator/. Stop the service, check ~/.orchestrator/ for nested folders, move them into place, restart.

  • WebSocket disconnects during delta sync — if you’re on Alchemy or another provider that closes idle connections, set pingInterval to 10 seconds in your webSocketConfig.

  • systemctl stop hangs — give it the full 10 seconds the script allows. If it’s still hanging, sudo systemctl kill orchestrator and then proceed manually.

  • Permission denied on ~/.orchestrator/ — you ran the script as root and the data directory is owned by your normal user (or vice versa). Check ownership with ls -ld ~/.orchestrator/ and run as the correct user.

When to run this

  • After a long downtime where catching up from your last state would take longer than starting from a fresh snapshot

  • After a corruption or crash that left events/ or queues/ in an inconsistent state

  • When switching RPC providers and you want to minimize the call volume during cutover

  • Any time the standard hard reset procedure would be appropriate, but you don’t have the RPC budget to sync from zero