Steps to Fix and Standardize all Orchestrators

Development How To
1

Orchestrator Fix & Standardize Instructions

Step 1: Stop the Orchestrator

Elevate to a superuser for all commands. This way you do not need to add sudo in front of each command.

sudo -i

Stop the orchestrator

systemctl stop orchestrator

Step 2: Standardize Binary Name

If you’re using the old binary name, rename it to the standard location. You can copy / paste this small script into the terminal as is:

if [ -f "/usr/local/bin/orchestrator-linux-amd64" ]; then
    sudo mv /usr/local/bin/orchestrator-linux-amd64 /usr/local/bin/orchestrator
fi

Step 3: Update Service Configuration

Update the service file if the binary name changed:

nano /etc/systemd/system/orchestrator.service

DO NOT COPY / PASTE the text below. Only check that the ExecStart path is ExecStart=/usr/local/bin/orchestrator 

[Unit]
Description=Zenon Orchestrator Service
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/root/.orchestrator
ExecStart=/usr/local/bin/orchestrator #CHECK THIS LINE
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

After saving, reload systemd:

sudo systemctl daemon-reload

Step 4: Update Configuration File

Update the ~/.orchestrator.config.json file:

nano ~/.orchestrator/config.json

Ensure the Ethereum network has FilterQuerySize set to 500:

{
    "Networks": {
        "BNB Chain": {
            "Urls": [
                "wss://bnb-mainnet.g.alchemy.com/v2/API"
            ],
            "FilterQuerySize": 2000
        },
        "BSC": {
            "Urls": [
                "ws://127.0.0.1:8545"
            ],
            "FilterQuerySize": 2000
        },
        "Ethereum": {
            "Urls": [
                "wss://eth-mainnet.g.alchemy.com/v2/API"
            ],
            "FilterQuerySize": 500 #CHECK THIS LINE
        },
        "Supernova": {
            "Urls": [
                "wss://rpc.novascan.io"
            ],
            "FilterQuerySize": 2000
        },
        "Zenon": {
            "Urls": [
                "wss://my.hc1node.com:35998"
            ],
            "FilterQuerySize": 0
        }
    }
}

Step 5: Perform Hard Reset (DO NOT SKIP)

Remove queues and events:

rm -rf ~/.orchestrator/queues
rm -rf ~/.orchestrator/events

Step 6: Update Orchestrator Version

Check current version:

/usr/local/bin/orchestrator version

If not running version: v0.0.9a, you need to upgrade. First ensure jq is installed:

# Check if jq is installed
sudo apt-get install -y jq

Then update the Orchestrator using these commands:

# Get the latest release from GitHub API
latest_release=$(curl -s https://api.github.com/repos/HyperCore-Team/orchestrator/releases/latest | jq -r ".assets[] | select(.name | contains(\"linux-amd64.zip\")) | .browser_download_url")

# Clean up
rm -rf orchestrator-linux-amd64.zip orchestrator-linux-amd64

# Download and install the latest release
wget -O orchestrator-linux-amd64.zip "$latest_release"
unzip -o orchestrator-linux-amd64.zip
sudo cp orchestrator-linux-amd64 /usr/local/bin/orchestrator

Step 7: Start Orchestrator

systemctl start orchestrator

Step 8: Verify Running Status

Check if the orchestrator is running properly:

systemctl status orchestrator

You should see “active (running)” in the status output. If there are any issues, check the logs:

journalctl -u orchestrator -f

Step 9: Set Up Service Monitoring

Create a monitoring script to ensure the orchestrator stays running:

nano /usr/local/bin/monitor-orchestrator.sh

Add the following content:

#!/bin/bash

# Check if orchestrator service is running
if ! systemctl is-active --quiet orchestrator; then
    echo "$(date): Orchestrator service is down. Attempting to restart..." >> /var/log/orchestrator-monitor.log
    systemctl restart orchestrator
    sleep 10
    
    # Check if restart was successful
    if systemctl is-active --quiet orchestrator; then
        echo "$(date): Orchestrator service successfully restarted." >> /var/log/orchestrator-monitor.log
    else
        echo "$(date): Failed to restart orchestrator service." >> /var/log/orchestrator-monitor.log
    fi
fi

Make the script executable:

chmod +x /usr/local/bin/monitor-orchestrator.sh

Set up the cron job to run every 5 minutes:

crontab -e

Select 1 to edit with nano and then insert this at the bottom.

Add this line:

*/5 * * * * /usr/local/bin/monitor-orchestrator.sh

ctrl+x to save and exit nano.

The script will:

  • Check if the orchestrator service is running every 5 minutes
  • If it’s not running, attempt to restart it
  • Log all actions to /var/log/orchestrator-monitor.log
  • Only attempt to restart if the service is actually down