How to Setup Orchestrator

INSTRUCTIONS TO SETUP AN ORCHESTRATOR

Orchestrator Infrastructure Requirements

The Orchestrator can be run on the Pillar itself or a separate virtual machine. The stand alone virtual machine requirements are:

CPU: 2 cores w/ 4 threads (4 vCores)
RAM: 4 GB
Disk: NVMe w/ 40 GB
Connectivity: Public IP w/ low latency (preferably datacenter)

If you separate the Orchestrator from the Pillar you will need access to a znnd node. If you run the Orchestrator on the Pillar itself, you can use the znnd node of the Pillar.

Setup ETH node

Before you get started with installing the Orchestrator, you will need to setup an Ethereum node. You can follow these instructions to do that: How to Setup an Ethereum Node. Take note of the ws or wss endpoint for use in the config.json file below.

Download the Orchestrator Binary

wget https://github.com/HyperCore-Team/orchestrator/releases/download/v0.0.7-alphanet/orchestrator-linux-amd64.zip

unzip orchestrator-linux-amd64.zip

Alternatively Build the Orchestrator Binary from Source

Get Source Code
git clone https://github.com/HyperCore-Team/orchestrator

Build the Binary

cd orchestrator
go build --ldflags '-extldflags "-Wl,--allow-multiple-definition"' -o ./build/orchestrator main.go

Setup the Config.json File

Move the Binary and create the config.json file

sudo cp orchestrator-linux-amd64 /usr/local/bin/
sudo /usr/local/bin/orchestrator-linux-amd64

After running the process above it should fail, but will create a working directory at /root/.orchestrator/

Next we will update the config.json file located at /root/.orchestrator/config.json

sudo nano /root/.orchestrator/config.json

Update the config.json file with the correct information.

  • Make sure to update the websocket Urls for the Ethereum and Zenon nodes.
  • Make sure to provide the passphrase for the ProducerKeyFilePassphrase. This can be found by running the sudo ./znn-controller then select status.
  • Note the EvmAddress is unique for each Orchestrator and should not be changed. This will be created by the orchestrator. Do not change or remove it after creation.
  • Add the Bootstrap address: "Bootstrap": "/dns/bootstrap.zenon.community/tcp/55055/p2p/12D3KooWBVQYaz3yuJor8oW7bUqoAGDZDpFBGbGerL3SprHn57pQ"
{
    "DataPath": "/root/.orchestrator",
    "EventsPath": "",
    "QueuesPath": "",
    "GlobalState": 0,
    "EvmAddress": "",
    "Networks": {
        "BSC": {
            "Urls": [
                "ws://127.0.0.1:8545"
            ],
            "FilterQuerySize": 2000
        },
        "Ethereum": {
            "Urls": [
                "INSERT"
            ],
            "FilterQuerySize": 2000
        },
        "Zenon": {
            "Urls": [
                "ws://127.0.0.1:35998"
            ],
            "FilterQuerySize": 0
        }
    },
    "TssConfig": {
        "Port": 55055,
        "PublicKey": "",
        "DecompressedPublicKey": "",
        "LocalPubKeys": [],
        "Bootstrap": "/dns/bootstrap.zenon.community/tcp/55055/p2p/12D3KooWBVQYaz3yuJor8oW7bUqoAGDZDpFBGbGerL3SprHn57pQ",
        "PubKeyWhitelist": {},
        "BaseDir": "/root/.orchestrator/tss",
        "BaseConfig": {
            "PartyTimeout": 90000000000,
            "KeyGenTimeout": 600000000000,
            "KeySignTimeout": 280000000000,
            "KeyRegroupTimeout": 60,
            "PreParamTimeout": 600000000000,
            "EnableMonitor": false
        }
    },
    "ProducerKeyFileName": "producer",
    "ProducerKeyFilePassphrase": "INSERT",
    "ProducerIndex": 0
}

Copy the producer file into the DataPath. This assumes you are running the orchestrator on the pillar.

sudo cp /root/.znn/wallet/producer /root/.orchestrator/producer

Alternatively, if you are running the orchestrator on a different VM than the pillar, follow these instructions:

On the Pillar Node

sudo cat /root/.znn/wallet/producer

Copy (right click mouse) the entire key (including the curly brackets).

Then go back to the orchestrator server

sudo nano /root/.orchestrator/producer

Paste the key (make sure that you don’t double the curly brackets). You can paste in terminal by right clicking. Save the file: ctrl+x then y

Setup Orchestrator as a Service

sudo nano /etc/systemd/system/orchestrator.service

Copy and paste to following contents into the terminal.

[Unit]
Description=Orchestrator Service
After=network.target

[Service]
User=root
Group=root
ExecStart=/usr/local/bin/orchestrator-linux-amd64
ExecStop=/usr/bin/pkill -9 orchestrator-linux-amd64
Restart=on-failure
TimeoutStopSec=10s
TimeoutStartSec=10s
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=orchestrator

[Install]
WantedBy=multi-user.target

Activate and start the orchestrator service

sudo systemctl daemon-reload
sudo systemctl enable orchestrator
sudo systemctl start orchestrator

Check the status of the Orchestrator Service

sudo systemctl status orchestrator

Open Firewall

Make sure port 55055 TCP is open on your firewall for gossip between nodes.

Fuse 120 QSR

Make sure you have fused at least 120 QSR to the Pillar Producer Address

Steps to Update the Orchestrator Binary

wget -O orchestrator-linux-amd64.zip https://github.com/HyperCore-Team/orchestrator/releases/download/v0.0.2-alphanet/orchestrator-linux-amd64.zip #Note this could change as releases change
unzip orchestrator-linux-amd64.zip
sudo systemctl stop orchestrator
sudo cp orchestrator-linux-amd64 /usr/local/bin/
sudo systemctl start orchestrator
# check status of orchestrator by running next command
sudo systemctl status orchestrator

Notes

  • After starting the Orchestrator you can see your EVM address "EvmAddress": "", in the config.json file. You can see that address by typing sudo cat /root/.orchestrator/config.json
  • Search for node PubKey journalctl --unit=orchestrator -n 100000 --no-pager | grep "ic.priv"
  • Check listening ports netstat -tulpn | grep LISTEN
2 Likes

Do we need a full node for ETH running in on the same IP ? It could be great if Sumamu provide docs to do the whole thing and set that up since the deadline is quite close.

I’m using a remote ETH node on a different IP. If you don’t run a full node yourself, I’ve found Alchemy to be pretty good. They have a free tier that should work.

The tutorial worked flawlessly for me. I used https://www.alchemy.com/ and sat up the orchestrator on a different VPS. All up and running.

1 Like

@sumamu through trial and error we found out that the orchestrator runs differently when running it under the bridge administrator. What are the exact key differences and why?

Can you confirm when the orchestrator is running under admin it does not participate in the tss key ceremony and does not listen on port 55055?

When running a znnd node, it is common practice to restart the node perodically (e.g. once a day), since for unknown reasons the node will slowly eat up all system RAM.

When znnd is restarted the RPC interface is unresponsive for a period of time. When the orchestrator fails to communicate with the node because of this, it shuts down/crashes, with the following log messages:

INFO        rpc/znn.go:47        Error when dialing ws://127.0.0.1:35998, got: dial tcp 127.0.0.1:35998: connect: connection refused
INFO        rpc/znn.go:47        Error when dialing ws://127.0.0.1:35998, got: dial tcp 127.0.0.1:35998: connect: connection refused
INFO        rpc/znn.go:47        Error when dialing ws://127.0.0.1:35998, got: dial tcp 127.0.0.1:35998: connect: connection refused
INFO        app/manager.go:37        failed to create the node        {"reason": "cannot connect to any urls to a znn node"}
cannot connect to any urls to a znn node
systemd[1]: orchestrator.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: orchestrator.service: Failed with result 'exit-code'.

It would be good if the orchestrator handled this situation gracefully and waited for the node connection to be restored.

2 Likes

Why does the orchestrator need to be stopped before it’s started? I figured we only stop it before we modify the service.

Restart-always will NOT allow you to stop the service ever. Even when you stop it manually. So the instructions to change it this time are a little different.

1 Like

If you want the orchestrator to restart multiple times after it crashes (in instances where perhaps it can’t connect to the Zenon / ETH node and needs to try again/wait between attempts):

Stop the orchestrator:

sudo systemctl stop orchestrator

Followed by:

sudo nano /etc/systemd/system/orchestrator.service

Then add StartLimitBurst=999999 and RestartSec=30 as seen below:

[Unit]
Description=Orchestrator Service
After=network.target
StartLimitBurst=999999 #try to restart it 999999 times

[Service]
User=root
Group=root
ExecStart=/usr/local/bin/orchestrator-linux-amd64
ExecStop=/usr/bin/pkill -9 orchestrator-linux-amd64
Restart=on-failure
RestartSec=30 #wait 30 seconds before each restart
TimeoutStopSec=10s
TimeoutStartSec=10s
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=orchestrator

[Install]
WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
sudo systemctl start orchestrator

Shared by @sumamu

what was happening? it was trying to restart and did not restart?

I am guessing this is related to those who have an auto restart on their node due to memory issues. I had same problem on automated restart of zenon node the orchestrator could not connect to zenon node and exited. The 30 sec should allow the zenon node to come back up then orchestrator connects.

2 Likes

Updated, testing the new settings. I had my node tweaked to restart once a day so the orchestrator wouldn’t stay up. Let’s see.

Yes this is exactly what was happening.

I missed your key signing ceremony so I am essentially sitting on the bench… I am orchestrating the hell out of this bench though

4 Likes

Will there be any rewards for Orchestrators ?

not now. But I want to learn and get better at running this orchestrator b/ once the side chains become available I think it will use this orchestrator and that WILL be incentivized. So we are learning how to run this and it will (should / could) receive rewards in the future.

Since the last update I have an issue. One orchestrator is going down


image

You have an idea ?

Some others have reported issues. Two of them changed some values in config.json

  • GlobalState : 0 (it was 3)
  • LocalPubKeys : nul (it was )

I have not been able to trouble shoot why or discuss with HCT. I’ve been traveling and will be back on Tuesday to look into this further.

I tried that but it doesn’t work for me. I will look again this afternoon.

1 Like

I changed my GlobalState to 0 and it now runs error-free. Had a similar error as @Chadass

My LocalPubKeys had a bunch of keys, so I left that unchanged.