Important Orchestrator Log Messages

Helpful syslog messages for Orchestrator Operators

KeyGen threshold was not met
Key generation threshold was not met during the keygen process

orchestrator successfully started
Orchestrator started

websocket: close 1006 (abnormal closure): unexpected EOF
Orchestrator failed because znn node is down

websocket: close 1001 (going away): upstream went away
Orchestrator failed because eth node is down

Stopping orchestrator
Orchestrator stopping

Started ECDSA Keygen
The process to start a new key generation has started.

I think the best way to monitor the Orchestrator is to watch the pid (Process ID). If it goes down or up, alert. Also, we can look for websocket: close 1006 or websocket: close 1001 in the syslog. alert if either message is found.

The main reason the orchestrator goes down is because it loses connection to a node.

Example commands to query syslog

journalctl --unit=orchestrator -n 100000 --no-pager | grep "websocket: close 1001"

journalctl --unit=orchestrator -n 100000 --no-pager | grep "websocket: close 1006"

journalctl --unit=orchestrator -n 100000 --no-pager | grep "Stopping orchestrator"

journalctl --unit=orchestrator -n 100000 --no-pager | grep "Orchestrator started"

journalctl --unit=orchestrator -n 100000 --no-pager | grep "Keygen"

journalctl --unit=orchestrator -n 100000 --no-pager | grep "KeyGen threshold"


That’s right. It’s intentional, but it assumes there’s a service to restart it automatically if it crashes, so there should always be a service that restarts it and it shouldn’t run standalone.

1 Like