ATTENTION - Imminent soft fork for vulnerability patch - APRIL 17, 2025

General
1

During the past week, HC1 members discovered and investigated a critical vulnerability in the go-zenon software.

A future post will detail the exact nature of the issue, but what we can say right now is that to our knowledge it has not been exploited.

HC1 then directly contacted several bridge operators to shut down their orchestrators, without disclosing the existence of a vulnerability. This is the reason the bridge has been down.

The currently liquidity for ZNN on CEXs is under 1 USDT, so we will consider it irrelevant.

Based on our understanding, the issue can be fixed through a soft fork.
HC1 will provide instructions in an upcoming post.

We strongly urge all pillars, orchestrators, and node operators to backup their go-zenon nodes at this time.

This will help prevent your node from needing to resync from scratch if it ends up on the old chain.

For a worst case scenario, HC1 snapshotted the chain at height 10102550 on Tuesday.

Until the chain is stabilized and most pillars are patched, all transactions after 10102550 should be considered invalid and at risk of rollback.

As an immediate next step, HC1 will begin rolling out patched nodes and HC1 members will privately patch their pillars to start the soft fork.

This soft fork will function the exact same as the old chain, until someone exploits the vulnerability.

Unfortunately, once we publish the patch, we have to assume that someone will reverse engineer how to exploit the old chain.

At that point, the chain will split into two.

If you patch your node, it’s possible that your nodes will eventually join the right chain, but it’s also possible that you will need to resync.

So please take a backup asap and not when your node is already on the wrong chain.

Understandably this news is terrible to hear.

While we hope to mitigate this issue as smoothly as possible, many might question if there are any other issues that will require the same kind of response.

This is something that HC1 has considered. In an upcoming post, we will describe the long term steps that HC1 will be taking.

7 Likes
3

To mitigate this issue, we will first publish patched znnd binaries so that nodes can upgrade without us needing to fully disclose the vulnerability.

This of course requires trust in the HC1 team.

These binaries were generated in a private GitHub repo using the same build steps as the main go-zenon repo.

To mitigate the risk of trusting a single HC1 member, 0x will post the checksums as well.

Next steps will include publicly posting the patch, review by the other go-zenon maintainers and creating a release in the main go-zenon repo.

At that time, we will suggest all nodes verify checksums against the main repo.

In my last post, I mentioned the need for long term consideration around mitigating bugs and exploits.

After the chain is stabilized, HC1 will:

  • post a full port-mortem
  • establish formal vulnerability disclosure guidelines
  • establish a bug bounty program
  • include security auditing in all work packages
  • anything else that comes out of the post-mortem

The patched binaries also include a very big update.

We have discussed with Shai and have added him as a Community Spork Address which will be valid for a year.

This allows his address to create and activate sporks.

This was done for several reasons.

I should first mention that the vulnerability involves code written by the founding devs.

As we move forward, HC1 will audit the codebase.

If other bugs/vulnerabilities are discovered, having a community member able to spork the chain will give us flexibility around resolution method, including soft fork, spork, halt, etc.

Because sporks are so powerful, we have chosen Shai as a neutral third party with an established reputation and a long term interest in the network.

Another reason is the activation of governance.

Without a community spork address and no signs of life from the current spork address, activating governance would likely have required a hard fork of the network.

While sporks allow elegant upgrades of the network, soft forks and hard forks can be very disruptive, as we can see currently.

While this vulnerability is disappointing, we have taken it as an opportunity to avoid a hard fork for governance activation.

While we intend for governance to remove the legacy spork address, we will keep the community spork address active until it expires in a year. This is to provide adequate time for code audit.

HC1 is doing some additional checks and will release the patched znnd binaries shortly.

We will then evaluate if the best path forward is to patch the Syrius embedded node or create a public release.

6 Likes
4

@0x3639 please confirm the following checksums

4bee99d2f69b71e22f5b29bc6d1866f9672bbe002487209447306c518ee8336b  znnd-linux-amd64.tar.gz
c7e26daef1dd7b9292af8e9fb036b67bfa2891c83294aa03d43f9e4bcf59ab83  znnd-linux-arm64.tar.gz
5

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I have checked these myself and get the following.

4bee99d2f69b71e22f5b29bc6d1866f9672bbe002487209447306c518ee8336b  znnd-linux-amd64.tar.gz
c7e26daef1dd7b9292af8e9fb036b67bfa2891c83294aa03d43f9e4bcf59ab83  znnd-linux-arm64.tar.gz

-----BEGIN PGP SIGNATURE-----

iHUEARYIAB0WIQQxCg6uqESXVM8X6Lv2uC0RVYedqwUCaAPENgAKCRD2uC0RVYed
q5dUAP9fkT2zuxZMPq9SWkY0jwtZcXg1ylH0cRknerJ7crn/lgD9EdWiW5rhwMye
JMUoa+ZW71wDQbarTJ21q3f8Q0HTbQM=
=WvQB
-----END PGP SIGNATURE-----

6

HC1 will post the public patch on Tuesday at noon EST.

It will first be posted to the hypercore-one/go-zenon repo so that everyone can verify the patch and install a verifiable binary.

We will also open a PR into the zenon-network/go-zenon repo so that other community maintainers can review and so that a binary will be built there.

All node operators should backup their nodes immediately if they have not done so. Especially anyone who has not updated with the private binary.
(see first post for instructions)

Once the patch is out, we have to assume that it is only a matter of time before the chain splits. Backing up will help ensure that you won’t need to resync from scratch.

Regarding Syrius, until further notice, the embedded node should not be used.
RPC should be used instead to a known updated node.

I decided against publishing a private patched Syrius as the trust required is far too great because the wallet handles private key data and can create signatures.

There are also few issues in the MacOS build process that will need to be resolved.

My primary concern at this point is the connectivity of upgraded pillars.
If connectivity is low, multiple soft forks can occur with different clusters of pillars thinking they are on the main chain.

To mitigate this, today HC1 will be spinning up multiple patched nodes and getting all of their pillars connected.

Tomorrow, we will post instructions on how to configure your node to ensure connectivity to these nodes.

3 Likes
7

Reminder: Public Patch will go live tomorrow at noon EST

As mentioned yesterday, HC1 has spun up several nodes to ensure high network connectivity so that if the chain splits, it will not fragment into multiple competing chains.

Please check your node’s connected peers to see if it is connected to one of HC1’s nodes.
This is critical for pillars to prevent downtime in the case of chain split
You can use the following command

curl -X POST http://127.0.0.1:35997 -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "id": 40, "method": "stats.networkInfo", "params": []}'

Below is the list of nodes and a snippet for your config.json
If you are not connected to an HC1 node, please add this to your seeders and restart your node.
Even if you are already connected, it doesn’t hurt to add to seeders.

{
	"Net": {
		"Seeders": [
			"enode://321603fcaa05f94685eed6f21237128ce73426ec8dafda1391d24ff15657e773493b0d3027e8bbb31b026871c34d90e9784bd81f17ebd65c98bb82ba30c1fa1b@109.104.153.179:35995",
			"enode://725d1d2039dfcd18639da9ecaf842c48ad0559d04b6d64b8984973c87a917c761815804a725dd9fc3ebde6207588d40a46677a7ceb8401b82f978529f3f4de87@203.23.128.244:35995",
			"enode://cf747c665d858a521948c61726be2353ba5436d501e24a8947cb5810667fb5e10fecc08aedc8a563da1a105e960e2203025bd494144bd3b22b2f9955e4290212@103.106.228.23:35995",
			"enode://e44faaa3a6936d690bfa7631a16f44ca48ee7932c07e412ed1ae8321f04ca36d27202587ec313041b0d758dbc3bec66630e22ff1ed5f76f156bae69ae321d3a0@147.78.1.111:35995",
			"enode://f24dba632f6c6fcdf11266bb3dd4368f4e19898ef5831afb265c57cbc68d3e78b28750b5f411b86da80390def7c8548d4e4dc84c3c8a120ad177d9d84e3f9a13@23.95.77.201:35995",
			"enode://70bb10c735ca16c122fb108afdab7a1641b977ad9a5c863d0c45195bf24f3b2854a2451d57d35912ce8d1c1309ea86c672dc05c4935f28aa9a76686926f98a0b@185.192.124.162:35995",
			"enode://c26068988ba35648776d800bff9541edf6e4ff20dba45394511659e18ab4c9472e7741778dbc0960c4ed386a679aa1c20a9e904844dba6e282027781059d1a92@103.75.119.93:35995",
			"enode://1b9f3522e91d7a3e6a54b02d57309ae547d985014b6d06e14c29c44865cfd6f08e5abc81b6c45fd41e98c4077a508daaea266711fc9f09ce14f3bf7e0c985279@45.154.205.72:35995",
			"enode://d6a262ca6d9c0429008bf88048a2b8fd6e84482efe59d13709132d710bc6706d24e6ffc426f7f63b8ecd56156763581bb0b6171f2fcd2f0eed3f7132468aa447@37.143.130.57:35995",
			"enode://b64c4289d1ca42fd8af9ad3e6a6ed2d6e79b8ac2c05d6af7187313b7edc001d7ec8296e9081ba514bc511df72463714c99668abfbbff731bac61917c00c98dae@185.186.78.207:35995",
			"enode://1f541366fc3c7ca56cbbe549465a9d100942e27c3fbdda881311c3878f628cd03b08372e607e898afa0ce4655d417b94993602f1a626ee66fe00b5dc42c48d55@65.109.138.231:35995",
			"enode://f23b11c75ea9b3348f6e402096cb717767f577c7ab76aa9a7c6aa6cb0f1cbdee72ca3ac3fad61d761d74f0c6787268f6b9bd51e16d051591f4545d250c7434a2@162.55.53.126:35995",
			"enode://232db9014fbdd94c364ea89733ad6adbc0924f8e7f55b5be4b12a3e8ef4d586053e5af9d50addaab5e52d47a5589b7440ad1f8e9b3dc2d75a8f237dd16bf31a8@49.12.218.89:35995"
		]
	}
}

In addition to these nodes, HC1 has spun up a few other nodes that are hardcoded to connect to these nodes to ensure everything is well connected.

1 Like
8

What about sentrified pillars? Can we simply modify the amount of peers and add these nodes to the config?

9

No, I would leave your pillars to connect only to your sentries.
Instead, you can check if your sentries are connected to to these nodes.

From a networking perspective, the sentries + pillar makes one logical pillar, since the pillar is internal.
Good question!

1 Like
10

After we post the public patch tomorrow, it’s hard to say when someone will try and trigger the vulnerability.

Given that there is no liquidity, there really isn’t any direct financial incentive, but sometimes people just do things in decentralized systems because they want to.

It could be very shortly after or it could not be for a while.
To be clear, once the vulnerability is used, pillars who have not upgraded will be forked out until they patch and resync.
Currently this likely includes many pillars.

While HC1 has spun up many nodes to help ensure only a single chain survives, HC1 cannot run all of these nodes indefinitely.

One thing HC1 is considering is to preemptively split the chain ourselves so that we can quickly get through any uncertainty and chaos.
This would be “flushing” the network of unpatched nodes AFTER the patch is made public.

This would likely be done through the creation of a spork using the community spork address, which again is controlled by Shai.
Unpatched pillars will not see this spork creation transaction as valid, while patched pillars will.
To be clear, this would entail only creation, and not activation.
It would be the spork that we use to activate governance later on.

While HC1 felt that quickly introducing the community spork address was the best path forward, using it for this purpose is something we are still discussing internally and should probably be discussed with the community as well.

2 Likes
11

Do you mind explaining a bit more what would be the cons/pros of not flushing the network? I believe there’s an extremely low probability of someone just triggering the vulnerability based off community numbers alone.

If someone doesn’t trigger it for a few months or whatever that’s more time for people to upgrade but since hc1 is spawning new nodes I get the sense that it might be hard to monitor when the chain is splitting.

Seems like flushing would give us a clearer picture of where the network stands.

Feel free to edit this template so it’s more accurate.

Flushing the Network (Preemptive Split)

Pros Cons
Ends uncertainty quickly. Causes immediate network disruption.
Reduces the exploit window. Instantly forks off unpatched nodes.
Forces faster network upgrades. Requires using the new community spork address.
Reduces need for temporary HC1 nodes. Penalizes slow or unaware operators.
Allows controlled split timing by HC1/Shai.

Not Flushing the Network (Wait for Natural Split)

Pros Cons
Avoids immediate forced disruption. Prolongs uncertainty and potential chaos.
Gives operators more time to patch. Split timing is unpredictable (malicious actor).
Avoids using the community spork address now. Leaves the old chain vulnerable for longer.
No split if never exploited (unlikely). Indeterminate length not feasible for HC1 nodes.
Risks network fragmentation on natural split.
2 Likes
12

We released the public patch and are waiting for approval from other community maintainers: V0.0.8 patch by georgezgeorgez · Pull Request #50 · zenon-network/go-zenon · GitHub

When we released the patched binary, our idea was that after the code was published, binaries built using the updated code could be matched against our closed source binary using the checksum.

Unfortunately, while trying to recreate the binary using the public repos, we discovered that reproducible builds in Go were not possible until version 1.21. The master branch of go-zenon is currently on version 1.20 of Go.

To reduce trust on HC1, we encourage all nodes to build their node from source using the public hypercore-one patch branch. GitHub - hypercore-one/go-zenon at v0.0.8-patch

2 Likes