WP1 Network Launch Code & Config

georgezgeorgez · December 13, 2024, 5:04pm

@vilkris did you let your computer sleep at all?
i think i might have only seen the issues when my laptop slept but i kept the node running

On some systems the monotonic clock will stop if the computer goes to sleep. On such a system, t.Sub(u) may not accurately reflect the actual time that passed between t and u. The same applies to other functions and methods that subtract times, such as Since, Until, [Before], After, [Add], [Sub], [Equal] and [Compare]. In some cases, you may need to strip the monotonic clock to get accurate results.

georgezgeorgez · December 13, 2024, 6:12pm

Yeah I think I was able to reproduce by telling my laptop to sleep

Producing momentum ...
[Momentum inserted] Height: 350, Hash: abc09b5e43e490481d2599360b0b48b537b5463a5548f142d8f7e978e45e392f, Timestamp: 1734110975, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:29:35, Txs: 0
Producing momentum ...
[Momentum inserted] Height: 351, Hash: 7b7c7fb34bb4853b040e0cf4db5033ca87f2fd0f2078c94cd0d708868fd948a3, Timestamp: 1734111035, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:30:35, Txs: 0
Producing momentum ...
[Momentum inserted] Height: 352, Hash: 788baf76cdc75fa5a52d8d2af9d98b83730db2bbabcda5a4a974fb0a248331e3, Timestamp: 1734111095, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:31:35, Txs: 0
Producing momentum ...
[Momentum inserted] Height: 353, Hash: cbf7a745c62d88fe841011cf0128a9273248373935572ea183ee004472944fd8, Timestamp: 1734111155, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:32:35, Txs: 0
Producing momentum ...
[Momentum inserted] Height: 354, Hash: 708844703035efd936be47ca385a4af5d9156414e77c18e1ccf7185b1a09832c, Timestamp: 1734111215, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:33:35, Txs: 0
Producing momentum ...
[Momentum inserted] Height: 355, Hash: d8a5d322e845bc0eb6cece6822469376b6ca1b76f0234943a44cde08904314e2, Timestamp: 1734111275, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:34:35, Txs: 0
Producing momentum ...
[Momentum inserted] Height: 356, Hash: 42de37b26518275c19186cc0b8e6b09fbc9e45262b832ba46e079fc0ee828b0a, Timestamp: 1734111335, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:35:35, Txs: 0
Producing momentum ...
[Momentum inserted] Height: 357, Hash: 939ca8f89974f23e9d2a51507e3846e042f6c0fdc0bc1376d3b7e2dce3895a4a, Timestamp: 1734111395, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:36:35, Txs: 5
Producing momentum ...
[Momentum inserted] Height: 358, Hash: f0294126a11b780f6d15213240adde573e746eac26a04e61e4d9185b5b801100, Timestamp: 1734111455, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:37:35, Txs: 5
Producing momentum ...
[Momentum inserted] Height: 359, Hash: 25e7fd0511df32b6bb9784d70e4cac2cae363a41a61f5d3db16292de73481de8, Timestamp: 1734111515, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:38:35, Txs: 0
Producing momentum ...
[Momentum inserted] Height: 360, Hash: 04425a68a99b3c5a4a1efedf1d75aacaa488804dda486f3a7b72b3bad05ece29, Timestamp: 1734111575, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:39:35, Txs: 0
Producing momentum ...
Producing momentum ...
[Momentum inserted] Height: 361, Hash: cf900f4fef5bf3afba3354847185e57c172ff5120dad731f2dccaae617bbde5e, Timestamp: 1734112055, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:47:35, Txs: 0
Producing momentum ...
[Momentum inserted] Height: 362, Hash: c066dd49839e1108b463b65aba435e748b249d7807a93d98588901ba6c687500, Timestamp: 1734112115, Pillar producer address: z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh, Current time: 2024-12-13 12:48:35, Txs: 0
Producing momentum ...
Producing momentum ...

t=2024-12-12T20:49:34-0500 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-12 20:49:35 -0500 EST EndTime:2024-12-12 20:50:35 -0500 EST Producer:z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh Name:Local}" reason="current time is before start time"
t=2024-12-13T12:47:23-0500 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 12:40:35 -0500 EST EndTime:2024-12-13 12:41:35 -0500 EST Producer:z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh Name:Local}" reason="current time is after the event's finish time time"
t=2024-12-13T13:04:40-0500 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 12:49:35 -0500 EST EndTime:2024-12-13 12:50:35 -0500 EST Producer:z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh Name:Local}" reason="current time is after the event's finish time time"
t=2024-12-13T13:07:02-0500 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 13:05:35 -0500 EST EndTime:2024-12-13 13:06:35 -0500 EST Producer:z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh Name:Local}" reason="current time is after the event's finish time time"
t=2024-12-13T13:07:34-0500 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 13:07:35 -0500 EST EndTime:2024-12-13 13:08:35 -0500 EST Producer:z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh Name:Local}" reason="current time is before start time"
t=2024-12-13T13:08:34-0500 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 13:08:35 -0500 EST EndTime:2024-12-13 13:09:35 -0500 EST Producer:z1qpeql04cww98g5qczqkgxxjgdtwmyjfn3x0adh Name:Local}" reason="current time is before start time"

You can see where I slept the computer yesterday night.
And these messages only show up again right after the double Producing momentum... which is when I put the computer to sleep

vilkris · December 13, 2024, 8:54pm

Nope, no sleeping. The issue starts immediately once I start the node. Not sure how much time we want to spend on this right now, since the root cause may be hard to figure out and fix.

Adding something like this would solve the problem in my environment:

diff --git a/consensus/consensus.go b/consensus/consensus.go
index 1c86bad..ec061fd 100644
--- a/consensus/consensus.go
+++ b/consensus/consensus.go
@@ -172,6 +172,13 @@ func (cs *consensus) work() {
                        case <-cs.closed:
                                return
                        case <-time.After(event.StartTime.Sub(time.Now())):
+                               // In some cases the time awaited by time.After is slightly incorrect.
+                               // This loop ensures the event is not broadcast too soon.
+                               for {
+                                       if time.Now().Unix() >= event.StartTime.Unix() {
+                                               break
+                                       }
+                               }
                        }

georgezgeorgez · December 13, 2024, 10:09pm

I agree. Let’s not dwell on it.
Does it stop all momentum production or only make it unreliable for you?

I am hesitant to make a patch for everyone if we don’t understand the behavior.
It sounds like your current env is a laptop. I think we should first see if you or others have the same issue on the actual servers they will run the hqzd node on.

vilkris · December 14, 2024, 9:33am

It doesn’t stop completely but it makes it unreliable and sometimes it can go a long time without being able to generate momentums. Then at other times it will generate momentums with no problems for a while. See the logs here, this happened shortly after I started the node and computer was not in sleep mode:

t=2024-12-13T09:47:42 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 09:47:43 EndTime:2024-12-13 09:48:43 Producer:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr Name:Local}" reason="current time is before start time"
t=2024-12-13T09:48:42 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 09:48:43 EndTime:2024-12-13 09:49:43 Producer:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr Name:Local}" reason="current time is before start time"
t=2024-12-13T09:49:42 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 09:49:43 EndTime:2024-12-13 09:50:43 Producer:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr Name:Local}" reason="current time is before start time"
t=2024-12-13T09:50:42 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 09:50:43 EndTime:2024-12-13 09:51:43 Producer:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr Name:Local}" reason="current time is before start time"
t=2024-12-13T09:51:42 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 09:51:43 EndTime:2024-12-13 09:52:43 Producer:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr Name:Local}" reason="current time is before start time"
t=2024-12-13T09:52:42 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 09:52:43 EndTime:2024-12-13 09:53:43 Producer:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr Name:Local}" reason="current time is before start time"
t=2024-12-13T09:53:42 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 09:53:43 EndTime:2024-12-13 09:54:43 Producer:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr Name:Local}" reason="current time is before start time"
t=2024-12-13T09:54:42 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 09:54:43 EndTime:2024-12-13 09:55:43 Producer:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr Name:Local}" reason="current time is before start time"
t=2024-12-13T09:55:42 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-13 09:55:43 EndTime:2024-12-13 09:56:43 Producer:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr Name:Local}" reason="current time is before start time"

Fair enough. We can table this discussion for now. I’m using a Mac mini, but you’re right that the main environment will be Linux distributions. I’ll spin up a VPS to test it there.

vilkris · December 14, 2024, 12:48pm

I ran this sequence and it is working as expected. Pillar does not get registered and the following message is shown in the log:

t=2024-12-14T12:32:43+0000 lvl=info msg="generated embedded-block" module=pillar submodule=worker send-block-header="{Address:z1qqzsrnzpfev0dvu5eqge2t8kx4uzpmq30hpzpr HashHeight:{Hash:9d3a17fafa8fe4bb8e79cf9b0b8d19076dc65cc47c2ce6a928a513d0acb1eeee Height:10
}}" identifier="{Address:z1qxemdeddedxpyllarxxxxxxxxxxxxxxxsy3fmg HashHeight:{Hash:5b17fdc01ee060eba2df37a0cad59d6ad6dbbd08a636b4534f1434ed038a5f54 Height:4}}" send-block-hash=9d3a17fafa8fe4bb8e79cf9b0b8d19076dc65cc47c2ce6a928a513d0acb1eeee returned-error="a
ddress cannot call this method"

Is it the plan to activate the no pillar spork via genesis config on hqz?

Also, no momentum production issues to report on Ubuntu 22.04.

georgezgeorgez · December 14, 2024, 5:14pm

Yes we will activate it using genesis config for the actual network.
I figured having it on devnet for now as an explicit spork activation would help explain how its working.

CryptoFish · December 14, 2024, 5:29pm

I ran into no problems when executing these commands.

coinselor · December 16, 2024, 7:57am

Managed to build and run it on an ubuntu 24 VM. However, it got stuck at height 2. I submited a plasma.fuse tx to see if that would move it, but didn’t help.

btw the repo is public but I had to use https to clone it, so maybe safer to use that for future tutorials

Update: Height 3 happened 19 minutes later. Tx showed up, plasma.list updated correctly.

georgezgeorgez · December 16, 2024, 8:48am

what are the specs of this vm?

coinselor · December 16, 2024, 10:26am

This is a fresh install. It’s possible this is going to sleep on idle somehow. It’s up to height 6, but the multiple “Producing momentum …” lines are not normal right?

Update: This is printing exactly one momentum each 30 minutes… ? I restarted the node at height 8 to see if it would help. It pretty much printed height 9 instantly then print slowly.

georgezgeorgez · December 16, 2024, 5:40pm

Can you dump log/zenon.log and the error.log as well?

vilkris · December 16, 2024, 6:56pm

I’m having issues getting all unit tests to pass and this is happening with the go-zenon codebase as well, so probably not related to hqzd changes. I can reproduce the problem with Ubuntu and macOS.

Can anyone confirm that all unit tests are passing for them? Running go test -run ./... in the project’s root will run the tests.

georgezgeorgez · December 16, 2024, 7:03pm

Which tests?
I have had this issue before.

One thing I have urgently for future work packages is to baseline the code.

vilkris · December 16, 2024, 8:10pm

These embedded pillar tests:

TestConsensus_1
TestConsensus_2
TestPillar_RegisterRevokeRegisterPillar

And then most embedded sentinel and embedded stake tests are failing.

The weird thing is that if I run the tests via VSCode then the tests pass. Downgraded Go to 1.22.10 but same results.

In any case this a bit off topic since it’s not caused by the hqzd changes.

coinselor · December 17, 2024, 1:38pm

zenon.error.log

t=2024-12-16T07:36:04+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:d9ccdca74382fd098ca346f842a11e8b4389b2c8fe74c94b5c333cfb8586e601 Height:2}" reason=too-late
t=2024-12-16T07:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:3b3d9a31fdaebc49101dfae712d5bda0e745633aa1e42d11e155af909cdaee16 Height:3}" reason=too-late
t=2024-12-16T08:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:b5572903d74c77009a1bdf2c3dbc7e8a4faa0e61a2802dc63209e943ef3ebf31 Height:4}" reason=too-late
t=2024-12-16T08:39:44+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:da534816fb75e17b1446003c2810e651d9fcf35ed998e6bb9acc213275db7cf9 Height:5}" reason=too-late
t=2024-12-16T08:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:e6719d6c1400458997fb263f2cb0e42eef78c55c69b8172c50c2cd74faf592fa Height:5}" reason=too-late
t=2024-12-16T10:14:53+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:271db8d005646f1049fc4897e67adce490e1d258c990b5530efd626c81f6cbef Height:6}" reason=too-late
t=2024-12-16T10:25:23+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:2e133a6b31e1a968f72168f2c77d82c18a7d1e1e2faaea32135f56f520340b5c Height:6}" reason=too-late
t=2024-12-16T10:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:68fc19e0e7913f481deba3b44514568071d071b62fa6fc96c15e766b21f5be37 Height:7}" reason=too-late
t=2024-12-16T11:25:23+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:dfc1c6c01a64d6faf99a43c28f95e49b5fb347ebcc30fed4840dfeec2dd248ab Height:8}" reason=too-late
t=2024-12-16T11:32:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:6d154188e46c4ae7dc86a4046d3e0adc132c91f33726d2cebef52012bb8cab4d Height:9}" reason=too-late
t=2024-12-16T11:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:656ae1783db5bc0d6f7f315b6b99bdac637726effefb0cbfe690306d455bc986 Height:10}" reason=too-late
t=2024-12-16T12:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:b7a6947a78eaad02f0d4eae6215605bc3e51ff095ba5d1086091a07c30a99a9b Height:11}" reason=too-late
t=2024-12-16T12:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:ac9c019e9a50cadb21a5297361a96babbeec063e38614f0e51d47411e840c475 Height:12}" reason=too-late
t=2024-12-16T13:25:23+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:5adde345639ca999417eea888e9445fd709754471272c815c7900fa9f36d4b90 Height:13}" reason=too-late
t=2024-12-16T13:25:23+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:5adde345639ca999417eea888e9445fd709754471272c815c7900fa9f36d4b90 Height:13}" reason=too-late
t=2024-12-16T13:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:d7bb77034996825d2a376bc8220138f7317271033b2cb05cbd7682686cc76b81 Height:14}" reason=too-late
t=2024-12-16T14:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:e5876d73d401d450e22a7664245c0d6954cd348396fd37cf397ee6b82374056b Height:15}" reason=too-late
t=2024-12-16T14:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:e48169a4e4a0330507dab5a1f4f6c785b1ae538fc8c9b03ccc2025b993362bb1 Height:16}" reason=too-late
t=2024-12-16T15:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:b97c64cb2ee39449b010370601aeb29a8f68ff22f9f20483f53d4a039bdaf901 Height:17}" reason=too-late
t=2024-12-16T15:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:14cef4b36d741623b51beb8a99863561f54a98c80296e00bfae0f8f5d5ee5a2b Height:18}" reason=too-late
t=2024-12-16T16:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:5bdc8feec1782f9588db45e8f969c660f0ba68007566df3ed4535f365e318b32 Height:19}" reason=too-late
t=2024-12-16T16:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:7decfba0401ac07b1b5a450361f5e95b082cbee802ca95e1c3663318d1e69ac6 Height:20}" reason=too-late
t=2024-12-16T17:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:7b92a29d8cdb6bfd0ad9967f9344b59390fd66df9a3c4ad7c28c4c51bd090b9a Height:21}" reason=too-late
t=2024-12-16T17:55:23+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:1f862c5a0d0cada146cabd4c634f67de0a3deb0c6f2ba887f925132a67e73655 Height:22}" reason=too-late
t=2024-12-16T18:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:72f752e83513621251eee6bb9b131204e76e8724a6eddf90597ae657d7ff6cbc Height:23}" reason=too-late
t=2024-12-16T18:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:8202c58f12ad9656cc6a2cfea48a7153b6679d3c722924f10c43567e8783dffc Height:24}" reason=too-late
t=2024-12-16T19:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:8df5a66caa946cc6df8e6e52829777e6f7c6de56b511b2407823cb759a2c0ff6 Height:25}" reason=too-late
t=2024-12-16T19:55:23+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:92e84f16589fbbf1b5c26c67df696856a048ced1d4cb8f6d99b083bd8ed32520 Height:26}" reason=too-late
t=2024-12-16T19:55:23+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:92e84f16589fbbf1b5c26c67df696856a048ced1d4cb8f6d99b083bd8ed32520 Height:26}" reason=too-late
t=2024-12-17T09:07:43+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:da356c800350590c1ec8f81e891d1984bd0bb72303d4c5eb1d4ce30bf01b8a4c Height:27}" reason=too-late
t=2024-12-17T09:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:5d2c4ab64f0348914e61fc958e7cedf4f840c8de327b72759ab98b19547827a4 Height:27}" reason=too-late
t=2024-12-17T09:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:72a6a314bc836c15f4b6698c7fb9b99df947b9a3f92f8a2561248c29d67960f2 Height:28}" reason=too-late
t=2024-12-17T10:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:561f239a611becfb420b695e0fc393ba250489325f860d1d1775a92d4c603603 Height:29}" reason=too-late
t=2024-12-17T10:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:cf61f44d3606ec004f889a134605c7ab8cf74b704b37b6878eb81bbb88d7ba72 Height:30}" reason=too-late
t=2024-12-17T11:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:a9774a97d2e928cad602d5ce97c503dc027b358dc76f253219d99abb6002d48d Height:31}" reason=too-late
t=2024-12-17T11:40:57+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:ab945f818166612a9f1363c086a09525f1a111cfd33e6da5732d2602f76e9598 Height:32}" reason=too-late
t=2024-12-17T11:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:1b4a58ebb5dbdb451dd123fead2c8bb5dbaa9d6b59c81d24985c36c20fa77eb5 Height:32}" reason=too-late
t=2024-12-17T12:25:23+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:c4e2c7444531ac722afc5307f76b60602d69924f98cbbf9ff972bd4a77fc1786 Height:33}" reason=too-late
t=2024-12-17T12:55:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:f5c6f6c4faf2a6a5aa27abe645af56ad2eaac8f2569046b61772201b4843c1db Height:34}" reason=too-late
t=2024-12-17T13:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:0ba4444452a46c7d23d77a56b9497bebeaddbc24ae25fcdc08e505bb223458db Height:35}" reason=too-late

zenon.log

t=2024-12-17T13:13:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:13:24 +0000 WET EndTime:2024-12-17 13:14:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:14:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:14:24 +0000 WET EndTime:2024-12-17 13:15:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:15:23+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:15:24 +0000 WET EndTime:2024-12-17 13:16:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:16:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:16:24 +0000 WET EndTime:2024-12-17 13:17:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:17:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:17:24 +0000 WET EndTime:2024-12-17 13:18:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:18:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:18:24 +0000 WET EndTime:2024-12-17 13:19:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:19:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:19:24 +0000 WET EndTime:2024-12-17 13:20:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:20:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:20:24 +0000 WET EndTime:2024-12-17 13:21:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:21:23+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:21:24 +0000 WET EndTime:2024-12-17 13:22:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:22:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:22:24 +0000 WET EndTime:2024-12-17 13:23:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:23:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:23:24 +0000 WET EndTime:2024-12-17 13:24:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:24:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:24:24 +0000 WET EndTime:2024-12-17 13:25:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:25:22+0000 lvl=info msg="momentum producer triggered" module=pillar submodule=manager event="{StartTime:2024-12-17 13:24:24 +0000 WET EndTime:2024-12-17 13:25:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}"
t=2024-12-17T13:25:22+0000 lvl=info msg="producing momentum" module=pillar submodule=worker event="{StartTime:2024-12-17 13:24:24 +0000 WET EndTime:2024-12-17 13:25:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}"
t=2024-12-17T13:25:22+0000 lvl=eror msg="do not broadcast own momentum" module=pillar submodule=worker identifier="{Hash:0ba4444452a46c7d23d77a56b9497bebeaddbc24ae25fcdc08e505bb223458db Height:35}" reason=too-late
t=2024-12-17T13:25:22+0000 lvl=info msg="start creating autoreceive blocks" module=pillar submodule=worker
t=2024-12-17T13:25:22+0000 lvl=info msg="checking if can update contracts" module=pillar submodule=worker
t=2024-12-17T13:25:22+0000 lvl=info msg="momentum producer trigger finished" module=pillar submodule=manager event="{StartTime:2024-12-17 13:24:24 +0000 WET EndTime:2024-12-17 13:25:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}"
t=2024-12-17T13:25:24+0000 lvl=info msg="momentum producer triggered" module=pillar submodule=manager event="{StartTime:2024-12-17 13:25:24 +0000 WET EndTime:2024-12-17 13:26:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}"
t=2024-12-17T13:25:24+0000 lvl=info msg="producing momentum" module=pillar submodule=worker event="{StartTime:2024-12-17 13:25:24 +0000 WET EndTime:2024-12-17 13:26:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}"
t=2024-12-17T13:25:24+0000 lvl=info msg="broadcasting own momentum" module=pillar submodule=worker identifier="{Hash:391f7d150fdf78146b9a700a3040065d939bb1da5de87b66e511e417594d2109 Height:35}"
t=2024-12-17T13:25:24+0000 lvl=info msg="creating own momentum" module=handler submodule=broadcaster identifier="{Hash:391f7d150fdf78146b9a700a3040065d939bb1da5de87b66e511e417594d2109 Height:35}"
t=2024-12-17T13:25:24+0000 lvl=info msg="inserting new momentum" module=chain submodule=momentum-pool identifier="{Hash:391f7d150fdf78146b9a700a3040065d939bb1da5de87b66e511e417594d2109 Height:35}"
t=2024-12-17T13:25:24+0000 lvl=info msg="computed producers" module=consensus submodule=uniform-election-manager proof-hash=b4465b4de809da61911cea5d19fd07a73a158d628ddff8447546636f2c01fe4a proof-height=34 delegations=[Local@0] producers="[z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369]"
t=2024-12-17T13:25:24+0000 lvl=info msg="broadcasting own momentum" module=handler submodule=broadcaster identifier="{Hash:391f7d150fdf78146b9a700a3040065d939bb1da5de87b66e511e417594d2109 Height:35}"
t=2024-12-17T13:25:24+0000 lvl=info msg="propagated momentum to peers" module=handler num-peers=0 momentum-identifier="{Hash:391f7d150fdf78146b9a700a3040065d939bb1da5de87b66e511e417594d2109 Height:35}"
t=2024-12-17T13:25:24+0000 lvl=info msg="announced momentum to peers" module=handler num-peers=0 momentum-identifier="{Hash:391f7d150fdf78146b9a700a3040065d939bb1da5de87b66e511e417594d2109 Height:35}"
t=2024-12-17T13:25:24+0000 lvl=info msg="start creating autoreceive blocks" module=pillar submodule=worker
t=2024-12-17T13:25:24+0000 lvl=info msg="finish broadcasting momentum" module=rpc module=subscribe_api identifier="&{Hash:391f7d150fdf78146b9a700a3040065d939bb1da5de87b66e511e417594d2109 Height:35}" elapsed=2.143µs stats="&{NumNotify:0 NumUninstalls:0}"
t=2024-12-17T13:25:24+0000 lvl=info msg="checking if can update contracts" module=pillar submodule=worker
t=2024-12-17T13:25:24+0000 lvl=info msg="momentum producer trigger finished" module=pillar submodule=manager event="{StartTime:2024-12-17 13:25:24 +0000 WET EndTime:2024-12-17 13:26:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}"
t=2024-12-17T13:26:23+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:26:24 +0000 WET EndTime:2024-12-17 13:27:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:27:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:27:24 +0000 WET EndTime:2024-12-17 13:28:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:28:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:28:24 +0000 WET EndTime:2024-12-17 13:29:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:29:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:29:24 +0000 WET EndTime:2024-12-17 13:30:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:30:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:30:24 +0000 WET EndTime:2024-12-17 13:31:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:31:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:31:24 +0000 WET EndTime:2024-12-17 13:32:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:32:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:32:24 +0000 WET EndTime:2024-12-17 13:33:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:33:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:33:24 +0000 WET EndTime:2024-12-17 13:34:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"
t=2024-12-17T13:34:22+0000 lvl=info msg="do not process current event" module=pillar submodule=manager event="{StartTime:2024-12-17 13:34:24 +0000 WET EndTime:2024-12-17 13:35:24 +0000 WET Producer:z1qq4ghtxdvmcdm5rdue6ut95n9hplx7fsh8d369 Name:Local}" reason="current time is before start time"

Seems like I’m getting that same issue vilkris had with the time, and then there’s that “do not broadcast own momentum”.

georgezgeorgez · December 20, 2024, 3:28am

Just some ramblings:

func (ea *uniformElectionAlgorithm) uniformRandom(groupA []*types.PillarDelegation, context *AlgorithmConfig) []*types.PillarDelegation {
	var result []*types.PillarDelegation
	total := int(ea.group.NodeCount)
	sort.Sort(types.SortPDByWeight(groupA))

	seed := ea.findSeed(context)
	for len(result) < total {
		random1 := rand.New(rand.NewSource(seed))
		arr := random1.Perm(len(groupA))
		for _, index := range arr {
			result = append(result, groupA[index])
		}
	}
	return result[:total]
}

In my testing with 5 pillars, I am getting a consistent 5 min time between blocks by the same pillar. Well obviously with the seed being set once, random1 will always produce the same permutation.

The following code should be functionally the same:

func (ea *uniformElectionAlgorithm) uniformRandom(groupA []*types.PillarDelegation, context *AlgorithmConfig) []*types.PillarDelegation {
	var result []*types.PillarDelegation
	total := int(ea.group.NodeCount)
	sort.Sort(types.SortPDByWeight(groupA))

	seed := ea.findSeed(context)
	random1 := rand.New(rand.NewSource(seed))
	arr := random1.Perm(len(groupA))
	for len(result) < total {
		for _, index := range arr {
			result = append(result, groupA[index])
		}
	}
	return result[:total]
}

This consistent permutation should be fine for our purposes for now, but I wonder if we should hash the seed or something to produce a new permutation each go around.

georgezgeorgez · December 20, 2024, 3:39am

Added a nomctl generate-hyperqube-genesis -qubepad-csv <FILE> command to use exported CSV from qubepad.

georgezgeorgez · December 20, 2024, 3:01pm

georgezgeorgez · December 20, 2024, 6:01pm

genesis.json (9.4 KB)

for testnet
genesis timestamp dec 21 noon est