Cedric Fung

Mixin me 25566

Robustness of Blockchain Nodes

Jul 10, 2020

We run many blockchain nodes, that’s many programs of about 28 different blockchain, and almost half of them are forks of the original Bitcoin Core software. When we decide to run a new blockchain, the most important factor is its software robustness.


Ethereum is the first blockchain in Mixin Network, not because it has the stablest software, but XIN is an ERC-20 token on Ethereum. I still remembered the painful experience with geth, slow synchronization, constant stuck on some blocks and high resource usage. And when we tried to trace transactions of standard ERC-20 tokens, we found it almost impossible to do that with geth, I just couldn’t imagine some official software had no support for the most popular usage of that community.

Then we switched to parity which is now renamed to openethereum. Parity did good traces of not so smart contracts and was more reliable than geth, I even heard some praises from friends that parity was the best Ethereum node software. I think we had some weeks of good time with parity, then we faced the same constant stuck on some blocks issue as geth. Our solution is simple, do constant restart of the parity daemon. After about 1 year, we found parity was never able to catch up with the mainnet, it lagged behind more and more, and restart helped nothing. It was not the disk issue because we already had a good SSD attached since the beginning, we tried many configurations and finally we made an upgrade of the CPU architecture then problems gone. The daily transactions volume hadn’t changed during that period and even much lower than the peak, so what prevented the software from working with a lower end CPU?

Now openehtereum works as expected for Ethereum when we do the constant restart properly. It also had the exact same lagging issue for Ethereum Classic recently, and we made the upgrade of CPU fixed that. Need to mention that a month ago openthereum caused massive mainnet reorganizations of the Ethereum Classic chain after Phoenix hard fork.

Do you fear Ethereum 2.0?

Bitcoin and Forks

Bitcoin is the second blockchain in Mixin Network, we runs the exact Bitcoin Core software. I can blame nothing about Bitcoin Core, it works so flawlessly that you guess their engineers do nothing important. When that famous major SegWit fork happened, Bitcoin Core also processed requests without any issues. This is good software. Especially for a decentralized and distributed network, the most important factor of a program should be reliability.

All Bitcoin forks were more stable than other blockchain software, until this week $DOGE soared and its Dogecoin node started stuck to some blocks now and then. We thought maybe too many transactions in Dogecoin network recently and made an upgrade to its instance CPU. Wow, issues still exist. And I noticed some logs about timeout downloading blocks which was an very old issue someone reported to Bitcoin Core. But I never met the same issue for our bitcoin daemon yet, and can’t find fixes for Dogecoin, so we did the same constant restart for it.

The fork of Bitcoin, Bitcoin Cash made blocks much larger than Bitcoin and caused some issues on parsing the block data and consumes much more CPU and memory to do a single RPC call. It’s lucky that at most time almost no transactions on Bitcoin Cash network, and this problem only occurs when developers make some transactions load test.

The fork of Bitcoin Cash, Bitcoin SV has the exact same issue with Bitcoin Cash, but acts even worse, because there is no hard limit in their block size, and their developers do much more transactions load test and lead to many hundreds of megabytes sized blocks, and even gigabytes sometimes.

Bitcoin forks still work better than Ethereum software, even after when they made so much stupid modifications, that proves the quality of the Bitcoin Core software and how genius the core developers are!

EOS and DPoS

Restart almost fixes all issues, but you won’t want to restart EOS. There is almost no reliable way to stop the EOS node gracefully, at most time you need to do a full hard blockchain replay and it means days even on a very fast computer. However it’s lucky that EOS developers make something good called initializing from snapshot, which allows you to download a recent snapshot of all accounts to initialize the blockchain state, and we used that whenever we need to make an EOS reboot, e.g. when upgrading a EOS release. The con is that we will lose the access to all previous transactions, but who cares about those daily gigabytes of useless token mining transactions.

The ancestor of EOS, BitShares, is capable of upgrading and restarting without any issues though, it does have many daily transactions and consumes much less memory. How come EOS fails to inherit these pros of BitShares?

Another famous DPoS blockchain is Cosmos, stable until now unless they publishes new releases. I think many people would be shocked by how they make new releases, they make a snapshot of all accounts and use that snapshot as the input to a new release, ergo starting a fresh new blockchain by dropping all previous transactions. How brilliant idea! And I think this mechanism helps a lot in early development days but hope they would stop this when Cosmos is stable enough. And there was some serious accidents happened during the the Cosmos Hub 3 upgrade process, which causes the network stuck for hours and many exchanges stopped accepting Cosmos for days.


Developers, please make the node software stable with your hundreds of millions of money raised at your ICO. If you found all you could do is releasing a 2.0 or 3.0 release of your thing, but can’t make a stable 1.0, please refund your ICO for god sake.

References List

  1. Massive mainnet reorganizations of the Ethereum Classic chain after Phoenix hard fork
  2. Bitcoin: Timeout downloading block
  3. EOSIO Snapshots
  4. Cosmos Hub 3 Upgrade Post-Mortem

About the Author

Core developer of Mixin Network. Passionate about security and privacy. Strive to formulate elegant code, simple design and friendly machine.

25566 @ Mixin Messenger