Beam Sync in 80 Seconds, Using Meta-Witnesses

The Trinity Team wants Ethereum clients to launch faster. We also want to verify that block transactions are valid, and look up arbitrary state. The status quo is to wait for hours or days to get synced, and it's unacceptable. So our team developed Beam Sync.

Beam Sync is designed to solve the slowest part of getting a new node up and synced: state download. It does this by jumping to the latest block, and downloading only the state needed to verify that one block. It must download and verify fast enough to stay at/near the tip of the chain.

In the first Trinity prototype, Beam Sync identified missing state on-demand by executing the transactions and downloading missing state account by account, and contract storage slot by slot. This introduces a lot of round-trip latency, waiting for each individual sliver of state (ie~ state trie node) to be identified and downloaded. It was largely serialized: state could not be requested until the previous state was downloaded. This unfortunately meant that it took close to one block's worth of time to download the relevant state. It was right on the edge, we played an arms race with the increasing gas limit. At 8 million gas it was not a problem, 10M was a challenge, and 12M it was too tough to keep up. This seemed like a losing game, so it was time to pack our bags and go home. :(

Except... if we had some way to know ahead of time which state the block would use, we could download all of it in large batches, and in parallel from many peers. That would speed up state download by more than 10x, plenty to make Beam Sync plausible again. Ideally, state metadata would be built into the protocol by including state hints in the header. In practice though, getting it included in the protocol is a long, slow process. Plus, we are only relying on this feature during initial sync, so it's fine to be a bit optimistic. We can look around for at least one altruistic peer who will give us valid metadata about the block. Of course, there was no way to actually ask for this metadata yet.

So we built one, in collaboration with Nethermind. We made a new devp2p subprotocol called wit/0. We pass around a meta-witness: it identifies the state trie nodes required to execute a block. For now, this meta-witness is in the form of a list of trie node hashes. ("Meta" because it only includes the hashes, not the trie node bodies).

This is not just a theoretical win. Watch below to see Trinity sync to head in 80 seconds! The speed of a sync will naturally vary with the machine, connected peers, and random chance. This is just one example run.

The live-recorded demo starts with an empty database, and ends by fully verifying recent blocks and querying arbitrary state. As long as Trinity is connected to a meta-witness provider, it can catch up and stay caught up to mainnet, fairly reliably. (Note that Trinity is largely a research platform, not intended for any production use.)

Things to watch for in the video:

• At 12:58:51, the client starts
• Trinity connects to a hard-coded Nethermind peer that serves meta-witnesses (this will be less important as more clients support wit/0)
• This demo skipped much of the header-download time by using a recent checkpoint. The header checkpoint is optional, and header download is not nearly as time-consuming as state-download.
• By 13:00:10, a block at the tip of the chain has been fully validated, and all the related state stored
• Although essentially none of the state is downloaded yet, we can still request arbitrary account balances using Trinity's json-rpc endpoint. The client loads the state from peers, on-demand.

Trinity built the meta-witness downloader and Beam Sync machinery to prove out the concept. The Nethermind team played a critical role by seeding meta-witness data on the network. A huge "thank you" goes out to that team, especially Tomasz Stańczak.

One challenge that remains to be solved is how to minimize the cost of peers who might grief your Beam-Syncing client, by broadcasting invalid meta-witnesses. In the worst case, this could erase the speed advantage of Beam Sync. There are some natural options that aren't fully vetted, like assigning peer reliability scores, or moving the meta-witness into the consensus protocol. These will take more time, of course. The benefit of griefing the network this way is fairly contained, because it only slows down new peers from syncing, it doesn't affect peers that have finished syncing. (Beam Sync is a transitional process, it eventually downloads all the state and becomes a regular full sync)

The Trinity Team believes the concept is proven enough that all clients would benefit from implementing Beam Sync and wit/0, now. As a client user, it is deeply compelling to have a sync time of only a few minutes. If you are a client developer, feel free to reach out. We're happy to help however we can. If you are a client user, it's time to ask your favorite client to add Beam Sync with Meta-Witnesses! Python is slow, so they should be able to beat 80 seconds no problem. ;)