Improving client diversity with Vero

Transcript

Hello. Uh, let's get started. Hello everyone. Uh, my name is Luca. Uh, I'm the CTO at Serenita where we specialize in staking Ethereum. Um, just out of curiosity, how many people here in the room have heard of Pharaoh before? Okay, cool. Um, so yeah, today I want to talk to you about client diversity, the importance of it, why um why we decided to develop Pharaoh. Um, so yeah, let's get started here. Um, for those who don't know, client diversity, just wrap it up, wrapping it up real quick, uh, is the concept of, um, Ethereum not relying on a single single client implementation, but instead relying on different client implementations. And that way we don't get, um, like chain holds and that sort of stuff on

Ethereum. So, if we have multiple client imple implementations of the protocol, uh, we get a lot of resilience because it's very unlikely that all of the clients would suffer from the same bugs. Um so uh currently on the client diversity side there's a lot of choices that node operators can make. Uh there's six different uh CL client implementations consensus layer um and five different execution layer implementations and we're probably going to get a few more. Um in ideal conditions that would mean that each client um is used by about like 17% of the network on the consensus layer side and uh about 20% on the execution layer side. uh we know that the actual client usage distribution is a bit different.

So it's not as ideal as it looks on the on the charts on the right. Um the question arises, do we actually need client diversity, right? Is it worth going through all the trouble implementing all those clients? Um and I would argue that we actually do. Um and to support that statement, I've uh kind of summed up all of the client bugs that we've seen over the years. Uh this is the well I wanted to like sum it sum them all up in one slide but I didn't manage to because there were quite a few. Um these are not all the client bugs that we've seen. These are only the ones that actually affected consensus.

So these are the ones that actually made clients like end up on different chains or not accept uh valid blocks that sort of thing. So these are the client bugs that we've seen on during the proofof work era of Ethereum. I wasn't around for many of them so I don't know like too many details on most of them but um I think about three of them made it to mainet so they actually affected uh mainet operations for some time during proof of work. Uh moving on to proof of stake where I was already involved quite a bit. Uh we see a couple more bugs here. Um the first one during in 2020 was before the beacon chain even launched. So this was on a very early test net but it affected the test net in a huge way. Um a lot of the test net

validators were running prism which had a very interesting kind of bug which made prism think that it was actually uh it basically like uh wound um prisms time ahead one full day. Um then we then it got quiet for a few years but um last year in January we had a like two bugs hit mainet. Uh, one of them affected Baso and the other one a few like only two weeks later affected Nethermind which kind of revived the whole client diversity discussion which was nice. Um, people actually started moving away from Gats after this these two bugs happened.

Um in March of 2024 there was a nasty bug on the pris uh girly test net and just this year um in February and March we had some testn net bugs on um the hleski test net and sapoleia. Uh these were quite quite big and uh I would like to just go into a bit more detail on the hleski incident. Um I could talk about this one for like the full talk but I'm not going to do that. If anyone's interested in the full detail, I would recommend you guys to um watch the protocol Burke talk that the loadar team did. But just uh quickly going over it. So the spectra upgrade was scheduled on Heski on uh the 24th of February and uh so it was scheduled at 9:55 p.m. And only a few minutes later,

something happened on this test net um at at like six minutes into the network upgrade. Um at this point it was not really clear what was actually going on. So there were like some initial symptoms. Some clients reported blocks as invalid. Some thought they were okay. We saw a bit low uh a low participation rate on the network. So like even half an hour in I think it wasn't really clear what's going on and people thought like okay some people just forgot to update update their nodes. Um, so yeah, in the end it turned out to be a big consensus bug. Uh, and like about an hour after the bug manifested, uh, we see a comment from Julio from, uh, he's

from Aragon, right? Um, so yeah, he posted that. Okay. Um, actually Aragon and Reth is in the right here. And all those clients that were mentioned here like G, Nethermind, and Basu were actually on the wrong fork. So this kind of got ugly um in the end like this was more than an hour later and at that point it was finally clear what is actually going on. Um at that point it was already kind of too late because the some validators have started to vote for uh the buggy chain to finalize.

So a few hours later it was really hard for all of those validators on the correct chain to actually um sync up and like validate proposed blocks and in the end uh the whole test net was basically um deprecated in a way and a new test net had to be spun up. So uh how bad does do these things get? Uh you could see like the worst case scenario right um but I wanted to explain like it depends um on many things. So in the ideal scenario, all the validators are on the same chain. Uh they have the same view of the network.

Um but in a buggy client scenario, um I wanted to show like this is a case where 20% of the validators end up on the wrong chain. So they have a client that has a bug. Um in case only 20% of the validators is affected by something like this, the impact is pretty negligible. um we Ethereum never loses finality because those 80% of the correct chain validators are able to just like continue going in a case where 40% of the validators uh start following the wrong chain. Things get a bit more messy and it very much depends on how this client bug manifests itself. But the worst case is that um these 40% on the buggy chain end up uh

starting to vote differently than the ones on the correct chain. And this may in the worst case result in up to seven E of your stake being lost. So that's about a quarter of the stake. Um and then the correct chain would take 12 days to finalize again. And then we have the worst case which is basically what happened on the Hleski test net where um in this case we have 70% of the validators on the wrong chain. Um in this case all of those 70% of validators are able to finalize this buggy version of the chain. Uh which means the correct chain stops finalizing. Uh once the buggy chains finalizes, it's really bad because the validators that ended up voting for this

chain are have no way to go back to the correct chain. The impact of this is pretty huge. Um it's you basically risk losing the entirety of your stake in this case. Um finality would only return to um Ethereum in several weeks. And the scary part is also that for this to happen, we only need to have two simultaneous client bugs on Ethereum right now. So circling back to the Hleski incident where three E clients uh had the same kind of consensus affecting bug um and ended up like voting for it to finalize. I would say it's not that um on um um what's the word here?

It's not it's not impossible for it to happen on mainet, right? So, um with this talk, I wanted to go further and not just like hope that it doesn't happen on mainet, but let's actually aim for it never to happen on mainet. There's a couple of approaches how to get there. Um I'm going to start with the simple approach where we reduce the impact of clan bugs by like spreading your validators across different uh clamp implementations. Say you're running 3,000 validators, you would split them up into like parts of 1,000 each and you'd connect those to separate um client pairs.

This is uh very easy to set up. It's good for the network and if everybody did this then we'd be in a very good place. Um but reality is different. Um in this case you the impact of a single client bug can still be significant because it could still affect like a third of your validators. Then there's a couple more advanced approaches. One of which which is uh running vouch. Vouch is a multi-node validator client which is able to combine data from multiple client implementations. And basically if you run vouch with a combination of these three clients and you set set up the majority at astation strategy vouch will um use those three nodes and make them basically agree on what the correct state of the chain is.

So this is um a very good kind of setup. It's used by many professional node operators already. And I also put it as a con this fact because if vouch ends up having a bug. Um you basically need to wait for vouch to get fixed. And since so many different node operators also run vouch you basically are all offline at the same time. So that could be not so great. Um then there's DVT. Uh two implementations SSV and OAL. Um in this case um you'd like split up your validators into different validator shards. Um and each chart would uh each like part of that validator would need to agree with the rest of the cluster to come to consensus and at a station on what to like attest to which chain is

correct. So I um today I found out that SSV doesn't fully do this. So um if you're running SSV and you think you're predict against the type of client bugs that I'm talking about then it's not completely true. Um but yeah, if you want to know more, then just talk to me after the after the presentation. Um then what would we have? Um the more advanced approaches that I just mentioned, what do they have in common? Um, basically they're pretty complex to set up and they're like if you want to switch them it involves a lot of complex operations and if you want to switch back it also uh is not that easy because

you have like in the case of sorry in the case of um SSV or DVT you need to like split up your validator keys and in the case of vouch you need to uh use vouchers remote signer implementations. So, um, this got me thinking, can we reduce some of the complexity while still achieving the same result of avoiding these client bugs? Uh, that's what made us implement Varrow. Vero is a multi-node validator client. It's very simple um at only less than 4,000 lines of code. It's secure. It never has access to validator keys directly. Um it's highly observable. So it gives you some nice metrics, statistics and it's

compatible with all the CL client implementations. So Ver being a multiode validator client, what does that mean? Um it means that just like vouch can talk to multiple client combinations at the same time and combines their output. This way, even if one of the clients goes down or it ends up on a different chain, Vero just continues going because it it only needs two out of the three clients in this case to agree on what the correct chain is. So, this is a a short summary of the the pros and cons that Vero has. It's obviously not biased at all.

Um, these are some of the metrics that you can get out of Pharaoh, so you can see exactly at which point into the slot. um it attests uh this these are all included in the GitHub repository. It also exports tracing data. So this is really useful if you want to like figure out what went wrong during your blog proposal, which part of the blog proposal took too long. Um one other thing that Barrow can do is it listens to slashing events happening across the entire uh network. If it detects that any of the validators it manages got slashed, it actually stops performing all of its duties. So that prevents further validators from getting slashed. So this is what it looks like on the dashboard. Um this is the normal

state of things, everything's working, duties are being performed and then when once uh a slashing event is detected, uh it just shuts down all duties. So you don't have to like uh react yourself. The software reacts automatically. If you're feeling a bit like this and would like to um try out Vero, this is how easy it is to get started. So um say you have a very simple kind of setup with a consensus layer client, execution layer client and your validator client. Uh what you need to get started is um you need to move your validator keys out of your validator client into a remote center. So that's the first step because Ver never has access to the validator keys directly.

It just doesn't want that. Once you have those keys in the remote signer, it's very easy. You just swap swap your validator client out for Vero. It's just uses the same kind of flags. It's very easy to do. Once you have that, um I would say you should get started running different clients. So spin up some additional client implementations, additional client pairs. I would also recommend going with some of the minority clients. So in this case showing by sulload star wrath grandine um once you have those spun up you just check on their performance see if they're like working well set up your alerting all of that once you're satisfied with that you just

point verarrow at all three client combinations and that's it that's as easy as it as it is. Um just to prove how easy it is this is the command line like um the the command that you need to run. So you just say that you want to run the Vero image. You say where Vero should go to find the different client implementations and you show it where to find the validator keys. That's it. So with a three node setup that I showed uh in a few slides back um your your your setup is completely immune to any single client bug. No matter like how many different validator operators run that client on network, it doesn't matter. You're completely immune to any single client bug. Um, you can perform

maintenance on one of the client pairs anytime you want. And um, yeah, of course, this kind of setup helps make Ethereum more resilient against client bugs. If you're a professional node operator, um, you can also go a bit further, get uh, busy with a five node setup. Um, in this case, you would basically run five different CL clients, five different E clients. If you manage to do that, you're you can like ramp up the setup to be immune up to to up to four different well to a bug that is shared by up to four different clients.

So that's really cool and I don't think you can really like configure this in other with any other approach. Okay. Um the goal of this talk and the goal that I have in mind with uh developing Veraro and working on this is to get 40% of the network onto a client bug resistant uh setup. So um I think this mainly lies on the professional node operators in on the network. Um I don't think we can really expect homestakers to be running like five different client pairs. It's a bit too much. But um yeah the homestakers don't make up the majority of the network. So I don't think it's their responsibility.

Um yeah, and if we get to this point, we never actually will get to finalize a buggy version of the chain, which is like huge a huge thing. Um because like if we actually do end up finalizing the wrong version of Ethereum mainet, that would just be a huge catastrophe and um yeah, it's something that Ethereum may not even be able to recover from, right? So yeah, this is the way we run things at Saranta. Um we run five different CL clients and uh four different EL clients at the moment. So our setup is uh completely immune to I think two client bugs at the moment. Um I would also encourage other professional node operators to run such setups. Um it's up

to you to step your game up. Um just because you're uh you're happy with the current client combination you're running doesn't mean that you're always going to be happy. If that bug strikes, then you're it's you're not going to be happy and it's going to be too late to do anything about it. So, yeah, that's it for today, I think. If you want to learn more or uh stay in touch, here are some links. Uh you can follow us and yeah, I think I'll leave it up to you guys. Now, thanks for being here. [Applause] Any questions, Sam?

does this uh or how does it affect the latency? Not as much as you'd expect. Um basically, if you're running like three clients, it takes the fastest two of the three responses. And since those client pairs all execute the transactions in parallel, it doesn't add a lot of latency. Got it. And even if these different client combinations are not in the same location. Yeah, we run ours across Europe, so across a single continent is fine. Um, once you would want to like run it across the globe, it may be different, but yeah, I haven't tried it yet. Okay, thank you.

Thank you. Um so uh if you're running an operating system um uh there there is risk that the operating system may not be maintained and uh we we choose Ubuntu LTS so that long-term support thing is quite important to us. What assurances can you offer the the community that there um uh uh this is Ver Vero is LTS? Yeah, that's a good question. Um, I think the best I can do right now is as long as we're in business, Vero is in business as well.

Thank you. And could I ask a little press a little bit further into this? How many developers you do you have working on? Um, it's just me working on this right now. Okay. But, uh, like it's, as I said, it's very simple. So, it's less than 4,000 lines of code. I think anybody could get started with it quite easily. Okay. And do you have any backing? Just our own. Just our own. Okay. Thank you so much. And we received the grant from the Ethereum Foundation just to be clear as well. But um yeah, mostly it's us funding it. Thank you so much for producing it. So it sounds amazing. Yeah, sure.

One more question. Yeah, two more questions actually. Hi. So, have you put any thought how would these react with a hleshki type incident because it it sounds to me like potentially you would be helping finalize the wrong chain. I know that's like very catastrophic uh type of scenario but maybe I know stop like voting because like uh I don't know have you put any thoughts on such a scenario?

Yeah. Um so the thing you can configure is the what we call the consensus threshold. So if you're running three nodes uh you can set the consensus threshold to three out of three and Vera would only attest if all three clients agree. So with five clients you could set it to five out of five. So if you wanted to and would want to be like really conservative, you could like up the thresholds all the way up and then you wouldn't like vote on the wrong chain in any case as long as one of the clients disagrees. We should stop. So very short question please.

Um yeah, hey this is Izzy. Uh like amazing piece of software and I hope we can support it and get node operators using it off like at least on Lido and Hoodie to battle test it out. Um my question is like about like the the locus of uh dependency if you want to think about it this way. So you use like web3 Signer basically as the only remote SER that you currently support. Are you worried about so if you get to like 40% of uh stake running on Veraro or Vero or vouch are you worried worried about over reliance on like one specific piece of remote signing software?

Not that much. Um, I think the worst case with Web3 Signer would probably be validator downtime, which I mean it's not great, but as long as the entire network doesn't run on Web3 Center, it's also not catastrophic. Or did you also mean slashing wise? Slashing wise and from like a key security perspective also. Okay. Well, slashing wise, I'm planning to add slashing protection to Vero as well. So there will be like two layers and uh in terms of keys like uh there's a lot of options you can you have with web3igner. So I think that's also quite flexible.

You don't have to like use the same way of accessing keys. So yeah thank you everyone. Thank you for your talk and yeah if you want to talk to the speaker we invite you to do it outside so we can welcome the next one.