Why Should You Run Distributed Validators

Transcript

hello everyone thank you very much so nice to meet you all my name is ocean kind I'm CTO and co-founder of obol and today we're going to be talking about should you run distributed validators sweet and before I get started I just want to give a little bit of context around who Oba labs are and what the oval network is open Labs is a research and development team based all over the world we have 25 people on the team at this point and 15 advisors mobile Labs goal is to work on decentralizing and making more fall tolerant the ethereum and other proof-of-stake blockchains

and the oval Network more broadly is basically the community of people and teams that are involved in this effort there's more than just overlabs and you'll see more of that as we go along so what are distributed validators let's start at the top so everyone here I'm more or less assuming no certain ethereum validator is this is the thing that drives proof of stake for ethereum it's basically a private key that signs a message every six and a half minutes these things are very important to keep online and they're very important to keep these private Keys secure a distributed validator is a validator where this private key is split into shares like at three or four multi-sig

or a five of seven seven of ten Etc and these nodes all then play a consensus game to decide what design then they all sign with their like partial private key and then obol and the software in the middleware combines these private Keys together to form a legitimate signature for that validator which it broadcasts to the network and it looks like a normal validator to everyone else what this means is you have a highly available validator for the first time that's a web two word for saying you can have a machine go offline and no big deal your validator stays up the same way you know Google doesn't go offline

when they update software or a machine dies validators in future should be more or less the same distributed validators makes you more resilient to software failing if you have you know a mix of different software clients let's say for example what happened last week with the kind of present prism and techu issue you'd be more resilient if you know your full validator isn't just running on one cl on one software you're more resilient to Hardware failure we'll get into this a bit later in the slides looking at you know discs die machines don't live forever so being resilient to this is important for uptime um a more extreme one is private key compromise as I alluded to if someone steals your hot wallet or your validator private key they can slash you whereas

with distributed validators if you if you do a distributed key generation no one entity has the full private key so it becomes much more difficult for someone to steal the full private key in the like minimal example of a three or four someone needs to compromise three machines to be able to get a full private key and kind of related and the last one is rather if you right now a lot of people like delegate their stake to someone you're putting extreme trust in that entity and if they go Rogue become a bad actor there's very little you can do If instead you're using a Distributive validator and only kind of a subset of the nodes become malicious so like you know one out of four no big deal your

validator will stay online and things will be okay so how is this architected how does this actually work so one of the things around ethereum proof of stake is that it uses the BLS signature scheme and what's fancy about the BLS signature scheme is that you can combine signatures of you know the same hash or the same object relatively easily you don't need access to the private key you can just you know add two signatures together and this is important because it means you can build a read-only middleware that enables distributed validator technology and what that means is you put Caron is the name of our first client it's built-in go and it

sits in front of a consensus layer Beacon API and each of the car un nodes in your cluster talk to one another and they basically just do a quick consensus game saying hey we have to sign something next slot what are we picking they decide on a hash and that's about it at that point your normal validator that's you know sitting there controlling your private key maybe it's keeping them in Hardware doing something fancy they just sign a message assuming it doesn't you know break the slashing rules and they send it to what it thinks is its consensus client that's where karon as a middleware intercepts the signature shares it with its other nodes and once any of the

machines gets three or four signatures or you know a threshold of signatures they can just combine them all together and hand them Upstream to the consensus client and from then on it you know looks like a normal validator to the rest of the network so the goal of this talk was to talk about you know should you run distributed validators so the first question I'll talk about is when mainnet and where are we in that Journey so we've been working on distributed validators since as early as before Devcon 5 in 2019 and more formally is overlabs we've been working on this since about 2021 and we've been doing devnets for the last year or so and the first major one we did was Athena we

deployed our first validator on mainnet just at the end of last year so it's running for four or five months now this year we did our Bia test net which is our second larger test net we'll look through some of the stats on that in later slides and as of about a month ago we kind of finally gave the green light for our Alpha launch which is you know Max one validator give it a try um so yeah so far we have about seven mainnet validators and the plan was to bring it to just 25 and chill there and see how things go the goal is to get more production data to see how this performs in you know the mainnet setting as opposed to in Girly

and yeah that's kind of where we're at so far and now I get to talk a little about performance numbers data and all that type of stuff so um from these kind of First Data points on Alpha you have some interesting info here I will admit I was going through this last night and out of what some of these data there's one distributed validator we have that I won't name names but it did have an outage for three days so you'll see it kind of represented in some of the the moving averages and stuff and that's the only validator that didn't have two three nines worth of uptime um but yeah I think the main problematic

data point as you can see highlighted in yellow is of our first nine proposals two were orphaned and two are missed um so as you might guess latency is an important part of you know a distributed validator for attestations it's not too bad but for proposals you have the guts of about four seconds to get a proposal out so depending on how far away your nodes are that will impact whether or not you are you know likely to miss a slot or what happened in two cases they were orphaned in that proposal was made broadcasted but it wasn't fast enough to to not get kicked out so um still getting good like data on that and mainnet it's normally okay unless you get too optimistic about how far your nodes are apart from one another

and we'll look a little bit about latency in further slides so here are some graphs in line form the first one is validator Effectiveness the green kind of drop is the smoothed variant of of three days being offline for one validator but without that outage these things are more or less pulling like negligibly different between your rocket pools your intestines which are kind of two referenced examples for Main and validators your kind of protocol and your centralized one and the other figure we have on the right are your inclusion delays the inclusion delay is a good piece of data

around you know what your how soon you're getting on chain and this particularly matters if you're slow your rewards are Downstream of your inclusion if you're getting included late your rewards won't be as high and the real big spike here I actually believe is related to the exits hard fork and trouble just after it and I have like the only smooth graph but it's a bit noisier and then on the right hand side spike is related to the like prism and techu issue that were happening recently where we last finality if you lose finality you're missing a lot of blocks inclusion distance goes up so that's why everyone is taking a step up so the data set that we have more

information on is our testnet on Gurley and as I alluded to we ran um ba was the name of our test net we've segmented them here into professional node operators and homesteakers and we've restricted to the 90th percentile or the top 10 of our home nodes and validator Effectiveness on both are pretty good as I alluded to in Gurley about 20 percent of nodes are offline so Effectiveness doesn't look like mainnet and as you can see here the network average is only 37.5 effective so 52 49 are both very strong numbers for Gurley um uptime only okay the correctness

again this relates to if you're missing slots the correct answer is is you know not always obvious depending on who proposes blocks and the inclusion they are actually quite strong for girly I believe I have them in line grass here to give a better example and this actually is talking about cluster size so a distributed validator the smallest number of nodes you can have for the fall tolerant qbft protocol is three of four but you can also have you know larger clusters so we've been doing tests being like you know does it make more sense to have four nodes and only you know one node worth of

tolerance or maybe we go up the high end and have 10 nodes and three of them can be offline and the trade-off as you might guess is cost running more nodes is you know more work but if you have you know a group and a community of people that want to run nodes and they're like yep let's all you know collaborate and run a distributed validator together higher number of nodes are more flexible and assuming the latency is okay meaning you're all in the same continent or so there is more or less a negligible difference between a four node cluster and a Nintendo cluster um the data we have here I think shows that if you look at the be a public there's more or less no difference in inclusion

delay and an Effectiveness um the beer Pros I don't know what happened with that 600 clusters there might have been something wrong um and then yeah similarly these graphs this time are on girly so they're much noisier than they were on Maines and you can also see rather than effectiveness of like 98 you know your Effectiveness for everyone is around the 60 Mark um and then inclusion delay rather than being 1.01 or very close to it it's actually about 1.6 because you're missing 20 30 of blocks so you're only getting in later than you'd expect and this is the graph I'm most excited to show and this is a validator this is

actually a validator I'm running well with three other people it's a distributed validator and it's showing the red marks are times where at least well one of the nodes were offline they had some form of outage and the green line as you can see stays High normally if you've an outage you go to zero you're offline whereas here with distributed validators so long as you know a super majority are online no big deal so I think the worst of them you can see a slight impact in Effectiveness I think that was actually my node that was offline um but broadly you can actually keep a validator off online even if you have some outages um here's kind of some screenshots of

our grafana dashboard and our monitoring and what we're showing here is that even though one of four nodes was offline this distributed validator was still able to make a proposal and yeah the yellow and blue lines on the bottom left show that it was able to do the Rando protocol and propose properly so even if you have some of your machines offline you shouldn't miss your proposals and yeah one data point I have actually relates to the beacon chain disruption Paul from Everlasting is one of our early mainnet adapters running nodes and he was very happy to see that none of our main net nodes went offline during

the finality issue granted we have a lot of Lighthouse involved and Lighthouse wasn't vulnerable but it was nice to see that you know 60 70 of the network you know Miss slots and no DVD so that's one of the things I'm pretty happy about distributed validators and the goal is to prove yes you run a four node validator keep you know four different Els and CLS in there and if any of them have a zero a day you should be okay so the last thing I'm going to talk about a little is you know what what does it look like to go forward from here so as I alluded to in the roadmap this is kind of an alpha software so far we're testing in prod gaining more data getting more comfortability within

production integrating into more people's Stacks but the current variant we call it a version one um why do we call it a version one effectively you currently can't prove at the like solidity or cryptography layer who is online and who is offline you can do it you know with your monitoring you can say hey that node is offline but proving it to a Smart contract is quite tricky um we've looked at it for a long time Don crowd back in 2019 with talking about custody games trying to figure out it's called The Lazy validator problem it's how do you prove that one person isn't being lazy and just kind of coasting and

effectively the work is to make a distributed validator protocol that is provable and if you can have proofs saying yep I was online I did take part in consensus I did submit my threshold signatures I can prove I have access to a fresh Beacon State combining all of these together into a succinct proof that you can use in an ongoing game is the are the building blocks you need to have distributed validators where you don't know and don't trust your counterparty in version one to some extent you need to trust them you need to you know blow them up on Twitter if they're offline or you need to like take them to court if things get really extreme but the long-term goal is

that you don't speak their language you don't know who they are you're just like sure let's want to distributed validator protocol works if you're offline you're losing money and it's being distributed to the people online and this research will be over a number of years it has work in attributable consensus attributable threshold signature schemes and proof of custody games or proof of access to the beacon State and there's going to be quite a lot of work in it you'll hear me talking about it basically every talk I give for the next few years and I'm very excited to kind of dig into the nitty-gritty of that kind of over the coming months that's more or less everything I have

for you at the moment I'd like to say thank you very much for the time to present to you all and I'd like to know have anyone any questions for me thank you [Applause] [Music] I see some questions all right we'll bring up here first yeah so I have a question which is directly like to you and your team uh like when running this whole architecture is there like any flaws or like cons that you're afraid of like uh at the time when this is going to go

like big are there any security flaws that you are afraid of that could happen to jeopardize the whole architecture the whole system um so I think your question was I like are you concerned about particular security concerns about our architecture design correct yeah um so in short we spent a lot of time trying to make this as trust minimized as possible really go down to kind of keep it simple stupid kind of design um one of the major decisions on that as I alluded to is making Caron be a middleware and what this means is it other than helping you create the private Keys it facilitates a distributed key generation after that

you take the private Keys away from Karen they sit in your web3 signer they sit in your validator client they sit in wherever you want so if ever there's a scenario where you know the software is compromised the blast radius should be you know liveness not safety and the so yeah the main risk right now having one middleware implementation is if it fails you might run the risk of going offline and the mitigation to that is to have multiple implementations of this and yeah that's something we're actively working on as I alluded to with a distributed validator protocol rather than a distributed validator implementation um yeah my biggest fear when it comes to

validators is being near the private Keys promising to encrypt them transmit them do anything fancy like move them from one operator to another I I don't really want to be involved anywhere near those things similarly networking is another one um a lot of projects have kind of a centralized gossip Network where all the messages flow across this is problematic because normally there's at least someone running kind of message Brokers that you know share messages around the place if they go offline your connectivity drops or ball instead each cluster establishes direct TCP connections with one another so the goal is that Obel can go offline entirely and no cluster gets disrupted

so yeah safety concerns I'm relatively okay I will acknowledge there's still liveness concerns but we're working on that both at like a networking layer and also on a protocol and implementation side to things okay I think we've got time for maybe one more question cool how's it going uh big fan of DVT I think it has a lot of potential in the future um who would you say your Target demographic is like is this for people who say don't have 32 Eve but want to contribute to to to being a validator maybe grouping up with their friends maybe each of them have five eighth or three then they find 10 other 10 other friends who have the equal amount and they can

be validators then or is this more targeted towards like LSD projects or larger projects that that want to divide up their their risk a good great question um the short answer I'd say this technology can be used by every validator and put different types of validators use it for different reasons so the homesteaker as you alluded to um at this point you know the price of a validator is prohibitive for more or less everyone um and the goal is you know not to end up in a world where only like you know people with sixty thousand dollars lying around can stake so we are very focused on allowing like the long tail of small validators to collaborate and share a stake we see this technology as the

analog of mining pools back in you know the early days of Bitcoin everyone just mined solo they hope to get a block every now and again and within about 18 months they were like this is prohibitive we need to start to share and we see distributed validators being the same thing for proof of stake it's you know everyone ran their own validator they hope to get a proposal once every couple of years and now it's like okay we need to share validators we need to kind of break them down and you know work on them in groups um however there's also a lot of reasons for them to be used on the kind of more Enterprise side of things both in the like centralized provider and for the like liquid staking protocols the centralized providers they're using distributed validators to reduce their operating costs most of them because you

couldn't have a validator go offline they built something very expensive that like runs in the cloud they have kind of ways that they can take off the keys they've never gone into bare metal because if a bare metal machine dies you just have to sit there and wait whereas now with fall tolerance you can move towards Paramount machines commodity Hardware have you know simple fault tolerance and the centralized providers can massively reduce their operating cost because a lot of them are frankly underwater because they designed them when it was 4K and the APR was 10 and now neither of those things are true and then the third person which I think is actually the one that is most important to me is the liquid staking protocols these protocols right now recruit node operators and hand you know

a really large amount of money over to a node operator and pray that things are going to go to plan the biggest slide of validators each of about 700 million dollars in the stake and lighter more or less has to trust them like those Road operators can disappear they can go offline there's very little they can do these protocols would much rather say u4 operators are sharing stake so then if one goes row goes offline no big deal thing stays online crisis averted so that's the one that you know liquid staking protocols you know people like people are using them the goal is to really decentralize them and not have a situation where one org has a huge amount of a protocol and kind of can hold them to Ransom okay I think that's all the time for

questions thank you very much and I'll let you on to the next talk thank you guys [Applause]