Risk mitigation strategies for node validators
ETHCC 6 · July 2023 · 17:35
Sebastian Banescu reviews large slashing incidents and the limits of general security certifications. The speaker discusses how node operators can reduce the impact of slashing events.
Speakers
Topics
From ETHCC 6, hosted by Ethereum France.
Transcript
[Music] thank you [Music] so good evening everyone um thanks for sticking around for this talk I was expecting an empty room so I really appreciate people who are here this is going to be a beginner level talk so we're not going to dive deep into
these kind of mitigation strategies I would love to talk to you about them after the talk this talk on for the the duration doesn't allow that but I'm going to go through some of the sort of step-by-step things about slashing in ethereum so the outline is you know like an introduction slow introduction for anyone who's not familiar with the topic some major events that happen which you know are not as major as D5 hacks some limitations of traditional I.T security certifications when it comes to slashing and then what you can what node
operators and and stakers can do to reduce these risks and at the end some q a so as an intro I guess you know we're at efcc everyone here knows that there's a ton of eth being stake right so the blue line here if you can see it well shows that there's a bit over 24 million eth staked at the moment and you probably notice here there's a there's quite a spike after the um the um Shanghai upgrade this year in April
and um basically at the moment we have 45 billion worth of billion USD worth of crypto worth of ether staked but this is you know like at risk of being slashed because node operators might do some misconfiguration for instance they might you know like by mistake not maliciously double sign or they might suffer downtime and this could lead to slashing there are many ways in which people could stake right here we see a nice picture from the consensus blog you
could do solo staking you could offer staking as a service or pulled staking all of these you know have different uh you know advantages disadvantages for stakers but in the end someone's operating those validators right there's there's some machines involved in this process and if they are misconfigured this could lead to slashing and as you all know as a validator in ethereum you need to stake at least 32 if and then if you misbehave you get slashed you could get slash a small amount which is like one if if you don't you know if it's an isolated event
but if you do something that seemingly is malicious as double signing you could get your whole steak slashed there are some events neither of these events involved a total slashing right and they're spread across the years we didn't want to give the exact date or any names of the node operators that were involved in order not to you know Point any fingers but they're pretty spread out they're not very frequent as they should not be um we can we can go briefly over them but if you look at this table you can
basically see that things such as the slashing prevention database persistence is very important that that is the root cause of the first event there that lead led to adeath being and lost rewards being sort of the impact other things are bugs in the client software itself right this is the second event which actually so a fixed contained a bug this bug led to validators getting slashed and a lot of them so it's not necessarily always an isolated event actually if more
validators are involved the penalties are even higher for this second one I think the penalty was not even properly calculated but I think there was some sort of like forgiveness there for the third event um there was a configuration issue right so here every default configure every node was using a default configuration and due to a sort of deterministic scheduling or algorithm in the software the same node was being pinged or same validator was being contacted and and
this led to the entire system slowing down the fifth event um had to do with spinning up two clusters with the same key that did double signing and then the last one again sort of an influx of attestations came in and this this actually led to the sort of worst case scenario which is putting the entire network in league mode so that's sort of the worst thing that could happen and it happened recently so it's not that these kind of things are going away this year right and they were more frequent back in 2020 when things
were just getting started I didn't include all of the events here I just wanted to include different ones so that you can see what the different reasons for slashing could be the penalties are not extremely high they could be much higher right and what do node operators professional node operators do nowadays many times you see them getting certifications that you often see in traditional I.T companies such as sock 2 or ISO 27001 these are very
well-established security certifications but they do not check for any kind of slashing prevention mechanisms so if an ethereum to Staker goes and gets one of these even if they have zero slashing prevention mechanisms they will get the stamp and they will be we're certified they will have a great I.T operations process for sure they will have all the things in place for whatever those certifications check for but they fall short of any web free
specific protections and at quantstep we've started developing another type of assessment which we call in web free infrastructure audit where we specifically check for things which are not being checked by the sock 2 or ISO 27 1 certification processes let's say so we're checking for having the same validator keys at the same time on two different instances these other sort of like certification or certificates like sock 2 and 27001 will check that
those keys are safe you know no one can access them they're kept they're managed really well but they don't check that you don't have two instances with the same key right again similar thing is the deletion of the slashing protection history that's not not an item on the certification list and so on right they don't check for bugs in the client software they don't check for misconfiguration of your web free signer right or orchestration when it comes to web free signing because they don't have that I mean in other companies that do I.T so there's a there's a bunch of these kind of things that that need to be
checked in order to ensure that a node operator has a lower likelihood of being slashed and if even if all of this is done right there's there's no guarantee it's just sort of a snapshot at in a point of time so you need to monitor that this sort of like security posture of a node operator is maintained over time so node operators what they should do is they should also employ a strategy and I love this blog post by Emmanuel nalepa from kiln he posted like this very nice strategy that involves like a a layered approach
but he admits that even this layered approach has holes in it right it's like like cheese slices so you see that every layer prevents some of the hazards right like if you do if you have like a git Ops validator for public to check public key duplication that's great right it's going to catch some hazards then you have maybe like this validator client for the local anti-slashing database you do one for an external signing Authority and that's sort of like your Global anti-slashing database each of these you know blocks some of the things
even like the doppelganger detection but there might be others right the unknown unknowns or like one of these layers might have itself a misconfiguration in it and that could lead to a slashing event right and you know in case of catastrophic events you want your you know your your stake to be safe while you're sleeping at night right and institutions like Pension funds really want that right they cannot afford to lose their principal investment because there's going to be Health right there's no such thing as 100 safe and secure right I think this is something
which is quoted very often in the security space but no one knows who the author is of this quote in case of a catastrophic event like I said you know you need to be able to recover your stake and think about it like this when you're staking would you let your node operator drive your car around Paris without knowing that they have insurance you know like depending on what car you drive but like would you let them drive for a year in Paris without knowing that they have some kind of insurance like there's so many things that could happen even if you you know put all sorts of things around your car to protect it
from getting scratched or whatever you know who knows what might happen right so trusting a node operator with your if is pretty similar right and what you can do of course is to buy insurance and this is where I want to introduce chain proof where like the world's first and only regulated insurance provider that covers slashing insurance we're licensed in Bermuda we're incubated by quantstamp reinsured by the world's largest reinsurance company Munich re and our most notable investor is Japan's second largest Insurance Company
so our policies are backstopped by these traditional players they're not sort of like unregulated crypto native Solutions and they're also structured in slightly different ways in conclusion I just wanted to go over all of the things that we've talked before there's a lot of ether staked right validators if they misbehave or like you know they get unlucky they might get slashed and slashing could occur due to different reasons either double signing downtime or others and if a node validator is relying on traditional certifications like sock 2
or ISO 27000 that is not sufficient something more specific to web3 is important to do and not only that they should employ anti-slashing strategies that have multiple layers but in the end there's no such thing as 100 safe and secure so in order to sleep better at night maybe consider Insurance that's it from my side thank you [Applause] any questions I don't know if we have time for questions one question
you know thank you very much I'm curious to know is there like a set of best practice already that are being implemented within the industry around like the different validators there are best practices in terms of what to do in order to prevent slashing but we don't see them as widely implemented so every validator is very different in in that regard in that regard I think the the drawback is that you cannot you're not you know the the infrastructure of a node validator is not openly visible like smart contracts on chain so they can do whatever they want under
the hood and even when we do these these engagements they need to give us access right and sometimes what we see is not so nice and we help them fix everything but yeah I think this this lack of transparency when it comes to the the staking providers is very problematic yeah it's kind of surprising because I like there is what I heard like 29 validators right now on Lido for instance and and knowing like the the community is pretty much using a lot of Open Source product I mean everything almost but you saying that actually in the practice that how we Implement that
is still like very everyone has their own Silo and it's it's pretty it's pretty close no no they they use open source software well not the software I mean more in terms of practice you know is there like a space where actually everyone share a little bit what they do and maybe being able to like giving recommendation to each other you know to help each other there are some communities but I'm not sure how much they actually because the problem is none of them can see what the others are doing so even if they don't do a good job they can say they do a good job and who's going to know right but because we want to give them insurance we need to know and then if they want insurance they
need it need to show it to us right because they're very hesitant right we reached out to many of them and they're like no are we going to show you what we're doing right thanks for the question all right thank you very much thank you foreign
[Music]