How Core Devs Crashed Ethereum

Transcript

Hi everyone. Uh my name is Radc. I am working at offchain labs on the prism consensus client and today I would like to tell you how cordvs crashed Ethereum or what happened when the holleski test net got updated to spectra. So we first need to understand small prerequisites. So valiators apart from voting on the head of the chain they also cast something called an FFG vote and everyone knows this picture. There are epochs. Epoch consists of slots but apart from that there are also this notion of checkpoints and each epoch has one checkpoint and it's the first block

of the epoch or if the block is missing it's the second block and so on and so forth. Here in the diagram we can see checkpoints for epochn and epoch n plus one. And whenever you cast this ffg vote a validator says that there's a pair of a source checkpoint and a target checkpoint. And uh a checkpoint can be in three states. It can be either finalized, justified or neither finalized nor justified. And so if a checkpoint is justified such as here in uh epoch n plus one and validators uh cast the votes from

um that say that the source is in epochn. They casted votes the source is epochn and the target is epo n plus one and more than third of viators casted that vote then the checkpoint and epoch n got justified. Okay. And then later on um you can imagine the same arrow going from checkpoint n +1 to n plus2 again with more than 2/3 of validators and that justified checkpoint n plus1 and finalized checkpoint n and so if a checkpoint is finalized then nothing then blocks before that checkpoint cannot be reverted anymore unless uh this oneird of the whole state gets

slashed And let's imagine this scenario. Um, a validator casted its vote where it said the source is at epoch 1 and the target is at epoch 2. And then later on they cast a vote saying the source is epoch zero and the target is at epoch 3. Now this is something called a surround vote and you can't do that for security reasons. If you do that then you get slashed. So um a valator would never do that and clients have something called a slashing protection which prevents validators from casting such a vote uh incidentally.

So now now that we know that uh we can turn to holleski and see what actually happened over there. So what happened was initially 0x424242442 etc which was the deposit contract address and holis. Now when we were updating to Pectra G, Netmind and Bess uh execution layer clients, they did not set the configuration value of the deposit contract to this value and red and Oregon argon did and so because of that we had a fork get nethermind and best accepted at block that was supposed to be invalid and red and Oregon did not Oregon did not accept it. Now you could

say that's not a big deal because these clients who are um have a debug can just release a patch and we can all go back to the valid fork and be happy. Well, the problem is that because they had more than 70% of the stakes. So more than 2/3 so val running these clients had more than twothirds of the stake. Um, and so the chain could get justified and it got justified. So the invalid fork got justified before they released the patches. And so what happened is something like this. So the validators on the bed fork voted the with the

source at epoch one and target epoch 2 and then after updating to the patch they would vote for the source at epoch zero and target epoch 3 and all of them would get slashed. So if they turned off their stringing protection or if they turned off their val their validators all together and would like to turn them back on they will get slashed. Apart from these consensus layer um or consensus issues, clients also had issues. So first of all from prison's perspective, we had problems finding peers on the valid fork initially because most of our peers were on the invalid fork. Then we had this issue of falling back to syncing because the

valid fork the the last block of the valid fork was before um the last block of the invalid fork. And so when we got to the head and we had most of the peers on the invalid fork, they were telling us, hey, I know a block which is much higher than your block. And Prism would try to get all these blocks and switch to the invalid fork. Now finally when everyone got back on the fork where we should have been since the start there was a problem with a very long non-fality right because the last finality uh occurred before the fork happened and it was already a couple of hundreds thousands of blocks

and this caused problem because what clients usually do is they store they store all non-finalized data in memory. So meaning that the longer we have non-finality, the bigger the resources that the clients are using and both for memory for uh CPU and uh also on the valid fork there was about 2,000 consecutive slots without blocks and this is not your regular case. So prison was not prepared for that meaning that the paths uh for empty slots were not optimized enough and this caused also a lot of resource

usage when we were churning through these 2,000 empty slots. Okay, so we have these issues. Now how do we fix this? So first of all we can fix prism. So the first thing is we added an option to blacklist bad blocks. Now it's kind of technical why this is necessary but the thing is that if you sync with something called an optimistic sync where the execution layer that can't really tell you yet because it doesn't it hasn't synced enough state yet if the execution portion of the block is valid or not. If you receive a bed block, you send it to the execution

layer saying this is the head of my chain, it will just accept it and then you will keep on building on the bed block and then after a while once the execution layer uh syncs enough of the state to tell you, hey, this block you know 5,000 blocks in the past was invalid, Prism would try this very long reorg of a couple of thousand of blocks and it would just crash. It wouldn't work. So we had to explicitly say hey this block that block is invalid so that we are not accepting it from our peers. Now secondly even if we're on the correct fork whenever you restart prism or restarted prism it started from the last

finalized checkpoint which was quite some time ago. So even though we had all blocks in our database we would still we would be on the correct fork. we would um process these blocks, but every single time when we restarted, we had to process these blocks again and again and again and it took time and resources. And so we added an option to sync from an arbitrary block and you can for example specify head and then it would just start syncing from the head. And lastly, the internal caches that we had were not prepared for such long knowing finality uh because there's there were limits in uh how much data these caches can store and so we needed

to patch them and um add some more um you know increase the sizes of them so that they can handle this uh problematic scenario. Now what about holy this is a much bigger issue right? So because these bad validators voting on the that voted on the bad fork uh are more than 2/3 that means that we can't reach 66% of participation without them and we can't uh justify or finalize the valid fork so that holleski can restore and we there are basically two options that we can consider. First option is we make all of them get slashed at once. So

we kind of coordinate a mass slashing event and uh what this will give us is that they will get these bad uh validators they will get slashed um they will be unfortunately they don't get removed immediately because that will just save us um save us but they get put an exit queue so it will take some time before they are exited from the chain and the good validators now will will get to 66%. But if we coordinate mass slashings then there's something called a slashing correlation penalty. When if many validators um make a slashing at the same time the penalties that they get are multiplied.

And so that that that that this correlated slashing is a good idea. And secondly, there's also an option that you know the slasher gossiping of slashing putting slashings into blogs doesn't work perfectly in which case if all the sl if all the bad validators get up they can vote once. So we can justify for sure then they should get slashed but if initially we have enough participation maybe we can get to the next epoch with still more than 66% of the network including which means good good validators and bad validators that haven't yet been slashed and they can vote again and we can finalize the good fork. we would still have to wait then

um until the good ones have 66% but at least we could restart our internal um state right because we could prune everything that has been finalized. The second option is to just wait in which case these bad validators they would after some time get a penalty called an inactivity leak because they're not voting not doing anything and they would slowly get ejected to the exit queue. Now more or less these options would take 20 to 30 days uh before holleski can get restored since this event happened. And so what we decided is option one is better. So

we got together tried to coordinate this mass slashing wanted to get to 66%. And got to 53. So unfortunately this didn't work and we kind of had no options left. We couldn't deposit new validators because there's a part of the spec saying that you need uh to have finalization after you deposit. So in order for the validator to become active. So you can't really deposit new validators when there's nonfality. And so uh yeah what now?

So the first thing is we need to wait until hleski gets restored if we want to continue the chain and eventually on March 10th which is 10 days after the slashing event we were able to finalize hleski. it got finalized and we could keep it as our uh consensus layer test net. But the problem was that the state that the Hleski was in was not really good enough to um to test for the real hard fork. And also if you look at it initially it had like 1 million validators and to get back to that state of 1 million validators distributed through um consensus clients and um

staking pools etc and regular users. It will take about a year for Hleski to redeposit all of these validators. And so instead of that a new test net called hoodie was introduced. And so hoodie with 1 million validators was just like a copy of Hollleski with validators redistributed as holleski had. And u yeah and now hoodie took the place of hollleski as the as the main consensus layer test net. It was just the overall I think easiest and and best way forward at that time.

So lastly, let's think about what have we learned from that. So firstly, we learned that we need better um validation of configuration. That's for sure. So either manually or automatically uh with some tooling or better yet all of these combined um yeah we need to make sure that such an configuration error never happens again. um this custom checkpoint um syncing where you can specify a custom block is very helpful not maybe in this scenario maybe in some future uh problems or for debugging. So that's a good flag for all clients to have in the future. And uh very importantly as part of our testing schedule for the

next uh hard fork we should definitely test non-fality periods in terms of can clients handle long periods of non-fality because all clients found all sorts of bug bugs during this um holy problem and this is best utilized on a devet which is uh you know not publicly facing and so even if there are issues because uh and finally can doesn't have to be stopped on a public test net and uh yeah and we can just restart the dev net whenever we feel like if if things uh don't work out. And lastly um we need some better uh incident response or some standard incident response so that in case a similar situation happens in the future

there is no panic everyone knows what to do and these uh lessons learned uh points I um I have extracted from a postmortem which is available on the foundation's GitHub and other than that there are also postmortems from client teams uh from Prism. I know there's one from Lighthouse and probably from other clients as well. Okay, that's all I had I had for today. Thank you so much.

Yes, if we have time time for questions then um I see a hand raised. Yeah, this here. Okay. Uh is there a mechanism in the clients that would maybe prevent new block creation if uh well the chain is not finalized for some period of time like for example if finalization stop then we kind of decrease the per well u the ratio we are not anymore creating blocks every 12 seconds but then we kind of create them like twice as less then again twice as less and so on so that we kind of you don't face this issue of like running out of memory as you described is Uh, is there something like that and would it help if if it was in place? No,

there's nothing like that. Um, so you need to have 12 second slots and one block per slot. Would it help? Um, there's a limit on how many attestations, so votes you can put per block. So if you have blocks, um, yeah, I don't I mean, if you have blocks spread out more, then I think um you would get less votes, but other than that, like something like you described would be a really major change to the specification because it would affect everything on how to vote, how to propose blocks, etc. So, I think it's it's it's not really something feasible to do in my opinion. Just some chains

are doing that. That's why I'm Yeah. Just saying that. Yeah. I think so. Another one. No more question. Yes. First of all, thank thanks for your talk. Um I I had been in a similar situation in a different blockchain. So the incident as an incident manager. Hope you have the commiserations from our teams because I see this might be stressful. So I wanted to ask you because you showed your slide about the lessons learned in terms of processes and these kind of things. Um

can you maybe go a little bit into the human part of things? How you guys pull through into this aspect aspects? How hard was to actually coordinate across teams? How was was the you know what are the things that you recognize that in this disaster recovery scenario that you make actually guys succeed into finding a solution. Yeah, I think it was very hard to coordinate especially the mass slashing thing where not everyone wasn't bored even like is this the good bad idea is this a good idea will it help us will it not help us and then you know how to coordinate between also not only client teams but also like uh no staking pools

etc please turn off there your slashing protection please turn on all your validators and this and that time that will speed up the recovery of the test net and Um I would also say that even though it was it was of course stressful but at some point we kind of knew that it's not going to work meaning that you know probably Hleski is dead because from the very beginning it was apparent that even if we did the slashings or weight would need this many days to recover and it would prolong the hard fork and so it was a good testing ground for non-finality thing and a potential recovery. But we kind of knew at some

point that we would more or le anyhow because Holleski was just in such a bad state already that you know and it was pretty close to a hard fork that we couldn't really afford waiting like several months for everything to stabilize and so that the staking posa could just um start testing everything again. So yeah, um it was stressful, but um I think at some point we just knew that it's uh more of a test right now and uh we're going to start with something fresh.

Okay. Thank you. [Applause] [Music]