Holesky Validator Incident Response Call

Transcript

[Music] Thank you. Um, okay. Welcome everyone to the Alleski mass slashing uh event. We're going to try and get uh all of the slashing protection disabled on the different holleski operators in the next 10 minutes. See if we can get people to attest to and finalize an epoch. Um and then see if the network can process the slashings as this happens. Um, does anyone have a false check on like how

many operators uh we have right now live? I'm looking at Dora being seen not too unhealthy. Um, but yeah, I don't know if anyone has has looked at the number of of uh valid or proposals uh over like some longer period of time. um not having a data from a tool which was created but uh from sync aggregation I can see something hovering between 65 and 70%.

There's already slashings that are happening already. Yeah. And there are blocks. The slashings have been happening for a while. I think someone turned on without slashing protection ages ago. Oh, it could be the it could be the nether mine nodes maybe. Yeah, we've been slashed at the very beginning, but later on it slowed down significantly. Sorry, Su, can you mute?

Thanks. Um, okay. And so we were aiming to uh start the slashings at uh let me get the exact epoch number. Um at uh oh sorry at slot 3737760. Uh this is in 8 minutes. So what should happen now uh for this to to happen? We're hoping that a lot of operators are here and scheduling the restart soon.

Okay. And when should they actually restart without slashing protection? That depends. That depends on the operator. Like at Prism, they are looking into like 90 seconds or so. So, they're going to start about 90 seconds before that time. That's about the same for tech. Um if you're using a validated client it's going to be about a minute to two minutes. Okay. Is there any operator or person on this call who like runs validators and is not sure what they have to do or has question about this?

uh do we know which uh greater than 50k operators are not on this call? Um I do not think we do. Um, yeah. No, we don't. And I think maybe like Yeah. to get the uh opposite of this, like can people just post while we're waiting, can people just post in the chat? Yeah. uh like which operators are here but also

how many validators they have uh live that are um yeah that are going to be uh like the testing and and slash or I guess just a testing and we'll see how many slash okay I'll try to tally up the numbers It's okay. From what I can gather in the chat, it

seems like there's about 1.1 um or 1 million maybe 1.1 million uh validators in the chat now. And yes, like yes, and it's true. Not all of the ones that a test will be slash slashable, but we need um Oh, yeah. Not all the ones that a test will be slashable, but we do need uh 2/3 of the stake to at least finalize something. So, assume we have like 1.1 and then the total uh validators is 1.7. So, I think we're probably just

we're just under we're at 64% with everyone on this call. Let me see if we can. Yeah, if there's anyone that else. So the exact number we would need uh let me just is 116.

117 is the exact number we would need to finalize. Yeah. Yeah. Does anyone want to share their screen to actually monitor the slashings as they come in? Um I can share Dora but um I will have many tabs open and yeah okay let me see if I can open up a separate tab just for Dora Alex's note is not what is not proposing thanks oh Alex doses

Yeah. So, it's been offline. It's only 5,000 validators. Um, it is not straightforward for me to get them online. Take the results and not propose it. Yeah. Can you give the validator keys to someone else so that they can run it? if someone's able to do that. Were you asking me?

Yeah, I'm happy to. Uh, somebody wants to run them. We're definitely going on this route. Just We're definitely a go. anyone? Okay, I'm going. Yeah, it's now the prison guys are trying to restart. Okay. Yeah, let's restart this. Okay. Sorry because I'm a little behind.

I was traveling. We're going to just try to launch everyone at the same time and go for like a huge mass slashing. Yeah, that is happening now. Okay. Yeah, we'll go now. What we would like is to finalize a block as well as part of this. So, it's not just slashing people. So hopefully hopefully the ones the the the slashable validators have disabled the slashing protection and they're able to attest to a block on the valid chain. All right. We're publishing at stations.

Nice. Nice. participation is up, but I doubt very much we're going to get to 66.

Yeah, it'll really be that second epoch. I think that's going to be a better indicator. Yes, slashing hits quite not quite much now. We can refresh the page maybe. Yeah, Josh, can we refresh? But I guess UI of Dora also somehow doesn't keep the pace. I saw plenty of contestant once slashed. Doesn't help that we're already reorginging some blocks.

The slashings in the last few blocks at least looks less than a couple of eight blocks ago. There are the stations in the latest blocks which is nice. Yeah, participation is definitely going up.

What's the earliest we would know? Um, I guess it would be like six minutes after that first uh start of the like at the end of this epoch. Yeah. I think realistically the second epoch the second epoch should things should have stabilized and anything that missed that first bit should be running. So I think the second epoch should justify if it's if we have any shot at all. Got it.

If you click my link um on Dora that it shows you the epoch overview. So it also streams um he also counts attestations as they come as you process blog. So you probably have a good idea by slot 20 whether it's going to uh just justify now because of yeah there whether there's sufficient votes in the lock. It also shows estimations and currently it's about uh 43% estimated vote participation.

Yeah, let me get one up. PK does report the the balance of the slash validators as well in the participation or or it takes only the the unslashed ones all the we can probably go to the game.

Yeah, let's go to the game. Sorry, I didn't I didn't get it. PK there's some background noise. So Zoom is working. Maria, can you mute me, please? So it uses all the active balance. Yes, all the active balance. So it checks the data set if they are active and not slashed. Oh, it takes Okay. Active and not slashed. Uh no because when when a validator is slashed it gets an exit epoch uh epoch set and till then it still counts as because there are duties for these validators.

Yes. But the thing is that uh for attestations we don't count the attestations of slashed indexes. So we we consider active valid active and non-slashed in the numerator divided by every active in the denominator. Oh, that's interesting. I'm not sure if Dora has that built in. I'm checking. Yeah. So if if Dora doesn't check this, then probably it's like overly optimistic already in Dora. Sorry, what is happening to my screen

share? It's doing some transition where it like becomes slanted in like the Z-axis. Wait. Okay. I can't see the zoom screen. It's fine now. On a Mac. Is this on a Mac? Yeah. Then it's probably that feature that you have some screens on your left. Yeah. Yeah. But the problem is I need all my screens right now. Let me see what I can do. Tim, do you want me to share screen? I can do.

Yeah. Actually, Puja, if you want to share this thing, that would probably be helpful. Um, yeah, it should be visible now. Yeah, I can see it. Thanks. Okay. Thank you, Puja. Okay, this is maybe a dumb CL question or EL question, but like why why is it possible to have 84% proposals and only 54%? Is it just that like the

either we have a lot of validators running without their Yeah. If people don't enable attestations, does it mean that they did not turn off like their So one thing is that you there's not like a linear number of attestations per block to validators. So yeah, I wouldn't really try to correlate those like given the fact that there are miss slots and also it looks like there's at least one fork if not two. So it's just chaotic right now.

Yeah, the slashes are surely are overwhelming some nodes and then repacking those at the stations is not trivial. I mean, that depends on a little bit how

how that the station backer is built. But what might be a problem is that there's not enough space in the blocks for at the stations for from more than a few slots prior. Do we know is there like of the 15% that's missing, do we know like a large chunk of

identified offline validators that we could bring online or I would imagine anyone we could get is on this call right now and 15% is like what quite a it's like a huge I guess yeah 15% would be like two three client teams worth no but also I think we need much more than 15%. I mean these numbers are counting also to slash validators.

So I I I I think we I mean we would need much more than that. Maybe if teams go through and just make sure that each of their nodes is posting out of stations. Uh especially the big client teams that have got 100,000 and we just see if we've missed anything in our logs. Okay, looking at the link that Camille

sent, um it seems like um we have uh 50K offline from Loadar, 50K from RockX, 50K, 30K from base suit, 30k from Grand, 30k from Nimbus, 25k from Prism, and yeah, like there's not I don't even know if like maybe bringing literally everyone

would help this. But we can sort this page by offline if that's what I did. Yeah. Yeah. So sorting by offline there's like 200k that are that are unnamed. So you know probably can't do much but then it's loadar rockx basu grande nimbus prism. Uh then I have sorry I have checked this also in the past. I don't know if it is accurate because Bzos should be all but 10,000.

So should be 19 90k There's only 10,000 that are not up. And same with Loadar. We're monitoring logs and can you Yeah. Can you please share where you can see that we are offline? This doesn't seem at all like this is this is the page and then uh puja if you want to go if you go to the right of the offline column you can sort uh uh sorry to the right at the right uh right beside exited it's a bit compliment

no so to the left of exited down arrow to other arrow but yes down arrow right beside exited Yeah. Um, so yeah, I don't know how accurate this information is, but that's But it says here activated everything, right? Or what what numbers do we need to look at? Um, the page basically aggregates the attestations over the last three epochs. So activated shows all the active validators and online shows all validators that uh are tested in the last three epoch based on these activated ones PK. Do you know for offline? How does

that relate to exited and slashed? Um, exited and slashed are no longer activated. So, they don't count into online or offline. Separate from that basically. Yeah, that's helpful then. Um, yeah. So, I mean, we would just need to tally up any offline validators we can find. But it might be tricky because I think we're in the long tail at this point. But also like the the at surely this offline validators many of them are just that we didn't pack the attestations.

Well we don't know but yeah I'm sure that's some factor. Prisma appears there with 20,000 keys and we're all we're testing on all our keys. But as PK said, it's for fleet last epoch. Maybe it will update shortly. So we have the actual data in like few Yeah, but we have like a structural block, right? Cuz there we don't have one. We have miss slots. So we're already down some atistations. And then like the prism issue means yet more attestations can't really get on chain

when they should. So there's like a a reduced size pipe to get on chain right now. So, did we have uh did we discuss what to do basically if we can't finalize? Like, I guess we'll just let the slashings run and then just let the chain go, right?

Yeah. I mean, you can't there's much more than we can scramble egg. Yeah. Okay. The good news is that our attestation performance uh log at the end of the epoch says around 93% of attestation included onchain which is a good news in terms of at the station inclusion but not good for reaching the quarum. And so the idea of doing this was it would help validator or sorry it would help CL clients better uh target a

specific chain. Um now that we don't have this like what what do we think we should do for CL clients? Like will you all be able to stay on the minority chain for the next couple weeks? Um, should we try to like socially coordinate on like a a block on the valid chain and use this as like a base for syncing? Um, yeah. I guess anyone that's listening to this call, if they have a validator, they could um like restart like we are. And they're

going to be smaller numbers than what we've got here, but that would all help. and I don't know how long we're going to survive. It's just a matter of the size of the state at the moment. So, so we are putting up PRs now that should be merged soon to start to sync from an unf finalized uh checkpoint. So, we now at least can restart a node. We will be able to restart a node and not sync from the justified checkpoint at least.

I think Nimbus should be able to sync from from from anywhere and and keep at it for for a good while. Yeah, Lighthouse is also we also have a PR to uh checkpoint sync from an un finalized state, but we have to polish that up. We'll probably do that next week. Maybe an update from LOSA side what I'm seeing on our valid data monitors there are a lot of uh no aggregate inclusions which is the reason why the attestations are missed and then I see a lot of included in aggregate but missed so I guess it didn't make it into a block I

don't see that we are not submitting or So those two problems don't sound like making up that whole 10% deficit, 15% deficit. So yeah, we definitely need more validators online, I guess, ultimately.

So what? Sorry. So what will we get now by getting more validators online? Well, as long as no slashings start coming in, the slash we seem to be getting away with it so far. So um we just need to get up to 66% ultimately. We don't have the numbers at the moment by the look of what's happening. So there's either some configuration errors that um we're not currently spotting that we can we can fix some of our nodes or or there's just plain big numbers of validators somewhere.

Yeah. And what happens if we we cannot get a sufficient amount of validators to come online before the slashed ones I guess the slashed ones can keep attesting until they're out of the exit queue, right? No, they can't keep the testing. Okay. Sorry. So they So, so what happens if not enough validators come online and then validators get slashed? Um, at the moment we haven't really seen that many extra validators get slashed from this event. So I can only assume that no one's running a slasher that's detecting surrounds. I'm not suggesting that someone starts because it would be a mess. But um if we get the numbers, we

can still justify because it looks like we're not getting the slashings we thought we would get, right? Um so if there's specific issues like there's a couple where it looks like there's no one subscribed on a subnet, so we can try and restart some nodes and subscribe to all subnets and things like that. Um, but they're not going to be large numbers from from just small configuration tweaks like that. Yeah. So, unless we can get like

significant numbers of validators online, we're just at the status quo. And so, basically, maybe some get slashed, maybe some don't. Um, but basically we just now wait until enough of the bad stake bleeds out that we can then justify. But again, that might take like weeks. Oh, nice. And Rockx seem to be responding. Um yeah and I guess okay and we can also try to find if there's larger I I guess

yeah does anyone have a data on like whether this this uh larger set of unnamed validators has corated entities in it or I assume not. But well, the first step is just any named ones we know of that are not online. Yeah. And from there, the unnamed is going to be hard to track down. Yeah. So, RockX seems like they're getting online. Then we have a bunch of client teams. Uh we should reach out to Afrey.

Um yeah, and then a bunch of smaller operators, but these are all like NK. Uh uh so yeah, Google chain stack. Um no. Oh, Afrey is no longer running these keys. It's Grandine. Okay, so I guess Grand uh getting Afre's validators back online. Um, is Solius here and can we confirm that Grenine knows I need to do that?

I think Solius is here. Yeah. Yeah. Yeah, definitely. We will take care of that. Okay, great. Uh, Paul. Yeah. Um, if there's other validators that are that have done it like Tech who's got a mode where it runs all combined. So if if there happen to be people that are in that situation and they've restarted, it might take them an hour to get back to to actually syncing the chain because for whatever reason service restart seems to take a long time. So um just to get back to the head. So they may come online in the next hour or so, but I doubt it's going to be large numbers.

But I don't understand what's I mean why do we want I mean it's fine to have more people online but I why would we worry about this now? It's kind of hopeless if we're slashing validators. I mean each one of these slashings can be like thousands of keys give away. It doesn't seem like we are. Uh I I Yeah. So I lost my RPC. Uh do do anyone of you guys have an actual block? I mean, I see one adjuster slashing, but the number of indices there could be large.

I need to jump on. Stop liking it. Yeah, I think Alex to your comment that seems like the path forward. So, try to let this run over the weekend, get validators online, hope we don't see too many slashings, and then if we're lucky,

we'll see a block being finalized. Uh, and then if not, then on Monday we can discuss next steps. Yeah, I mean that makes sense to me. The only way we're going to get to finalization is if we get more like validators online, but we need like a lot. So, I'm not sure if that's going to happen. Yeah, let's try that. And I think polish to your comment like I think if we are not seeing slashings now then we should try and get the get the um get a block finalized and then if by Monday it still feels like we we can't um we we can't get this um then yeah is

obviously running right I mean the a slasher is clearly running and we might be just already tail of slashings. We are seeing slashings very from previous yeah very small numbers of slashings but I haven't had a look at a block yet so I don't know how many slashings was in proposed it's probably the guess here that this slashing that someone turned off the slasher and the slashings are just from sitting in the pool of clients.

Yeah, we explicitly asked for people to turn off slashes. Sorry. Why? So, I thought the whole point was to have this massive slashing so that the wait time was slower. Do they want someone to run them? Well, I'm sure someone's happy to run. We had agreed that the point was to try to slash as many people as possible so that we drop their effective balance quite faster because of correlation penalties so that we wait only 18 days instead of like the 24 that was computed. Well, if no one gets slashed in this event, then we're back to finalization and we have all of our keys and that seems like a very easy outcome, Yeah. Like if we if basically if we can

actually get something finalized, we should still try to and if that's not possible, it is what it is. But we shouldn't if we're not seeing mass slashings now and we haven't gotten anything finalized, there's not much value in in getting like a like yeah get getting the slashings now. I saw three slashings now from Dora and all the all those three are just containing one slash validators. So definitely not massive.

Yeah. So most probably then that these are in pools and not not new slashings from Yeah. Probably everyone turned off the slasher. I wasn't aware that people were asking not to turn on the slasher. I mean, it seems like a a neat outcome if we don't lose half the validators on the network because a lot of that a lot of the validators on the network are people are wanting to test these features with. So, it would be rather nice if we don't have to re-up them all and slow down any testing.

Okay, there's a chat comment about the magnet upgrade. Like I think just to be clear, we're still triaging this incident. Like we're not at the point where I think we can talk about a mainet date and regardless of what happens with Hleski like we discussed on awkward yesterday. like we need to figure out what is the testing plan, how do we allow like staking providers and others to um to test their setups. Um so yeah, we'll probably be discussing this through the next week, but I don't think we can like say anything about a magnet dates or anything like that right now.

to be able to be in a point to test, Like Yeah, correct. We can't really call it on mainetap until we can test things. No. Correct. And I think so in terms of next steps, I think we should try to get more validators online here. Um if by Monday we haven't finalized anything, we can decide if we want to run a slasher and and do like a larger slashing after the testing call. Um on the testing call I think we should also discuss uh what are the specific things we want to be testing that we we sort of can't anymore because uh Kleski is in the state um and you know whether this is uh like what can we get out of net 7 basically what do we need maybe to start a new test net for or to do some sort of hleski classic

revival so that we keep the state at a previous uh at a previous time. Um, but yeah, for now I think if we can try and get more validators back online, that is still the priority for the next couple days. Does that seem reasonable to people? yeah, it does to me. Yeah. Cool. I think we can probably wrap up the call and continue using the Discord to coordinate on next steps unless anyone has any final questions or things they want to discuss.

Okay. Well, thanks everyone. Uh, thanks Josh for the stream and yeah, talk to you all on the Discord. See you guys. Bye.