All Core Devs - Execution #206

Transcript

[Music] [Applause] Thank you. Uh, welcome everyone to ACDE number 206. Um, obviously on the agenda today we're going to talk about what happened on Hleski. Um, that might take up most if not all of the call. Um but if we have some time left over, there were a couple Fusaka related topics um that were on the agenda as well. And then um some questions around uh EIP44's progress and uh statelessness.

Um so to give a bit of context to anyone who's listening who may not have followed what happened this past week, we had the fork uh scheduled for the Hleski test net. Um and then shortly after the fork which happened on Feb 24th, we uh realized that some of the execution layer teams had not set the right address for the uh deposit contract on the test nets. Uh the deposit contract address is different on Sepia, Hleski and mainet. And then um because they hadn't said that now with PCRA that deposits can be processed directly from the E um they interpreted a block with deposits differently where

the ones that had the wrong address just ignored those deposits. This led the network to fork um and the uh and and this issue on the execution error side was caught fairly quickly. But then because there was this massive fork where uh the three uh three execution layer clients that represented the vast majority of the stake uh were effectively uh off the main chain. Uh it was quite hard on the consensus layer side to actually get uh validators testing and building onto this chain. So for the past few days we've been trying to get there.

It seems like the chain is progressing now in a quote relatively uh smooth manner. Um but that's a high level overview of the situation and then in the agenda I've linked uh ongoing postmortem that goes into more depth about the technical causes. So I'll pause here. Does anyone have a status uh update on like the health of Moleski right now? I saw another mind before. Oh, sorry.

Yeah, go ahead. Yeah, I I can go for what we have on Nethermine side. So, we fixed our infrastructure and we are running more than 100,000 keys right now and proposing blocks very smoothly. So, it's good. And we also spin up few nodes for a better pering. And I not personally but I saw the check from one of users on discord. We are proposing like 65% of blocks uh I mean for 65% of slots we have blocks. So it is way much better than it used to be even yesterday.

So things are getting better and better and hopefully we'll reach like 75% even today. Nice. Any one else? Um, does anyone know if the the PK Dora is still on the right fork? Okay. Yeah. So, that's what I was going to post in the chat. Um, yes. Okay. So, this is still on the right fork. So, going through it,

it looks like there's a good amount of proposals coming in. Um, yeah. So, I guess has any team not been able to get back to a spot where they can actually propose a block? Um I know certain operators might not be but like has any client team not managed to get their validators back online. Okay. Prism has some but not all but yeah has every team basically has at

least a subset of their Hleski node um nodes back on back online. Yeah, Basy's got all of our valaders back online and pursuing head. Got it. Okay. And then Nimbus as well. okay. So, it seems like, yeah, pretty much every team has at least some validators back online. Um, and you know, some teams might not be uh uh yeah, not be fully there yet. Um, Trent had a comment in the chat noting that there's only, you know, like 10%

attesting to every epoch. This is basically expected. So again for context um because the the quote valid chain was a very small minority chain there's only about 10% of the stake on it. So what needs to happen now is that we effectively need to leak uh or slash the validators on the canonical or sorry on the majority chain uh until the minority chain that is correct gets back to twothirds of the stake. Um, and I personally learned this week that even if you're a validator on an who attested to an invalid finalized chain, you can

still propose on another chain. So this is why we see a larger percentage of uh blocks being proposed than of attestations being collected per epoch. okay. Oh, nice. And then Rafael says, "Okay, based on the last 500 slots, there's 44% of block production at the moment." So, yesterday we had a call to discuss uh how to effectively recover Hleski. And the idea was that once we reached about uh 50 or sorry 75% of uh of of blocks being produced, we would uh start coordinating slashings um and and

and slowly process them. And then we also said that if we didn't get to 75% overnight, we'd figure out what to do on awkward, which is right now. Um, so assuming we have about 44% of blocks being produced, um, yeah, how do people feel about next steps here? Should we start coordinating slashing? Should we wait a bit? Let the change settle. Um, yeah, Roman, did anybody follow up with Michael and DeLine from Lighthouse who who said that basically there is an alternative proposal if we enable all of the validators at the same time, we might be able to finalize quickly.

So, uh, we talked about this entirely a bit. Um it would require people like patching um their nodes to not slash. Um the idea is just to like uh have even validators who would be slashed just to test to help us finalized um without actually slashing them. Yeah. Oh, okay. Because Michael's initial message made it seem as if we don't need to ignore slashing, but we just enable validators quickly and we might be able to finalize.

Oh, interesting. I might have missed that one. Potus. Yeah. So, there seems to be a lot of confusion around what are the mechanisms to recover. Uh so first on we don't need to leak anyone. I mean leaking might recover the chain but that would take months and we are kind of certain that we're going to die before we reach that point. Uh second we can and we do not need to uh we do not need to uh uh avoid write dispatches to avoid slashings. If we just set all of the we coordinate at starting with this epoch we are going to attest all of us then those attestations

will get in and we will not slash as many validators fast enough. So that once these attestations get in which is going to be like all all attestations from every validator that sets it at that epoch then we're going to justify that epoch and we need only two epochs of testing from these validators to finalize uh that first epoch and we will I'm sorry I think it's Katya. Yeah Katya can you mute please? Yeah sorry sorry. All right. So, so the the point is the point here is that if we arrange to have at a particular epoch and spin up all validators, I mean kill all slashing DVs and we test this before so that we make sure that we know how to

do this in a timely manner. Then we will finalize among within two box and that's it. We just move on. Then the slashings can be processed very slowly and that's not a problem. But this requires coordination. This requires that every validator sets at the same time in the same ebook. Okay. And this is different from Paul Harris's proposal from yesterday where we enable only uh we disable the slashing protection only for a thousand validators at a time. Correct.

Yeah, that's that's that's definitely a bad idea. that's going to slowly uh that's going to create load on validators by uh like having to process slashings and that's only going to leak slowly validators and we're going to have to rely on leaking to finalize. We we can rely on the fact that at the stations are good as soon as they are seen and then we remove the balance from these at the stations once we've seen the slashing and since we can only slash as many valid slow many little validators per block we cannot slash them fast enough so we if we have 75% of the attesting balance at testing we are going to finalize but the problem is that well I'm I'm not really sure how uh how much we can actually expect to have

75% of validators active. Yeah, right now if we look at the blog proposals assuming that's 40 something% I expect oh uh and then someone in the chat is posting 60 65. So I assume that's basically an upper bound on what we can expect for participation. Um yeah, so we need we need a buffer well over 66% if we want this to to work. And I honestly think this is the only way we're going to save the chain.

Okay. So, if we were to do this then like yeah, the next step is figuring out how do we get um how how do we get to that level? Um, oh, and then Jim is saying that this will only work short term because over a day all the validators will be slashed. Um, and that the lecture allows for many validators to be slashed in a single slashing due to change at the station format. Does that change or something? That's my that's my understanding is even if we bring all the validators online we can start seeing high numbers of slashed validators happen relatively quickly.

Um I mean am I missing something there? But you know again with the with the electro attestation format you can slash thousands of validators in a single test to slashing. Yes, but because of shuffling, you won't have enough in intersection in two other stations to slash a lot. So, you have you'll have just a few uh valid a few like maybe a 100 validators or maybe more, but it's it's really relatively few validators in the intersection of two slashable at the stations.

This is why it's going to take time. So, we don't really I mean, considering that we had justification on the other chain, there's going to be a number of attestations out there that are slashable that have a large amount of of validators in them, right? So, I mean, I I get it. It's not going to be like a perfect overlap, but it's probably still going to be a a relatively high number of validators potentially being slashed each block. I mean, even even if it's not, say takes a week, you know, we're still going to be at the stage if it's a week or a day, it doesn't matter where enough validators have been slashed and we've lost again, right?

Yeah. Yeah. So what was P photo saying is that we can finalize but then uh when once we slash everyone it will happen anyway but it will take time. Yeah. So so what we want is to finalize something one epoch and that's it. We if we lose finality later that's not that much of a problem. The thing is that once we finalize then we're going to alleviate CL clients because we can now checkpoint sync on a on a closer checkpoint. It's fine if we lose finality later, But I think Jim has a point because if we can't if I mean if we can pack many attestations in the first couple of epochs, then we might actually lose the finalization

in the second epoch already, right? So you need to have enough of the stake or the like yeah stake weights to uh actually finalize over two epochs before you go back down to 66% uh due to slashings. Right? So we want to we want to process two at stations from each one of these validators. Right? And because of shuffling again it will probably in the worst case scenario it could take like a thousand slots because we have just one slashing per slot you know I mean like to slash all validators that are slashable

it'll take a long time and what would happen if we if if slashing happens too quickly so say that we get everyone back online they attest to like a first epox slashing happens too quickly, you're going to have to leak those validators. So, if we slashing happens quickly enough that we don't count the slashed capitals, the slash balance for the current justification, then we're not going to reach what we need and we're going to have to leak process all the slashes like exit those validators and then once like all of the active balance actually goes down, then we might justify later on. But I think that's going to take a long time. And was that basically our plan

like our hope to finalize immediately? The hope would be let's finalize immediately and then we'll leak and that we don't mind leaking after we finalized. Okay. Uh wait why it will take it will take nearly the same amount of time to leak out. a bit less because of correlation penalty will take place 4,000 epochs after the slashing event but we will not be able to finalize during this period of time which is 18 days and that's I guess the worry right if we're suffering the way we're suffering at the moment after basically 4 days of finality do we think we're capable 18 days or 21 days depending on on on kind

of where we end up right if they if they act because bearing in mind the other point is the validators that aren't the testing are in inactivity leak at the moment. We can reset that clock by getting finalization, but it'll start again. So, inactivity leaks about 21 days to get to 16 E. Um, as as you mentioned, ML, if if we slash lots and we're 18 days roughly before we get to the, you know, the correlation penalty. Um, it seems like we're better off getting to the correlation penalty sooner. So, I understand this idea of of of slashing slowly and similar, but it seems that if we actually are going to be in trouble regardless, it sounds like if we get the vast majority, if you get a third of our validators slashed, right, 18 days later, their balance goes

to zero, which means they're not part of active balance, which means theoretically the remaining ones should be able to pick up a little bit quicker. Otherwise, we're looking at inactivity leaks of 21 days or or similar before it even gets to 16. And that still counts as active balance because they're only exiting and not exited. Um I I I you know I get I get what we're trying to do in the next day or two. I don't really get what we're going to be doing for the next week or two after that. Yeah. But Jim, the the problem we have now is it's uh it's an issue that we are still getting a lot of batches that are based on this invalid branch because peers are serving those batches and then like syncing in that scenario and starting a client that is trying to sync

from the last justified checkpoint is is the problem. If we finalize, we are going to be in a similar scenario, but now every batch that we get is going to be a good batch. We're not going to be blacklisting peers because every peer is going to give us a good batch. So, so that's I think the best we can hope now is we finalize something on the good chain then that that'll alleviate like restarting nodes will sync immediately from checkpoint sync and then we leak them slowly and we hope that clients will hold on the single good fork.

Understand? Okay. Yeah. I mean that makes sense. Um, as we said, if it's if it's the best we can pull off, that's that's okay. And I guess the other thing it does is it buys us time to look for clients. I know Lighthouse, for example, have released a new version or a new patch recently that gives them more ability to to handle, you know, longer term lack of finality. So maybe if if if that's what we're doing, we're buying the time to allow us to do that and and as you said, getting a reset to avoid more forks. Okay. You know, that that that's reasonable. I just think it's, you know, it it it's probably important to make sure it's clear to people though that it's not like, you know, we get to finality and we're done. It's it's still a long road from there.

We can keep fighting once we get to finality. yeah, Terrence, what does the social coordination look like to have everyone turning off slashing DB at once? Because I don't think this is something that we have done before, right? and something we probably want to code this assumption into the code itself because if everyone does it like manually that we may miss a few epochs here and there and I I I I don't know maybe we don't have to have have an answer for this code but this something to think about maybe we can carry this off offline and the idea for the slashing DB is that we want to get up sorry just to make

sure I understand we want to get up to a point where we have above say you know a a large margin above 66% of validators online and then in a coordinated way we get all of them to turn off their slashing protection. Is that correct? Correct. Yeah. Okay. Uh yeah, Enrico, I was just thinking about uh this DP eraser. So this should be fast enough and well coordinated in a way that clients should be able to actually do that without even restarting. So I don't know if if clients require restarting

then we end up in a messy situation where we could start losing blocks and we are not um yeah healthy enough to to to work anyway. So that's probably a conversation for well there's three three aspects right we've got remote well two aspects we've got remote signers which are durk and web3 serer and we've got local databases so um anyone running with durk um it can remove we can remove the slashing db shut the process down remove slashing db restart that process end to end is probably one that takes about 5 seconds

something like that right the startup up times relatively minimal because um there's like a batching system inside um inside D to to basically take all of the individual keys and encrypt them under a single entity. Um does web3 signer have something similar or can it remove the slashing DB hot? Yeah, it's more clients that are single clients running DB um uh in a single process essentially. So you have to recycle the beacon chain itself and and these kind of setups are a little bit clunky.

Yeah. For example, restart would take a very long time. So is that a restart? Would that restart for you be validated client only? Would it be the beacon node as well? Right. This is just this is just restart of the validator client. So it should be fast. Well, depends on your setup. If if you have nodes that so we have to migrate all the nodes to have separate processes. So you have validator client and beacon node. So if you want to restart, you can restart validator without losing the your backend beacon node which is probably um so we need to migrate all our setups to a to a stable situation where we can do

this quickly without restarting bigger nodes. If you had to restart as is, how long would it take? Well, it takes a situation where you you probably are going to miss town slots initially. So it it depends currently. So I think that a bigger node will become fully functional hopefully in in few slots but it could happen that you can be slower. I mean that doesn't seem unreasonable to me. I'm not sure we're going to coordinate down to you know within a few slots anyway. a few slots. We're talking about half a minute a minute. I mean, people are not going to coordinate

exactly to that point regardless. So, I think, you know, as long as we're not in a situation where someone is individually decryptting each of 100,000 key stores before they before they start their validator client, we're probably in a reasonable place. As long as the beacon nodes again have got suitable peering when they first start up. And obviously, we've got a list of that. Everyone restarts in the same moment. You could have even problem appearing. Also, the safest way is to have separated setup with two different Yeah. bigger node and validator client and it will be much much easier, more stable.

So, we probably need to get a we're going to need some document to to track this. Again, you know, mo all the all the cord devs out there who are running 100k validators each. We we need some vague idea from them about a if you restart, how long do you think it'll take? B, if you restart, does it require you restarting your beacon node as well? Because that's a very good point that if obviously everyone sits there and restarts their beacon nodes at the same time, all the peering will just vanish. Yeah, Paul. Yeah. So there are some features that we've proposed and we haven't really merged them yet luckily which are if you catch a slashing event a slashable offense on the flight like for example

double proposer uh then submit the slashing immediately so do not rely on the slasher and then the other one is if you catch a slashable offense by one of your validators that you're serving then stop serving everything from that node that will kill us if any client has actually implemented this uh and it's lights. Oh yes, we have it. I need to check if it is default on. I guess no. Hopefully. Yeah, if we have this defaulted then anything we're planning it just goes to trash from our side. Just a note if anyone's running what we call ESD, the emergency shutdown demon, that's exactly what

that's designed to do. But it's an external process. It's a sidecar on purpose. So if it is out there and people are running it then you can kill that then that will obviously stop um our nodes or similar from from shutting down. Um but yeah that that's only obviously for applicable for people running that. So I guess the other option here so like the reason we're doing all of this is that clients have a strong assumption around like using a finalized block as a a sort of checkpoint on which to build.

Um the other I guess path here is is it possible for CL clients to effectively set an arbitrary block um and and use that as like their starting point? uh the same way I on the execution layer side where we have like this set head feature where you're able to like force your client to a minority chain or like an I guess postmerge an arbitrary chain and say like I want to just care about a chain that has this block in it. Um but would it be possible for CL clients to coordinate on a specific block and say we will just treat this as like

finalized for the purpose of sync and peering and all of that rather than um like getting to going to your peers for because like yeah There was a comment in the chat as well around like you know what would we do on mainnet and then like my sense is if this is already hard to coordinate on hleski it's extremely unlikely we do something like this on mainet whereas you could imagine socially coordinating around a specific block or something and and uh use that to uh

yeah use that to effectively think uh or to be like yeah anchor for sync or are there other options than pis's plan basically? Is it okay to say it's a bad plan but it's the only one we have? I mean you know it's it we're we're in a bad situation, right? I mean there's you getting finality is good because it gives us everyone a point to work from.

Beyond that at some stage these validators are going to go away either being slashed or leaking out. So I think yeah you know like we said it it doesn't get us to some sort of happy ground immediately but it's it's certainly the best thing we have to do at the moment. Thanks uh Terrence. Yeah. So like pro plan doesn't make anything worse, right? Because if it doesn't work out for say we screw this up, then we're still back into the nonfality thing and then we still let the validator decal one way or another. So yeah, it's just additional step that we have as a guarantee that we can start with a nice finalized checkpoint without having to black this uh pierce from from the beginning.

Okay. So if we wanted to go about this then yes we need to coordinate with all of the different Hleski um Hleski node operators um have a clear guide for what they need to do. Um and then in parallel to this, we also need to bring up more of the existing Hleski operators and um yeah try to get the stake back uh back to well above 2/3 online. Um what's I guess in terms of actual like things to do like I assume the main thing is to just have a clear um have a

clear guide for how to disable slashing protection for whatever setup you're using and then agree on a time to do this and yeah have people do it at the same time. Is that like the right approach? Um, and then what's the timeline? Yeah. So, so I saw Merik's doc has some instructions like I don't know how comprehensive this is. It is. Um, does anyone have a link to that Andy?

And Andy? Oh, thanks Merrick. And then what's the timeline on which this needs to happen? hopefully today. Today, tomorrow hopefully. Should we like Yeah. Do you think there's value in scheduling it in advance? like should we tell people to do this at 14 UTC tomorrow?

Well, well, I think there are two steps, right? I think there are two step. The first step is is that to know there's is sufficient validators online and proposing block. The second is that figure out that epoch number to do this. But the question is that are we at the first step yet? Are we comfortable that there's there's sufficient block? So if that's not then we cannot decide the second step. Right. But then yeah, can we make like Yeah, I I guess as we get people online, it would probably be make comes easier

to just say like, okay, this is when we're planning to do it and this is like your timeline to get this up and running and worst case we can abort. But I would I would lean towards trying to like packaging the comms and be like, "Okay, you need to do this by this time and this will happen then." Um yeah. Uh yeah, Salius.

So, so if I understand correctly, the the key idea here is to try to to finalize again just to uh to help clients to function better. But the the problem is that after this reset assuming that it will go well but I don't think so. I mean there is just too much of coordination uh and the setups with slashing protection is is quite different uh uh among clients and signers and so on. So it's it's really hard to get it right especially by tomorrow. But the the key thing so if we reset that uh uh we finalize shortly and then we again go to

non-fality uh period and uh this only helps us uh by a few days of non-finality because it's uh like uh it happened uh like a 3 days or so ago. So we just cut uh three days off that uh bad period and then we will have a long period until we will get finality again. So I don't see that it's uh it's actually meaningful uh because we will in any case we will have a a long non-finity after that.

So yeah, I mean yeah, I just don't see that it's going to to help a lot. Uh even assuming that it will it we will finalize soon. Yeah, I I agree that this is most likely uh going to fail and this is why I actually support what Tim is saying of like trying to schedule this for like a time today and say tomorrow 14 UTC we are going to start slashing ourselves it doesn't matter if we reach 75 or 66 or 50 it doesn't matter because uh even if we do this at a coordinated uh time or

roughly coordinated time and if we don't finalize At least we're going to get the correlation penalty to apply and we're going to just see if we can manage all of the slashing and leak the fastest possible that we can at the current situation. So, okay. Okay. And so just to make sure I understand this means that if we try to say schedule this for tomorrow, if things go well then we've finalized and it allows clients some breathing room to uh build on top of a finalized chain. Um, like Solius was just saying

though, it'll be kind of short-lived breeding room because we'll get to a non-finalized state probably minutes if not hours after that and have to stay there. Um, if we don't finalize, then it means we're effectively just moving through with the slashings and waiting on the leak. Um and we feel that the like small chance of actually getting everything finalized and and providing some breathing room to clients is worth the coordination costs.

Is that correct? Well, it's it's just that I mean in the worst case scenario, what's going to happen is that we're going to slash a lot of people and this actually also helps with the leaking. Yeah. So, so why, so I guess if that's true, then why wouldn't we just start slashing like right now? Um, because if we get if we get to if I mean if we try to reach a high percentage enough that might allow us to finalize, that's going to help us. But if we start like leaking now, if we start like slashing right now, we are certain not Yeah. And and my point is that capital.

Yeah. What is the when you say it's going to help us to have finalized like you know there is a finalized block that's like probably 5 days old at this point and clearly that's no longer useful. Um what yeah what what I'm trying to understand is what's what's the extra value of a finaliz the extra value is that we do not need to start banning clients when we request batches. we are never going to get a batch from an unfalized section that uh is invalid. So currently when you try to peer and you try to sync a node, you spend a lot of time until you find good peers because there's a bunch of peers that are still trying to feed you bad batches.

If we finalize something further down, even those bad peers will no longer serve those batches. And I guess sorry and then one of the assumptions here is that the peers right now fitting you bad batches have run the like buggy execution layer client. Whereas if we finalize a new block on the correct chain, it means that even though we're building on a non-finalized chain, because we've patched the execution layer clients, everyone sort of agrees that this nonf non-finalized chain is valid and there's no longer like a competing chain. Is that correct?

Yeah. But yeah, but we're still peering with those guys and those guys are still feeding us those batches. Yeah. So, but if we finalize then you're able to say, "Hey, here's like a finalized block on a chain which is correct and therefore you can ignore any peer on the justify chain." That's okay. And moreover, moreover, peers that that are on that other chain either they will switch and even if they don't switch, it doesn't matter because we're no longer going to even peer with them.

Yeah. Okay. Okay. So, finalizing basically like Yeah. Finalizing a host Petra block which is valid um allows us to to manage a peer-to-peer network better. Yeah. Um Yeah. Let's do Mikuel and then yeah Daniel we should uh yeah we should do your question as well. Yeah. Uh but yeah, Muel um I just wanted to add that if we finalize something and then um as soon as we can slash everyone after finality, we will have like roughly 18 days of non-fality versus like uh if we just uh leak out uh nonparticipating what there

is now, it will probably take 6 days longer. than this 18 days. So I don't know how much of additional stress clients will take in the uh second case in the case of leaking out versus um like this short path to you know um to recover. I don't know if it matters, but this is something that might be substantial like 18 days versus say 24 days of nonfenality. Anyone have thoughts on this?

Um I think I think I think the numbers check out, right? I think I think that that's it is the right thing to do if we once we've got a checkpoint that's passed the fork. Um I think yes slashing everyone as quickly as we can makes sense because the other obviously if we get a third slash within the 18-day period then as each validator comes around it gets slashed to zero which obviously removes the active balance entirely which gives us better chance to to finalization. If we're leaking out even though we leak out after 21 days we'll hit 16 or whatever. um you know they'll still be around in the active balance because they're exiting not gone. So yeah, absolutely. I I mean I know some people are saying we should we should slash slowly. Um I think it makes more sense

to slash quickly because as as as um you know as we just heard um we are going to be better off with a shorter period of finality if we can slash more balance or remove more balance from the active balance faster. And slashing seems to be the better way of doing that than inactive. So, does anyone disagree with moving forward with this?

Okay, it seems not. Um so next step are one to have like a clean docu uh that explains how to disable the slashing protection for every client to reach out to everyone. Oh sorry radic. Yes. Um, so my only concern here is I think Dustin wrote this also on chat is you know is this really something that we can't h can handle in mainet even if it happens meaning that can we really have 18 days of non-finally on mainet and not declare Ethereum dead because if not because if this would require social

coordination and reverting because 18 days of finality is not something that realistically can happen. Then even if we recover Holliski right now, we still probably would need to think how do we change the spec I guess of Ethereum I think so that it's even not possible to have 18 days of non-finality. I know there are like you know points for that why uh we can't slash maybe too fast but if such a situation like here is also not acceptable mainet then we need to think you know how do we change the spec so that we don't get into a position where we basically ethereum is declared dead and we need social coordination meaning that even if we

recover holleski now and we need to change something in the future to prevent even what happened to holleski now then it means that you know all of the work that we are doing now is really not relevant to what uh can you know to to any you know future like um you know because we still need to like make big changes to Ethereum so that we don't have this such sort of scenario in mainet. So I guess my my basic question is like even if we recover halleski can we reasonably expect that if such a thing happens on mainet people will allow us to recover mainet like this. So I guess yeah there's two things here.

One is um are there changes we should make to the mainet spec based on this incident and I agree there probably are if not at the spec at least at the you know the client level. I still think though that going through and actually recovering hles is valuable because it allows us to like find all these discrepancies. Um like one of my takeaways so far is like yeah the ability of CL clients to sync to like non-finali arbitrary like non-finalized states seems extremely important and you know that is something that uh could allow us to handle like 18 days of non-finality better. So you know cons

like and and we can get into the whole conversation around like what's the like economic model around 18 days and should that be changed but if you take this for a given that like there are cases where you would get 18 days of non-finality on mainet then we want clients to be able to robustly handle this clearly today there's suboptimal in that respect and there's things that we need to do like for example we could not coordinate all of mainet to uh agree to disable the slashing protection at the same time or to all get slash at the same time. Um, but this kind of buys us breathing room as we're working uh through this issue.

So, I yeah, I I think we're going to need to have a follow-up conversation around like, okay, what does what did we learn from this and how should it affect like the clients and the mainet spec, but I don't think the work of going through this is wasted. Um, because at least so far it's provided like some helpful learnings, right? But one last thing maybe because before I let others speak is that pus wrote on chat that on mainet people will not just allow themselves to be slashed. So the thing that we are doing now is purely possible on test net because just I don't think I don't expect everyone in in on the mainet saying yeah sure slash slash us now you know you're not going

to wait for for the chain to finance you can slash us so that you can recover the same way you did with holleski and in that case yeah this slashing right now that we're coordinating effectively is a way to handle clients not being able to deal with unfalized forks as well as they should. Um, so you can think of this as like, yeah, because this is Hleski, we get to do this thing that gives clients breathing rooms, but we should not assume that we could do such a thing on mainet and therefore we should make the clients more robust, but we're not going to make the clients more robust in the next 24 hours. Um, yeah. So, so I think we like yes, we

we're not going to get a perfect replica of what would happen on mainet, but I do think it highlights some of the failure cases we would see on mainet and you know that we'll need to address and fix in in the next few months. Um and and right, the answer to your question is basically to review inactivity league. Um because if someone's not going to slash themselves, that makes sense. But equally, um you know, if if they're going to burn their ether down by not attesting at all, then that's the other side of that coin. So, and that's that's a lot of the conversation we've had in this hour where we've been saying do we let things leak out or do we slash them and potentially yes bringing you know increasing the inactivity leak so clients leak out within you know 3 days rather than 18 or something is an option

and it's not a cost-free option because it does uh increase the ability of a malicious actor to um make the chain inactive and obviously cost everyone a lot of money so it's it's not without cost but yeah I mean there are certainly things that that can be done. Um, but I I agree with Tim. You know, it's worth doing this anyway because if we can survive 18 days of nonfality, we could survive five or three or whatever we choose afterward. And so it makes sense to to do this anyway. But yeah, I mean, so certainly we need we need to have a discussion, I think, in and around if it makes sense for us to carry on within activity league where it is or if we should change it.

Um, porus. Uh, yeah. So, I added to the agenda something that you mentioned already. So, I don't know if you want to discuss this later or if this is a good point, but you you just mentioned this issue of like of syncing from an arbitrary Yeah. Yeah, please. Okay. So, sorry. Actually, sorry. Before we dive into that though, I just want to finalize the plan. Um, or Sasha. Yeah. Sasha, were you gonna comment on like practical next steps for this? Yeah. So I lost internet for a bit so I missed some things but I just um want to give the lighter point of view here in the sense that um even the sort of the non-fality leaking out over x number of days plan is not ideal for us because we

uh have not been able to test the actual hard forks. Right in between a hard fork happened and our oracles basically stopped working because of no finality. And so it'll be hard enough to disentangle bunker mode for us without trying to figure out if anything broke or didn't um because of the hard fork. So what what what that leads us is um and it's not ideal at all. But from our point of view, if Hleski can't be saved, um then waiting for a a full Hleski-esque test net to run the hard fork against is by far the safest option. Um a shorter live devet is better than nothing but we won't be able to have tens or hundreds of independent node operators interact with the

protocol or be able to set up all the parts all the monitoring all the checks all the infra and even if we could we might not see all the different types of event cases we might see as a result of full routine operations. So like the thing about Hleski for us is it's it's more like a staging QA environment rather than a sandbox. I just want everyone to sort of have that uh context there, right? And yeah, I think we uh yeah, we can have a a a discussion after around like okay, what do we want to test? Like we clearly like lost some testing capabilities by losing Hilleski.

The DevNet will help with some of that but not everything. Um I do think we want we probably want to yeah have that as a separate discussion item. Uh yeah, Roman uh would launching a loner live test net within the next couple of days help in this case. So I think one challenge as I understand it for like the Lido situation is that the DevNet that we're launching is probably going to activate Pectra quite early on. uh for something like Lido and probably a bunch of other projects uh they'd like a a test net where like they deploy all their infrastructure which was already the case um on Hleski and

then you know go through the fork. So there there's like a coordination difference here where like DevNet 7 will allow us to do some Peterra testing like say validator consolidations and whatnot in a like postspectra environment quickly but it's different from saying yeah here's like a full staging environment for a bunch of projects that are already set up that are already using it and now they get to go through the fork. Um and then Sepia does not provide uh that ability because the the validator set is is uh is gated. Um, so so I think we have to recognize that we've lost some ability for developers to to test on Alleski and not least of which is that, you know, we're never

going to go through a hard fork again on it. You know, it's it's it's in electro. It'll stay in Electra. We're not going to come out of that. Um, but yes, I mean, DevNet 7, I I believe we've still got about another 4 days before it upgrades from um from Denkun to to PCR. So there is some time there to do that. Um I recommend obviously you know for Lido and for anyone else who's who's in that same situation to do what they can in terms of of getting stuff up and running on DevNet 7 prior to the the upgrade. And obviously if ultimately there's not enough time or similar we you know we could create a Devet 8.

Again, we understand it's not going to give you all of the users that you'd hope for, all of the tooling that you'd hope for, and similar. Um, Devet 7 was basically a response from yesterday to to try and we're trying to get people what we can, right? I think what I would say is if if there are situations that DevNet 7 does not provide for people, if it's upgrading too early or if it doesn't enough, you know, um, enough ether you available to you or whatever else. I think, you know, just just let the community know and we'll see if we can do something about it. Um, you know, we recognize Heshki is a is a missed opportunity for a lot of testing.

I think we need to to do what we can to to patch it up where we can. Um, but again, recognize, you know, we're never going to be able to roll Heshi back to preh hard fork and upgrade it again. I think I think that one's just not available as an option. I don't know, maybe we can have Fusaka go on Hawaii. That'd be a sign we've done well. and and yeah, like there are different options here. It's like maybe we actually start a new quote like DevNet 8 and just fork the Hleski preetra state and something like that. So like there I think there's there's options here that we should consider, but I want to separate this from the short-term like save Hleski discussion.

it doesn't seem like there's any more objections to pus' plan. So just to reiterate to make sure we're on the same page, what we would need is shortly after this call a doc that explains how to disable uh slashing protection for all of the node operators and which versions to run to get online. Then we need to coordinate on a time okay we need to coordinate on a time for which to happen. Um, I suggested 14 UTC tomorrow, but I see there's like the EPBS call and the beam call. Do people want to do 15 UTC instead, or the people prefer to just It's not important, at least for the EPBS people.

What do you mean that we don't mind if it's if it happens at the same time? Okay. Yeah. Yeah. So, cuz like you you are the PBS person. So, um Okay, then. Should we just do 14 UTC? Uh we know this time works. Any objection with that? or like I mean like yeah Terrence and POS if you are going to run the EVBS call we can just wait an hour later and do 15 UTC. I think it would be good to actually have people's like full attention for that one hour. So if it's easier to do that at 15 UTC, I think we should 15 UTC work for generally all the client

teams. Um and I think it would be good so for tomorrow to have one rep from every client team who's knowledgeable about how to turn off the slashing protection there. Um maybe what I would do is like yeah we should probably pick some time where like we agree you know say like the call starts at 15 UTC that like you know 10 minutes after that people disable it or something like that. I'm not sure what's the actual timeline that we want here in terms of like time to disable the the slashing protection and and and all of that. Um but like yeah 15 UTC people would be

expected to come in ready to do the disabling and then um yeah like I I don't know what what ballpark is like the the right amount of time like is giving people like 15 minutes to do this correct and then we spend the last 45 minutes observing or something. Is that reasonable? Okay, I will take the no objections as reasonable. Um, and then yeah, we can observe how the chain goes and and take it from there.

Um, and yeah, thanks uh Phil for posting the doc. So like I I think uh this doc was maybe a bit uh may may not be fully up to date. So, I um yeah, I I I think it would be good to just make sure that we have a doc that's like comprehensive and that that has the uh the latest information. I can put something together right after this call and share it in the chat and then have um all the different client teams kind of uh post their their information there. Um, can I steal a minute to talk about Solia? Um, it's

Yeah. Can we just finish a week away? Yeah, I I'll do Sepia next, but let's just finish just to make sure we're on the same page. Anything else just on like the short-term plan for Hleski? Does this address like everyone's concerns and issues? Okay. So, let's do that and then yes. So, next step I think there's other stuff on the agenda but I do think given we only have half an hour left talking about Sapoleia um probably is what makes the most sense. Um we like Jim was saying it's scheduled for next uh Thursday I believe. Um we

have effectively a week until the fork. Um or sorry next Wednesday. Um so we have we have a week before the fork a bit less the even though this like hleski issue has devolved quite badly. Um the quote trigger incident on the execution layer side was quite trivial and easily patched. We have patched releases for it. Um and so right now on the E side, uh if you've run one of the three clients that had an issue, you would need to upgrade.

Um but if you're running or Aragon, you would not. And then on the CL side, I'm not sure what that looks like. um if clients have new releases and whatnot, but Sepia does have a fairly small validator set. So to the extent that there are new releases to distribute, we can probably get the validators to upgrade uh quite soon. The um and then if we did decide to delay it, then we would need everyone to also upgrade to a version that removes uh the sepoleia activation time. Um, so how do people feel about keeping or removing the current time?

I think yesterday we Yeah, sorry. Yeah, yesterday we we sort of said, has anyone got any good ideas as to why we should not upgrade? Um, is there anyone out there who can come up with a reason for us not to proceed with Well, that's exciting. So, I guess obviously people are watching, right? So one thing I think we should see and we should confirm every EL team should post uh their configuration for Solia so they can be cross-cheed either reference into the

code base of the current release or wherever they store these files wherever they store this information. It's worth manually cross-checking this across all teams and any external observers who want to look to make sure we've got that. Um there is a polio telegram group. I don't know if you want to have something on the discord as well or instead. Um but certainly that information should go out there just so everyone can gain a level of comfort that the problem we saw in Hleshki will absolutely definitely not happen on. Um obviously we need to upgrade the sepoleia execution layer clients to the updated version with these patches.

consensus layer teams. It would be great if you could consider if you want to stick with a current released version preki um and pre all these sort of minor patches that have gone in and similar or if you're happy with going with a later released version if there is one and again con confirming either way. Um the upgrade is 7 a.m. roughly UTC on Wednesday. So it would be good to get that over the next sort of 24 48 hours so teams can upgrade as required. I think we need to do much quicker than this. Like we should obviously update these three EL clients.

Does any does any CL client need their release updated from what's in the blog post? And if so, can you get a release out like today or tomorrow at the latest? But if no one voices it now, I'm going to update the blog post after this call and we'll say like, yeah, these three EL clients need to upgrade. We'll communicate to the Sapoleia validators. Um, and it seems like Limbus, Limbbus, Lighthouse, Prism are good. Is there someone from Teu and Loadar? Not necessary for TEU. So, Loadar, are you guys good as well? Okay. So all the CL

or grand yes. Okay. So great. So then I'm only sorry for load starts 127.1. Is that what's in the blog post? It's 1270. I just saw and you're saying 1270 will not work for Sepia. Uh there's like a weird edge case which might not be triggered if you're block proposals or but it's safer to update.

Okay. But I guess is it recommended or mandatory as an update? Mandatory. Uh yeah, I would say mandatory. Yeah. Yeah. Okay. Great. So for those four clients, Loadar plus the three CL's who had an issue, I'll update the release. sorry, Loar and the three ELS that had an issue. I'll update the the release. Everybody else will stay the same. We'll keep the time stamp the same for the three consensus layer, sorry, three execution layer teams. Um, can you please post something in the awkward devs chat to just share like your current config for Seapoleia? I think that would be good. And then we're not

going to get a automated test uh check done in the next 24 hours, but at least people can verify that uh the configs are correct. We've already had some folks double check. Uh I think Daniel's gone through all of the EL code bases to to double check, but uh it'd be good to just yeah, post it in the Discord. Um Felix Oh, you're on mute.

Oh, we can't hear you, Felix. Okay, Felix is gone. Um, sorry. Yeah. Anything else on the Solia upgrade? Okay. Um, yeah. And I guess also worth noting like one thing like we will have people from Panda Ops and other fine teams monitoring the update as uh as as it

happens. Um so like yes if there is anything that goes wrong like our response time will be uh hopefully quicker and better coordinated than it was this time around. and yeah, Justin, Basu, Nethermine, and Guest, I assumed were updating like the three clients that had an issue with the original uh smart contract addresses. Yeah. I'm just saying that the blog post needs to be updated. Yeah. Yeah. Yeah. So, this is what I was asking earlier. After this call, I'm going to update it for the three ELS that had a bug and for Loadar and then leave all the other clients unchanged. Yeah.

Um, and I wanted to wait to make sure we didn't delay Sapoleia and then, you know, we'd have to do a whole different type of announcement. So, um, yeah, I think we're good there. Okay. So, we've only uh we've only got about 25 minutes left. I think it's probably worth spending this time discussing um some potential uh next steps. uh in in terms of addressing this issue. Um and maybe um yeah maybe starting like quotas you were you were going to bring up syncing from unfinalized checkpoints. This feels like

one of the biggest changes we we would need to do. So um yeah you want to give some context on this? Yeah. So, so there there are some changes that are on the specification itself that would help like Jim was mentioning. We need to address this issue that it cannot take so long to leak out validators or perhaps changing the ejection balance and all of these things that have been discussed that are at the spec level. this required to hire for there are some things that are client-based uh feature flags that we can actually implement uh starting to like start from a particular head already load up loaded up in for choice this is something that uh lighthouse

does uh and prison doesn't because of safety issues and we had it and removed and and then we can deal with those things but there are some things that are in the middle of them and one of them is this thing of like checkpoint sync being a socially coordinated checkpoint where we agree that the following we are going to start syncing nodes from this checkpoint that we declare is the good uh the good branch as happens now in Pleski. We can take any of these checkpoints that are not finalized and say this is a good checkpoint we want to start from here but uh engineering this on clients uh varies in difficulty. So I can only uh attest for prism on how

prism works. And the way prism works is that it keeps a for choice being a tree at which root is the fi what we consider the finalized checkpoint. And it would be fine to consider an unfalized one there. But it would just uh we need to agree that these nodes will never sync a block that is contending to that checkpoint. So if we go this route with with prism like taking a checkpoint that is not really finalized uh but we have this finalized state for the corresponding checkpoint in our database and we sort of like magically agree that this checkpoint is a valid uh checkpoint that is a descendant of that finalized

state. Uh we can verify this and we can start our fortress with that checkpoint. But this cl this would make prism never ever sync a block that is contending with that final with that presumed finalized checkpoint. If we agree to this we can specify this out and we can change our code base relatively fast. Is there um this might be a dumb question but like can you just take an arbitrary block and call it like an epoch boundary? So like obviously one risk about this is like uh you know like core devs feeding some malicious uh what random checkpoint

whatnot. Um so like can can we have this feature built in a way that like any arbitrary block and potentially restrict this to like epoch boundary blocks is valid? Um or does this have to be like baked into the client somehow? Um well I would I would take it as an as an actual uh as an actual epoch boundary block. We we can verify that this is the case because we can take a block and we can actually verify this that that this block is a descendant of uh what we know to be actually finalized. Um we can take a state for that block. Uh so we can

restrict to boundary states that's fine. But the problem is that this is an extra trust assumption because now we are never going to sync a blog that is contending to this. So we're going to manually say that this is final. Yeah. Well, when you say we, I guess my question is, is this every single node operator making an independent choice or is it by default in the prism release? I think if it's No, no, no. It's going to be every single node operator that says I want to start from this checkpoint and manually fit a state and a and a block. So just allowing operators to do this.

I think it's something that is reasonable. Yeah. Yeah. Exactly. So that's very different when you say you know we're always going to exclude the block like the sort of uncharitable read there is like you know core devs decide what the valid block is and that's bad. Um but if it's saying there's a mechanism for every user to plug in an arbitrary block and state um like yeah that feels quite valuable. Um and and I guess how I'd be curious if there's someone who maybe understands both side, but does this differ from like set head on the E in any way? um because we effectively have it is it is different because you can

set the head on the EL but then the EL might actually will accept a reorg to a contending head whether here we're going to set uh a block as part as being canonical and we are not going to consider any reorg from that block and this is like yeah you said head like I guess the the E doesn't drive consensus anymore more and I'm not sure how set head would work in like a proof of work chain. Um but it isn't it doesn't matter much I think. Um yeah yeah Enrio so you're going to think about this

implement this in a way that it is effectively finalized but it won't be finalized. So in the in the peer status RPC you still have the actual finalized uh um um state in the end shape. Yeah. So I think I think the only the only change that prison will need is to decouple the condition that the final the actual state finalized checkpoint is the root of our choice. And you what are you about to do in the peer-to-peer? So you actually be you you will be connected to a potential

contending um correct. So to a peer that is potentially give you contending um um blocks. Yeah, correct. We could try to specify something that that would be more useful which is disconnecting the spear faster but I don't know how to do that in any easy way. Yeah because if we could actually be able to detect that and this it is effectively a social finalized state.

It's better to try to not being connected to that and reduce overall noise. So I guess yeah, does anyone think this is a bad idea to spec out and and and yeah to Mario to Mario's point like is this something that should be like a standardized across clients and then we can write tests for it so that um you know we can make sure that clients can take an arbitrary block on a non-finalized fork and and sync to it reliably. I think this is the value we get from having it speced out is that

like we can then have uh standardized tests around it. If it is a per client thing, that's fine too. like we don't need um we we don't necessarily need like a shared spec but there are there are parts of this sync that we do spec and you know that can be helpful and and and I think here it was such spec or not though I mean it just it just needs it needs to be done right like we said we we've got the use case we found it on hleski syncing to an arbit checkpointing from an arbitrary block allows us to get past a lot of the issues that we were facing if we had that available, it would have made things here an awful lot easier. Um, you know, and and the trouble in there's there's a side chat

going on, you know, how how do we allocate resources for this type of thing versus versus the next upgrade. But, you know, given that we have moved this from a theoretical could happen, good to talk about in presentations kind of thing to yet actually happened. I think that it's things like this are very very important. Now I get the security concerns and similar but it's not as as simple as this feeding error right you have to clear your database you have to point to the right checkpoint URL etc etc so you know this is an automated process this is something that enables us to recover a chain in a situation where we have very few block producers going on and very very little network connectivity far faster. So I think you know standardized

or not personally I think we should I think we should be standardizing flags and things because again a good example at the moment you look at the recovery procedures they're different for every single client flags are slightly different you know which is fine but for something like this it's easier if there's a single add this flag to the um you know to to the client kind of thing and it'll it'll recover that type thing makes sense. Um so I mean I think it's it's it's critical that this is an example of the type of thing we can do to say this is what we are doing not in this case to mitigate the problem from happening but to allow us to recover if that happens again or something similar happens again. So I think it's very important that we we absolutely do do this if we think a spec suitable that's

great but getting it done I think is very very important. Yeah, I I'd agree with this. And I think there's there's like two types of issues um that like will come out of this. Like some are like pretty salient short-term like things we need to address. And to me, those are the CL spec uh sorry CL sync uh in in in in like a chaotic state. And then what Felix wanted to bring up like standardizing the EL config files which is kind of the root cause of this. And then you know obviously for things like okay if we want to review how the uh the leaking uh penalties work. I think those are the

types of things that like are probably less urgent and then you you know we can have a conversation around does it make sense to focus on this now or should we focus on shipping pure dust. Um, but it does seem like at the very least like the two engineering root causes that are fairly straightforward to fix should be fixed. Um, yeah, PUS. Yeah. So, there's another engineering uh cause that actually is what what hurt us here. uh when we after the bug happened when we tried to like sync immediately in principle CL clients didn't have any problems uh it was only purely on the EL side but there is a feature which is

optimistic sync that doesn't just brought up in the chat that there are no tests for it which is optional but every client implements it otherwise we can't really sync and but the thing is that uh if we could make optimistic sync actually optional for the node oper operator. Uh that would have been much better because in this situation where you have this bad block that is justified, we we did it. We we restart our nodes. Now that block is no no longer synced because we're starting with a with a new checkpoint and then the nodes try the the CL client would import it from the bad peers and the EL would accept it because it optim optimistically would accept it. If we

had a flag to uh say I only want to sync uh not optimistically then we wouldn't have need to come up with these dirty hacks of declaring the block invalid like the vanilla client would have just spone up with the flag to sync without optim optimistic sync. I think this is an engineering thing that we should implement. Um and yeah Sean has like a PR for quote pessimistic sync. So like what's the what's the overall like how big of a change is this and what's the overall quote like idea or or or Yeah.

Oh the the the idea is very simple. It's just do not sync a block for which the E has sent has sent you back as either syncing or accepted uh response. Just sync the block if the E has told you that the block is valid. How and how big of a change? Like it sounds trivial when you say it like this, but am I missing something? Actually, I do not know. Optimistic sync is incredibly pervasive, at least in our codebase. Uh it's everywhere. I'll need to review. I'm not really sure how how hard of an easy change is this. I suspect it's not so hard, but I'm not sure.

Michael's dip here very small. Um but I don't know if he got this working. Um, so I go ahead. Uh, I can comment on this, but I tried quite uh hard uh around the merge to do this now that because they're going to say have changed. It failed on the merge. Spoiler alert. Um, and I think the challenge is not going to be in the CL per se. At least I don't think. um because it's not that hard to reorganize things to basically reject to treat syncing as as something closer to invalid let's say I mean not permanent but still um the problem I had uh was

that c uh eels's because they uh snapsync they will sit there and say syncing syncing syncing syncing forever it doesn't matter I what what I tried for example was to have this where it would kind of sit in a loop I've tried a lot of variations here's one of it will sit in a loop um and and and keep hoping the the e would take the hint to catch up and then that was before I learned how eels's worked. Um and and they just never would because they won't snap sync at all until you feed them a block till the near the end 100 128 blocks or so.

Uh this is now not all ELS are like this but some are and notably because as POTUS says um optimistic sik is so so ingrained um this is practically speaking it is required like I don't like it but it is required um that's why Nimbus does it. So, uh, that that's I think the ELS are the bottleneck here is the short answer, not CLS. Uh, Lucas.

Okay. So, uh, maybe we need something more sophisticated here because I think we have potentially two scenarios. Scenario number one is when EL needs to do a whole snap sync which is or something similar depending on the EL which is basically syncing the state the whole state right and then optimistic sync makes sense to use uh but then we have a scenario when uh E is uh doing either some small switch from fork to fork right and potentially I don't know needs to reprocess I don't know few few dozen

blocks at most or probably less um which doesn't make sense to do optimistic sync so maybe we just need to be more granular on on this and u that's it and on the second scenario when we are close to head already uh E CL should just wait for E and on the scenario that E is doing the initial sync uh then it doesn't make sense for it does make sense for CL to go to optimistic so we just maybe need to be more granular here and then okay so in the chat I asked like what does quote pessimistic solve that pol's checkpoint idea doesn't and it seems like the biggest difference is

just that um uh the the checkpoint one requires a custom user input so I I would lean towards short-term prioritizing uh having this sort of custom checkpoint sync and and we can discuss uh optimistic sync being more granular in parallel but I think in terms of just um in terms of just like addressing the root cause here it feels like this is a pretty useful feature that CL should have that is probably not extremely complicated to implement. Um, does that generally make sense? And then

we only have like seven minutes left, so I want to make sure we also touch on the CL config issue, but uh and we can obviously discuss CL sync more next week on on the call. But I think as a practical action item for now, like working towards this custom checkpoint sync feels uh quite valuable. on the E side. Uh Felix, are you still around? Okay, I'm not sure if he is, but the idea uh Felix's comment was that uh there's currently two different types of config files that uh EL's use. I think this like parody used to have uh one and then GU had one and then I believe now

everyone except Nethermind uses the same format as GU and then Nethermine is still on the quote like parody version. Um one is this correct? And then two, if it is correct, uh is it possible for Nethermind to switch to the same as GU and then we'd have a standardized uh version. Uh so I think it is correct. Well, I'm not sure if everyone uses uh the same format, but never mind definitely uses different one. Um few things on my side.

we probably would still need to support multiple uh so um allowing to enabling uh granular IPS so we cannot really so we would have to kind of have some kind of extension to the G format or keeping both formats uh which is not that great but uh I think the bigger problem is that in my opinion it doesn't solve the root cause because uh giving let me give you an example Even if we use the same format but for example in our uh classes which the serializes we uh have a typo in some in some uh name we don't the serialize this format correctly and we

might miss it. So what I actually think is should be the uh better solution is to use something similar as we have as fork ID. The problem with the current there are two problems with the current fork ID uh which is used for uh connect connecting between uh between uh def P2P uh def P2P clients but this is less important we are also using this to synchronize cord defafs between forks and we just check out if fork ids are okay but the fork ID has also two problems currently uh problem number one is uh it doesn't differentiate per EIP.

So if we have one transition on fork ID on one EIP, we do not verify that we have it on for example on four uh EIPs. So that's potentially problem number one. Um again this is maybe never mind specific because we allow single EIPs uh uh as to being instructed rather than just using a fork number uh or fork u alias for all of them. uh but actually I would say this is the the argument to going the other way around and to verify that uh for example if we say canon is enabled on on those

clients what actual EIPs do they mean that they enabled right maybe maybe someone I misconfigured their their canon alias and it doesn't enable some EIP when it should so actually going the other way around in my opinion is better and the second thing problem with fork ID it doesn't contain related properties. Uh it doesn't hash related properties that are related to EAPS but are not u transition numbers. So for example, it doesn't hash uh deposit contract. It doesn't hash uh blob uh spec changes, max block changes, etc. So what we should have is some kind

of format. Um we can like I said think of uh extending fork ID or changing fork ID to actually uh kind of create a hash of our current configuration and based on this hash we can compare it with between us uh easily. So that's my counter proposal. It doesn't have to proposal because it can be like complimementaryary we can also switch to the uh train spec for same notice format. So it's like but it's I think this is the actual solution to to some extent rather than uh just uh just something that doesn't have to work.

One question I had there is um I agree conceptually this seems very reasonable. Um in practice what how do we agree to the list of configs? Um, and I think we would have probably caught this one, but there's maybe a way where like we like Yeah, like the deposit I was looking through the EIPs and the deposit contract address I think in 611 is like set for mainet, but it doesn't specify the other ones. So I think whereas like all the the system contracts are uh are actually specified like in the in the EIP and there's like a clear address

there. So I think one challenge um if we do implement this is how do we what's the process to make sure everything is like included and that we don't just quote like assume the deposit contract you know is is is going to be fine. um which obviously after you know this week will no longer happen but I could have imagined a case where we had this prior to pectra and then we add all of the pre-ompiles or sorry all of the system contracts um in but we don't actually include the deposit contract because the deposit contracts already been deployed on chain and it's not actually something that like we've added. So um yeah Lucas so I think we need to identify something

that we can call EIP parameters that can potentially change between EIP uh you know implementation maybe not implementations but instantiations of of different networks and everything there under this should be included in this kind of hashing mechanism. Yeah, I think um so it's it's probably worth mentioning on the consensus layer, we we've tried to do this. Um every node is a standardized API that dumps a configuration and theoretically everything should you know should be possible to to look at and compare, but in reality it's it's nowhere close. Um there's there's multiple different sets of parameters. There are old parameters

from prior forks. So there's a whole bunch of stuff like that. So I think that absolutely we need to do this. Um I you know I have no doubt that it's a it's another test that we can do. Um I mentioned in the chat we can compare either against a single source of truth that you know everyone can look at understand or they can cross compare the clients just to make sure they're the same. Um but it's going to require specification. It's going to be it's just going to take a little bit of time and effort to do so. So um like we said you even on the CL side which theoretically has this already you know it's a long way from from being baked in and it's like we said there's a whole even even stuff as as ludicrously simple as the fact that we have some execution

addresses that are in lowercase hex and some that in IP55 format. So you know would not underestimate the amount of work that it might take to get there. Yeah, we're Okay, so we're at time. Does anyone have the bandwidth over the next two weeks to like investigate what this could look like and what the actual issues would be? If not, uh, then Oh, sorry.

I mean, look, yeah, we we we should we should probably we go for it. Uh my my view is again I think we start with with this consensus layer because it's it's a it's a smaller problem. It's more bounded. There's there's less APIs and all configuration formats, right? Um but I think that that will give us if we do that for the CL side that will give us some good learning and good information to be able to then sort of take to the execution layer side um and then start working on that. I mean the this is not a time critical item. I think we can manually check for Zapoleia. I think we can manually check for mainet even though theoretically things haven't changed but we'll do that. I think this is something that should be ready for you know the Fusaka test nets.

So um you know yeah I mean I'm happy to start kicking that off and talking to people about it and putting up docs and stuff but you know like we said this I don't this doesn't feel like something that needs to be done pretra. Yeah. Yeah. Yes. Uh I agree with that. Uh yes. And then okay so thank you Dano for stepping up. So I think like yes we should target this for Osaka or later and we should probably also use spectra and maybe even like the for before denun as like okay what would the diff have been like and and basically just use them as like old test cases. But I I agree we shouldn't like not ship until we have this but um improving for at least the next work would be good. Um

okay we're already a few minutes over. I know we had more on the agenda, but I think we covered the three most or four most important parts. So, we know what's happening with Hleski in the next 24 hours or so. Um 15 UTC tomorrow. We'll have a call to disable the slashing protections. Hopefully have enough validators online before that. We know that we're keeping Sepia fork uh the Sepia fork happening at the same time. Um there's four clients that need to be updated in the blog post. I will do this uh within the next hour. And then on the CL side, the biggest uh short-term engineering fix is this idea of like the custom checkpoint that PTO has presented. On the E side, figuring out how to harmonize constants uh and

parameters that go into a fork and test for that feels like the highest impact thing. And then uh looking into okay, can we actually standardize the config files, but there might be some trickiness there. So, if there's a way to actually test that every client uses the same uh configs regardless of their um of the format of their their Genesis file, that might be more robust. Um, anything else people feel is urgent to bring up before we wrap up today? Okay. Well, thank you very much everyone. appreciate the work uh over the past week or so trying to get Holleski back up and running. Um let's

see how things go tomorrow and I'll uh post some updates on the Discord uh as we get them. Talk to you all soon.