All Core Devs - Execution #140

Transcript

[Music] [Music] [Music] so [Music] [Music]

[Music] [Music] [Music]

[Applause] [Music] [Music] so

[Music] [Music] [Music]

[Music] [Music] [Music] [Music]

[Music] so [Music] [Music] [Music]

[Applause] [Music] so

Tim Beiko: uh good morning afternoon everyone um yeah this is all cordevs 140. uh we have a bunch of things on the agenda today uh first uh going through obviously the merge that happened on roxton uh talking through kind of any any issues there and and

next steps um there were also a couple different um little merge related items uh so there was one uh one issue we discussed on the discord earlier this week about the responsiveness of el clients when getting uh get paid on requests from from cl teams then there was uh something in the engine api to allow builders to set the gas limit or to allow sorry validators to keep control over the gas limit um and then some talk about uh the next two test nets uh sepolia gordy how do we how do we go through those um continuing the conversation on the

difficulty bomb uh there's been a specific eip that's been proposed for that and then uh there were two more eips on which we had updates uh four four four fours uh and uh five zero twenty seven uh so hopefully we get through all that uh but i guess to start uh does anyone want to walk through what happened on i now believe it's pronounced roopstin and not roxton in proper swedish uh yeah anyone wanted to walk through the fort there i i synced with perry before um he is not i believe he's not in this call so i can share his notes um with roosten uh pre-merge uh we were having some of the consensus layer teams

were having deposit tracking issues um essentially validation become a consensus on the state of uh the execution layer to import new deposits this was due to um some engineering assumptions about block times which were not holding in um in roopston and it has been patched by all teams um it's not something expected to be seen on maintenance then we moved towards ttd uh post dvd we had about a 14 uh participation drop nine percent of this was from the nimbus team nodes which were um configured improperly with the jwt secret um

about 1.8 was from another mine concert concurrency bug that required a reboot um and then about 2.5 to 3 percent was from nimbus besternodes that were attempting to use websockets which serviced a bug in the websocket's implementation they changed their configuration to use http so all that came back online and we see 99.5 participation right now now the mind conservancy bug and they can tell us a bit more still being looked at it occurs the transition restart fixes it uh the zero transaction blocks caused by timeouts have largely been fixed by aragon who they can expand a bit more about two to three percent of blocks are being proposed with zero transactions um

but we're still going perry has not yet isolated the combinations and is going to be looking into it looks like nimbus basu might be one of the affected combos um and mary's transaction bug buzzer has been started on the network to a very healthy transaction load it seems to be reflected well um no more flow blocks a lot sorry more full blocks and fewer zero transaction ones uh let's see zero transaction largely been handled obviously we should dig in a bit more here if we can uh so the shift is now in sync test and dap testing uh each staker's been running some sync tests um and sam from the ef is also going to launch a number of sync tests on robson as well rupertson pardon

me and we'll have data on on these next week so like turning el off turning cl off and more exotic scenarios that is the tlbr from

perry

Tim Beiko: thank you um yeah there's a couple couple things i guess that begin there um uh so the nimbus was just a configure on the jwt another mind you want to walk through kind of the issue sorry the issue that you all had and what the status is there

Marek Moraczyński: so we had one bug that occurred on transition and it affected only a few nodes it um

it is connected with concurrency how we are processing block we found this back in our cut but it hasn't been fixed yet but we should resolve it soon [Music] that is all i think

Łukasz Rozmej: if i can give more details we have a problem when we got uh blocked from the network and they go and from the cl client at the same time during the transition and we wanted to process them both [Music] at the same time when uh we cannot do that uh with our blog processor we

should uh schedule them one by one and that's that's really it

Tim Beiko: and when you say processing it's like you receive a new block from the network and you're already processed and and you get like a block from the cl at the same time so it's like a external and internal pain to the execution layer

basically yes yes yes

Łukasz Rozmej: yes that's the that was the problem and the problem was on the on roop stand because there was a lot of blocks with the same height going on the same time and that's why some percent of the some small percentage of the node got it at the same time and that's why they failed and

restarts fix it we'll just add the at the correct scheduling of this and that's pretty much it it's it's a fix that we will do next week

cool

Tim Beiko: and when you say yeah there were many blocks uh that's i guess that's because on robson when we were roops in when we were mining uh we were creating a bunch of uncles just because it was a bit of a sketchy situation to get all that hash rate at the same time so um is that is that right like just because yeah when you say all these blocks

Łukasz Rozmej: yeah i think i i asked about it and there was mining uh gefnode was mining on like two cars and it didn't uh i think it was overwhelmed and it didn't process the block and it

might the second one with the same uh uh same uh or second third one sometimes with the same block number uh before it managed to process the previous one so that's why that was i think the case that i got from someone as an explanation and yeah that's uh actually it's great that it happened because it showed our

park so

Tim Beiko: yeah it's good yeah no very very cool thanks for for sharing um and then uh the nimbus basie websocket does anyone want to just uh give a quick uh update there i don't know if it's something from the

basu or the nimbus side i can yeah that's on the bassy side okay

yeah you want that go ahead

Fabio Di Fabio: yeah sure okay so uh what we saw is that uh using web socket nimbus what not able to fetch the data that he needs after uh checking with the nimbus team they send us a option to uh make things works but then we had another problem and we had to disable also jvt authentication so now the setup that is working at least for us for us for our two percent of bezu validators that are configured with nimbus is to have web

socket with a special flag and a jvt authentication disabled

on the bezel side

Gary Schulte: specifically um we have um each subscribe and each unsubscribe are not getting um added to the execution engine endpoint it's just i think it's an oversight on our part when we combined uh the uh our json rpc and uh web socket uh onto the same port where we're not adding those two end points to that uh the engine endpoint so i think it's going to be a quick fix but uh like fabio mentioned we have to have a workaround for the moment for that combination

Tim Beiko: got it and is there issue with the jwt kind of independent of that or is it like

Gary Schulte: yeah we only see that problem when we're using the uh uh force pulling features i think i suspect anyway that the force polling uh workaround in nimbus is not sending uh a jot token we haven't we haven't investigated that uh whether it's actually in the header or whether it's um you know just expired or something to that effect but uh for uh for that particular work around to work we had to disable uh child authentication

Tim Beiko: got it um there's a question in the chat about uh

erp37 3675 which is the merging ip and like disabling the block gossip um we discussed this on like one of the testing calls i think last week but it seems that if we did this basically we couldn't do shadow forks and i believe that uh yeah there weren't any clients that there weren't any clients that disabled the gossip after after the merge i don't know if anyone who is on the testing call wants to give more context on this um oh michael i saw you come off

youtube

Mikhail Kalinin: i was just going to say that uh disabling

gossip should not like affect shadow forks from what i understand uh if there is another requirement in this eap to disconnect peers that send you uh block gossip after transition gets finalized and this what can break

um also works because yeah

Tim Beiko: um right does that answer your question proto okay it was about the terminal blocks okay yeah perfect um uh okay so yeah that was uh

basu nevermind um yeah aragon uh do you want to chat about uh the the zero transaction blocks a bit

Andrew Ashikhmin: um yeah sure so our mining has been experimental because we don't uh support gpu mining but after the merge we are going to support uh proof of stake mining still the code is kind of it's not as mature as we would like it to be so we made some quick fixes but we still need uh time to make the uh mining or proof-of-stake block building code more robust

yeah it's um it's a process

Tim Beiko: yeah that makes sense

okay um and then uh yeah one last thing i guess on on uh on robson uh so we now that we have marius's transaction fuzzer running it means basically every block should have transactions in it correct because it was mentioned as like two three percent of blocks without transactions i suspect that's something we would see on roops in like during normal operations uh but yeah so given that the transaction fuzzer is running uh we we probably don't want to see that and the the empty blocks we're seeing are the results of an issue is that is that right i i mean it depends on uh

the how dynamic the the gas is set on that transaction fuzzer because you could imagine it the base fee going up above what it is willing to pay and then having some zero blocks but i would at this point be monitoring if there is there are particular client pairs that consistently are having zero blocks yeah zero transaction blocks and marios could comment on the dynamic nature of base b and his transaction

poster i don't know if marius is

yeah yeah there's the big the big guest room but um okay okay oh okay marius is not not with us um okay yes about just um you know i think we pulled more infrastructure providers through this transition than we had in previous test nets is there any update on how that went

Tim Beiko: so i didn't hear from anyone that things were breaking uh oh actually well i didn't confirm um i didn't get confirmation that anything broke so at first there was some people thought that some smart contracts

had issues but it turned out there was just a user uh spamming weird transactions to it and that like coincided with the robson merge um the other thing is uh uh it seemed for a while like the the rate of failed transactions was higher but i i don't think that's that's quite the case um i'm talking with the etherscan people to get some better data on that uh the challenge with getting data around the the rough roofs in merge is um because there were so many uncles and empty blocks around the merge and the mining was like so weird um even etherscan had like a harder time like getting a nice data dump than they

usually do um but it doesn't seem at least like very high level that like there's an increase in like the error rate of the smart contract transactions on the network which is which is good um but yeah i'll look into that some more um but yeah aside from from like this this stuff no one no one has at least complained loud enough that that uh their product was broken so that's that's really good um anything else anyone wanted to to mention about the rootstone transition okay

so there's two other merge related i guess issues that we wanted to discuss i think it makes sense to go over those before we start talking about uh test nets and like more of like coordination um because it will obviously influence uh when and how we're ready uh but the first was this idea of like uh the el responsiveness to uh cl sending it get payload requests i know uh earlier last this week i believe uh we we talked about that in the discord where um some some cls were sending like too many requests to els and uh els are kind of working around it but there still seems to see or yeah cl sorry are sending requests for blocks uh

too quickly for uh els to properly respond uh and i know there were some fixes on the el side to better accommodate that but there was still sort of an issue on the cl side itself um i don't know if anyone has an update about that

Micah Zoltu: so just to to clarify a little bit i think the el should i think in general this like broad broadly speaking if there's bugs in the consensus layer clients the execution layer client should not be writing code that covers up those bugs we should get them fixed in the compensator client and so in this case the the bug is that when the consensus layer client sends a request for get payload uh i think it's

called the payless one correct me if there's more correct name there um yeah yeah so so if the execution layer receives a get payload request it needs to send a block as soon as it can it should not wait it should not delay if it has no block it should send an empty block it should not stop and go fetch things like it should have a block by now if the cl is sending you that and you don't have enough time to actually prepare a block then that is a seal bug and you should not cover it up by then you know saying oh i'm going to be i'll take my time and actually do the thing that cl wants and you really need to be sending block right away um on the cl side the cl if the cls are sending you a prepare followed by a get

that's like 10 milliseconds apart that is a bug in the cl or it does happen in the real world sometimes but it's very rare and it shouldn't um happen often and and again you should send immediately whatever block you've got if you don't have a box and an empty block like send something as fast as possible there should be no delay in this response right i i generally agree with this the and just for some additional context in 99.99 of scenarios with lead times of seconds you can center prepare and almost certainly when it's actually time to get that prepare will be correct you could potentially have some sort of reorg that might cause you to do a different repair which the

other one should be aborted and maybe there'd be insufficient time at that point to do the get with non-zero transactions but i think we should be generally not masking that bug and providing you know having cl provide uh the adequate time and 99.99 percent of these scenarios

Tim Beiko: um lucas and adrian both have their hand up i don't know who was first

uh the roosters

Łukasz Rozmej: okay let me go first so we implemented this workaround uh this was partly because we wanted to check our block production better and partly because this bug was very long standing i think we were we were trying to to

make people look at it for for a few months now so we wanted to to have uh some some code differently we will uh revert that but it's it's it also gives us some good and from national robston uh we made it i think same more or less same as gaff so we are waiting potentially up to 50 500 milliseconds right now uh which is in the time amount of one second for the for the response but i agree that it's something contrary to the spec and we will drop it

Tim Beiko: uh thanks adrian

Adrian Sutton: so i think there's a couple of things here is it's worth being clear that this shouldn't be all well this definitely isn't all cl clients i yeah i think it's only nimbus now um so we should be pretty clear that most of the time this should be working and if we're seeing it seeing those really short time periods with other cls then we've got more of a design problem we're seeing late blocks coming up um i'm also not as sure it's entirely clear-cut that we shouldn't add a delay because cl i mean maybe we should add it on the cl side that it tries to wait 500 ml if it knows it's got a bit more time and those kind of things but right now if we get a late block we will

you know a block right at the end of one slot will give you no time at all when actually we've got you know a total of four seconds to play with we could actually spare 500 milliseconds to create a block with transactions um there's no way to communicate that kind of trade-off to el currently um we're just trying to get the block right at the start of the slot every time

it definitely feels like

Micah Zoltu: the cl should make that decision and just send

oh so

Tim Beiko: yeah finish up micah and then uh andrew you can send the git later

Andrew Ashikhmin: um so yeah about uh aragon's implementation uh right now it's uh

really simplistic and doesn't have any hacks so when we uh receive a request for to to build a new payload we start building it on the side and we when we receive a get payload if if that uh building process is finished then we return the the build block otherwise we return a pre-populated empty block but that's probably too simplistic and what i'm currently thinking about doing is that when we receive get payload and we we still haven't finished building the block we kind of what i want to do is to add to stop adding more transactions into the block

but then we'll need some time to actually finish sealing the block calculating this to the state route and like finalizing the block so to my mind this get payload it won't there won't be any artificial uh waiting period but it won't be like super instantaneous it won't be a microsecond it will be i don't know maybe that ten million tens of milliseconds in some cases to finish like to finalize the block that's my current thinking

Micah Zoltu: my my gut just personally is that if you've got a block and you're kind of iterating

through the transactions and you get the gut payload um and you just need to you know cut out and you cut off iterating transactions and you just start your sealing process and then you send to me that feels like it fits the bill of as soon as possible i don't know how other people feel but i i don't personally have a problem with the strategy just described

Tim Beiko: yeah i i would agree anyone else anyone have an objection to that to that

i mean

Adrian Sutton: on the cl side we've got to build a block anyway so there is always going to be a bit of time and we we're probably calling

or hopefully calling get payload right at the start of our block build process so it's happening in parallel to a degree um it's not 100 milliseconds to create a block but it's not free either on our side so that kind of balances out pretty well

Micah Zoltu: yeah i think the main thing here is you shouldn't be like going and requesting a block from a remote server or a builder or you shouldn't be like starting to collect your transactions and execute them when you get the get payload if you haven't executed transactions by now um it's too late um whereas if you just need to do like a little bit of cleanup or finalization or whatever it is you know internal stuff i think that that's okay um to adrian's point you do bring up a good point adrian i think that the el knows about how long

it needs to build a block and that information may differ between execution layer clients right so another mind may build a block faster or slower than guest which may build faster slower than aragon and so how much time you need to give them in those situations where you've got a prepared and that's going to be followed very closely by a get um i appreciate the point you're making where it's not obvious like you want to get as soon as possible and you don't know how long the el needs to between a getting a preparer to reasonably prepare a block um does anyone have any ideas on you know how we can resolve that like

Adrian Sutton: so to to my mind i think the current process is fine

um and the vast majority of the time it should work out uh where the el has many seconds to work so i don't think we need to do anything quickly um and can kind of see how it plays out even go through the merge on moment i'd be fine current design but if we were to optimize it my suggestion would be that the cl can simply provide a kind of maximum time frame to the el when it sends a get get payload request so it can say you know i'm running behind i need it just now i'll give you zero or you can have half a second it may be as simple as you know saying you can take a small amount of time or not and maybe don't specify

milliseconds or we do specify milliseconds and cl's can kind of try and be smart but i think it is something like that of the cl can just signal to the el i i'm right at the start of the slot i'm on track i have some milliseconds to spare or i don't i'm already you know on the cusp of my my block being late

Micah Zoltu: yeah i wish we had versioning in the engine api so it didn't hurt me so much to pump this until after the merge

Adrian Sutton: we do

Tim Beiko: yeah and we'll need to make changes to the engine api after the merge uh so this is not the only time i will have to

think to this um just because uh yeah just because we're already like a third of the call through um it seems like we're roughly we're roughly in agreement here like uh nimbus seems to be the the main cl which has an issue that that uh where they need to address about this and uh danny just posted in the chat that uh they are working on it and prioritizing it um but beyond that basically assuming nimbus fixes it we can probably just leave things as is

and three yeah

Micah Zoltu: the reason i want to bring this up on all core devs instead of the seal call is because we do need guests to fix their cover up so guest currently delays 500 milliseconds

so they can build a one block and they need to stop doing that like another mine copied that behavior and that's bad behavior that we need to get fixed

Tim Beiko: right anyone from the get team want to chime in we can see that there's like five of you on camera so there's like ten of them and none of the single one has a like you know what i bet the one person on the mic is the one person that has like a broken mic they might have like a 10 second delay because they're on the moon

one basement yes um oh i see i see someone is

okay we can follow up with mario's after okay okay uh yeah so we'll follow okay

Tim Beiko: we'll follow up uh write this up offline um okay um [Music] okay yeah so yeah moving down from from this uh so yeah just to summarize uh it seems like most cs already have it fixed nimbus needs to fix it uh nethermine and geth have workarounds that they need to look basically revert another mine will do that and uh we need to talk with geth offline um the next thing we had uh was uh this was by alex stokes i don't know

if he's on the call and mikhail you had some input onto this oh yes alex is here i'm here yeah uh honestly you want to give us some some background on uh yeah on the build respect yeah yeah so uh essentially uh so yeah time to drop this issue in the chat and basically it it suggests that we add the gas limit as a parameter to the payload attributes so right now it is not um and the reason we want it to be is because it gives professors more quality over uh setting the gas limit during the build process and i think this pilot's like adding a v2 message i thought we want to change the v1 right now

uh but if we do this and vl clients support it then it means that you can use off-the-shelf software for external builders uh much more easily danny okay i think your last point was the at least the the rebuttal to what i'm about to say which i think micah said in in the chat in that validators presumably control their execution layer and so can set their config there um and well right so they do for the local client but this is if you're using the builder network then you might not be able to know like a builder wouldn't necessarily know for this proposal the builder have to like dynamically adjust some configuration right

sure what

Micah Zoltu: what is the scenario where the i thought even with the builder network you still had an execution client that was under your control and you received a block from the builder network which you then validated and executed with your execution client is that not correct right but so this is just a way to signal so let's say i'm using like stock gas to help build there's no way to tell geth right now like hey for this next slot you should use this limit versus that one this is the idea for the value this is for the builder who's servicing many validators who might have different configuration values

configuration

Micah Zoltu: we want to make it so

users only have to configure one of their two clients like so they don't have to configure death at all they can just double click it so to speak i might have seen something it's not a ux thing this is to help facilitate a builder who's a separate entity in the network who services probably many different validators who have potentially many different uh gas limits and so if they would be able to reuse this engine api to service others more easily if they could dynamically specify that gasoline otherwise they're going to modify the get software again this is for builders

the the idea of a validator is something this in their death note is totally fine for their el node so

Micah Zoltu: perhaps i'm missing something i

apologize um if i own uh i thought the the design was is a given validator would have a execution client and a consensus client that's under their control the consent client would send some stuff to the execution client saying hey prepare a block for me the execution client would then send that details about that off to the builder network to say hey i need to block from somebody and then those people would then send back i didn't think the builders were talking directly to some validators consensus client the am i missing the builders talked to relays who talked to consensus clients consensus clients can either get a block locally or from this network when

getting it from the network there are a couple of parameters that might be specific to a certain validator gas limit that they want being one and so for the job of these external builders to be able to reuse software more uh without to be reused el software in this api to service many validators they want to be able to some of these parameters gas limit being that one parameter right now gotcha okay so i think the piece that i was missing just to make sure that it was is that the builder network talks to the consensus layer does not talk to the execution layer and then when the test layer gets a block from the builder network it then asks its own execution layer client hey can you

verify this is good for me um but this whole process happens between consensus client and builder network that student client's not involved in that communication protocol at all correct that makes sense yes the the validator can get a block locally or maybe externally and then it always asks locally to import and make sure things are good

but when it's asking extra okay you might need to set up certain parameters and maybe so

Micah Zoltu: in that case i'm on board

i

Tim Beiko: well i guess yeah the trade-off is like we are changing the semantics of uh the engine api really late is that's correct i guess well i think i think this is why we make a v2 rather than a v1 okay like we don't

change like we if everyone is on board we could change the v1 but i think it's a bit too late for that yeah but i guess yeah and the question i would have is is there like a security thing here where like builders have control over the gas limit well we want to make it so that they don't right so currently using the payload v1 so say the gas limit is 30 million if if we're using the current api and the builder sends me a block that like raises it as much as possible but the validation no no so yeah i mean builder software could still respect you know the validator's preferences it's just having this makes it easier to use a bunch of software okay so it lowers the barrier to entry

so for now there will have builders will have to modify the software if we add a v2 the builders could go back to using much more less modified

software you got it and

is this really the only thing that's stopping us from having software that works for external builder els or their other rough edges of the engine api for them uh i think you're asking if there's other rough edges to the engine api for this use case i haven't found any one other parameter to that might be worth thinking about is the extra data in the el block it's it would kind of be the same deal with the gas limit but i think that one's less critical

sorry no you're good sorry

these are strictly improvements for the relationship between the el and an external builder this is not needed for the relationship between the local el and local cl right well i mean assuming that assuming that you know you can set your local el to have the gas limit all of your validators want but that's probably fine

okay uh enscar you have your hand up

Ansgar Dietrichs: yeah i just wanted to briefly ask um if we um all agree in the first place that it is um desirable for uh validators using an external network to basically

have the um the gas limit set by the validator because i mean the the gas limit is this somewhat weird parameter where i think we kind of agree that it's theoretically under consensus control we just for now put it under my validator control to to be able to react quicker not just with hard forks so just if really the only reason is to react quick into in case something goes wrong um it might actually be desirable to have fewer parties manually have to adjust the parameter um and in case anyone misbehaves and you know it increases the the gas limit too much uh we that that is already in a sensor network attack that we would have to manually intervene for so i'm just wondering like

probably the answer is we want violators to control this but i'm not i'm not sure this is like an obvious thing i think the danger of having just a few builders have control over it is greater than you know any risk we would incur by needing to suddenly change it and not being able to i agree we also did talk about this a few weeks ago and generally agreed that all the validators aren't necessarily importantly we can punish oh sorry daddy finish it in mika although validators and minors interests aren't always necessarily totally aligned with users interests um the short term the shorter term profit interests of builders are likely less aligned than validators and miners

would be and thus we decided that it makes sense for validators akin to miners to retain control for this um there's probably notes and stuff in the previous call we talked about them yeah it's also more attributable in the sense that we you know can react in all sorts of in social ways if there's an

attack here okay make sense thanks so

Tim Beiko: okay just to summarize so i don't think anyone is advocating

to uh have this part like to have this override the current v1 endpoints um if that's the case this is your chance okay uh if not um does anyone disagree with making that like a v2 or i guess maybe another way to frame this is like alex for this to be useful i suspect this v2 endpoint would need to go live before the main net merge right so here's the thing i mean like this like builders will still exist and they will do the job as the spec dictates it's just this would make it a lot

easier for like other builders to come online so you know we definitely shouldn't block the merge for this it's not like critical or urgent in that sense but you know the sooner the better

okay

Tim Beiko: and is this something all clients need to add support for at the same time i guess obviously like if you added first more builders might use you but like is there a i don't think there's this hard requirement that it gets activated at the same time is that is that correct i mean like theoretically no but i think there's network effects if everyone does it because we can kind of just be like okay this is what we use now

okay

Tim Beiko: uh mikhail you have your hand up

Mikhail Kalinin: yeah i just would like to add that um if we have a v2 epilouds attributes with two with this field i would like i think that we should make it optional for the those cases where uh home seekers or other stickers that choosen don't want to mess with the configuration and or probably just don't understand the gas limit implications like the change in gas limit and the implications of change in gas limit on the network they will just not do this and use like default values that we have currently um in el clients in the binary distributions of our clients that's just like to add on this topic

so if this yeah if this if the value of this field is not provided or zero or whatever the default will be then el

just said its own limit that is it has in the binary distribution okay um i guess

Tim Beiko: it seems like there's like some agreement to do this there's some details to figure out is it um to just continue this conversation async over the next couple weeks like i feel like we're not yeah i can i can start some pr's

and then yeah we can go that way

awesome uh and yeah i linked uh the issue earlier let me just link it again here for folks to look into

Tim Beiko: um but yeah uh not changing the the v1 and um yeah we'll see what comes

for a v2 um okay uh so the next like

Tim Beiko: i think actually yeah there's one more like pretty independent topic and then it all gets pretty intertwined uh i think that the next thing i just wanted to chat about is like the sepolia beacon chain i know we'd mentioned like we wanted to launch it as soon as possible i think there's been uh some progress around like selecting the validator set and fortunately i know

perry is not here but uh does anyone have an update on like the launch of the sapolia beacon chain yeah the current config is slated for the launch to happen on the 20th and for it to go through a couple of the through altair and then bellatrix over the next like 24 hours um i believe that perry is also generally locked down who's going to be participating in this it will be a gated contract so kind of akin to clique and we'll be using uh like kind of an erc20 version of the deposit contract so we can add validators uh but not people can't just get supply and create their own validators um

additionally they're looking into when you create the beacon chain out of thin air for this type of test nut you can actually inflate the supply so there's a bit of talk about inflating the supply on a few validators so that there's a lot of supply just in the background in case there ends up simply a holding hoarding so that's the last kind of thing that they're sorting through over the next few days before the configs are probably finalized around monday please take a look uh there is a link with the configs in terms of the distribution of validator set and the timing if you want to take a look and give you a thumbs up on that and just one thing uh so you mentioned

it's going to run through bellatrix uh basically soon after launch so i assume this means that even though it's like a restricted validator set we're just going to do a ttd override on sepolia when we're ready to actually merge it correct the ttd in the original config is set to a very high number um similar to what happened but we are planning the bell tricks to just happen because we're not inviting lots of the community members to test

on this or anything so okay thanks um okay i guess any question comments on the just the launch of the beacon chain there

Tim Beiko: if not um yeah so it seems like there's a couple interrelated things um so first is like uh now that we've seen robson fork what do we want to see and like what state do we want to be in before we move to other test nets um second is what the ordering of test nets should be so there we mentioned earlier that we wanted to potentially do uh gordy and then sepolia and then this week there's been more chat about maybe flipping them so that uh we actually go through uh we actually go through uh um septolia first um and then the the kind of third related

topic is like the difficulty bomb and and all that and obviously if we want to delay the difficulty bomb uh that might change kind of uh the not only the ordering but obviously the timing uh around which we do all of this um and i think it probably makes sense to just start to like hear from client teams uh especially on the el sides but also cl teams if uh given that some of them are here like what what are you kind of looking for to be ready to move to the next test nets like what would you like to see uh yeah in your software in in testing suites and whatnot

um

so if any the like el client teams have has thoughts about that uh marek i think you're first yeah okay

Marek Moraczyński: so for me a hive test should be passing in all clients maybe kurt does is green uh black proposals of course and we are missing i think terminal block has override we discuss it i think we are implementing it right now but i'm not sure if all eos implemented

it

Tim Beiko: yeah on that last point i think we we decided that not all els would implement

it because like geth already had equivalent functionality that people could use um so i think yeah we we had decided against like making it a must-have for all the clients there's a credible there needs to be a credible path to implement on els but we do not expect to necessarily have to use it so it has been the prioritized if people don't have the time

yeah

Tim Beiko: but then any client who already has the ability to set a specific head in the chain uh it's it's quite similar to the tbs tbh override um yeah um

but yeah okay so but beyond beyond the tbh uh the hive test uh and kurtosis passing this block proposal issue uh and then uh thomas also added in the chat uh uh fixing this uh concurrency bug uh which we talked about it earlier um that'll that'll make sense

uh lucas yes so

Łukasz Rozmej: um optimally i would like to have a code that we considered uh finished all the required things for for the merge which current eta with we have currently very high velocity on that is around four to five weeks but

we can go earlier if the that would be the consensus for the uh for the other devs but we would like to have at least one one test net uh after we finalized code

right yeah that makes that makes sense um andrew um yeah i also like uh for aragon

Andrew Ashikhmin: we still have to fix a lot of hive tests it's like we are failing 88 out of 110 probably there are kind of only limited number of underlying issues but still it's uh it's a lot of work to fix all the hive tests and

also to improve the robustness of our blog building code and yeah just more testing more like more code stabilization and we also like kind of we are still discussing how to tackle this issue and um the the latest block in the rpc request should wait for uh choice update head block confirmation but it means that you kind of you you then when before that you have a new block added to the state but you kind of you are supposed to

to point to the block to the prior block and in arigon we only have a single state how we handle that it's not fully decided yet so quite a lot of things but i'm not sure we'll be able to like resolve all of them 100 percent but on on for on our side it's we are not i i wouldn't say that we are kind of a couple of weeks away from being super ready we need more time

preferably

Tim Beiko: got it thanks um anyone from basu or geth

Gary Schulte: um i could say from the basu perspective uh i want to echo what what others have said about um like across the board execution uh clients passing all of the engine hive tests specifically i think the somewhere testing and expectations around post merge sync probably need to be ironed out i think that there's uh just a little bit of maybe uh scenarios and expectations that we should set uh like specifically for um fast sync or snapsync post merge uh basically requires a consensus layer and i'm not sure that that's an expectation across the board

so just kind of buttoning up what we think post merge sync should look like

Tim Beiko: so when you say basic requires a consensus layer you mean like to basically run on the network is that right

Gary Schulte: well in order to choose a pivot block we we don't if for post merge we won't choose a pivot block unless we get direction from a consensus client and i think that there's some um like on the some of the solo stickers have had uh questions about why why basu is not syncing when they don't have a consensus client on on uh networks that have already merged that's for any any client combination that's entirely why the whole optimistic thing thing exists in the consensus later so

that good heads reasonably good heads can continue to be given to the execution layer so it's requisite

[Music]

Gary Schulte: okay i saw some wargaming that was talked about around sync and post merge sync and uh one of the scenarios was uh execution layers sync without a consensus layer client

so i kind of

Tim Beiko: know the head of the chain

yeah completely agree we can't choose a good pivot

Tim Beiko: yeah i i i don't know that that would be or should be possible basically yeah it's not yeah there might be test cases where the execution layer is synced and the consensus layer is not and then the consensus layer comes online and syncs

um but i don't know exactly what we're referring to

yeah the you know

Gary Schulte: the execution api spec that i've seen leaves sync kind of a as a as a topic to be implemented by whatever method it seems like at least the specs that i've seen it'd be nice to see this like a um a spec for post merge sync just baseline expectations

Tim Beiko: i guess more from the el perspective right

because yes yeah it's more from the cl's perspective exactly yeah and there's there's

Gary Schulte: generally more lead time for an execution sync than there is an optimistic sync on the consensus side

Łukasz Rozmej: okay so nevermind uh also

is able to sync to the terminal block but it doesn't sink any further without the consensus layer and it doesn't sync the state so it only syncs blocks and headers to that point [Music] actually maybe not actually i would like because we have something called um another pivot that we bake in into our configs which is a block we trust a block from like few weeks ago a week ago from the current head that we trust and we can sync to that so um

we back we we up it on on every release uh so that's how nethermy works i'm not entirely sure if gef doesn't have some hacky way of thinking further but i might be wrong totally not saying get does i mean i think they generally retain their sync methods uh because the consensus layer just gives them heads and they do snap sync from there

so if they don't have constraints to say or they what happens they do not they're not the same which i think is fundamental here

so essentially what happens in get is that currently um

we just try to sync to someone so uh once the watch happens everybody will force the same total difficulty so we cannot really pick someone good enough uh currently what happens is that we just synchronize to some random person which is about reporting the most the highest auto difficulty but that obviously isn't really good and we will only do this during the transition and what we will do is we will add one more pack to the config which kind of states that yes this chip has already successfully merged we would i mean we would wait for the merge to happen and then flip it a week afterwards in the next release and after that

get will simply stop doing web assisting at all so if that flag is set then when you start upgrading it will just say that i don't have a weakened client the network the chain config says that this is a merged network so until you attach a beacon client i will adjust it i will not do anything and once the beacon client tells us what the head is then we can just snap having a beacon node here is akin to being able to follow the header chain and get what is the highest difficulty from what you can see from there and being able to sync from there if you don't have the beacon node client then you can't follow any of those like strategies

was

Adrian Sutton: the actual confusion that people expected to not have a consensus-like client or expected that the el would be starting and making progress on sync while the cl was sinking so i've seen the latter in conversations and that's just because we don't have a way to checkpoint sync on roxton because there isn't a place to get the state currently um so that kind of solves itself on my net when it's much more common to checkpoints and your cl is in sync straight away

so one thing that that may be worth noting is that the consensus client the beacon client doesn't need to be fully in sync so if the consensus client tells me that

the latest block is something that may be one month old that's already enough for get to start syncing because we already have one potential target and while the beacon client is just progressing with sync and finding new and new headers we will just keep switching the sync targets but the only thing we need is we need one starting point so that we can actually start sync and i mean thinking maintenance will take six hours so the consensus client has six hours to figure out what the latest

step is in the meantime

Adrian Sutton: yeah i mean sinking from genesis uh you'll take at least six hours on mainnet for the blockchain before you even get to the merge block before you get any heads from us at all

um but that's not normal right the right way to think the consensus layer is with checkpoint sync we're just in this difficult place at the moment with particularly roxanne which is shorter chain but you can't get a state from infuria there's we're missing sources to go and get that checkpoint state so we're always thinking from genesis and it's it's taking the consensus light layer some time to even get to the merge block which i'm not sure how long that is on robson but but then to get fully in sync as well it it's just taking you know it may only be an hour or two and you'll start getting data from us and we'll be tracking the head but i think that's causing confusion for users because they're seeing

the two clients start up and the cl is thinking and the el is just sitting there doing nothing because it doesn't know anything about the post-merge train yet

so one thing that um for example in get we we did see like it's bothering us is that the right even before the merge before the transition happens you have the beacon client connected to the execution client but both sides are kind of just silent so you don't really know what's happening and i think if we could just make it a tiny bit verbose in that the user has a clue that okay now the execution

is waiting maybe from that time is a time of life that it is actually doing something it's actually progressing i think that would be enough it would be nice to have some form of minimal feedback that something is happening

yeah

Gary Schulte: i think that makes sense to add just some um something from the sync process that's going to regularly inform the user that is waiting on a consensus layer direction i think that would that would solve a lot of the confusion

yeah i mean we we certainly pin the blame on the er when we're stuck thinking waiting for optimistic sync like we're tracking head at optimistic

sync we explicitly change things to say we're waiting on the el now um because it was causing confusion on the outside so it might be that i kind of need the opposite on the el side but you need to say waiting for trent head from cl so well the gap at least

has a similar mechanism if we are actually waiting for the beacon client then as far as i know periodically we will print out the message that yes we are still waiting for the beacon client to tell us something so that the user doesn't think that the client just died the question is whether it would make sense to expose just a tiny bit of more information so that we can say that

yes we are waiting for the beacon client and the beacon client is at i don't know this block out of that block or something so that might be worth considering to add but it's not necessary

okay and i guess

Tim Beiko: what's the right place or like we should be discussing that basically like um yeah i just want to make sure i guess we can use the merge channel but you know sync specifically for eos i guess we

have the execution uh dev channel so i think it's probably worth just continuing that conversation there

um i i i mean there's not oh interop yeah is good yeah that's true yeah

so i don't think there's too much to discuss here i guess the question is if somebody we can check out what it looks like what for example the logs look like when you try to synchronize robson currently with the current setup and we can just figure out that okay it would be nice to have this or that extra information and then we can just figure out how to edit like to add it i

i don't know if it makes sense to standardize

too much how to sing yeah i don't base you uh i know yeah you kind of

Tim Beiko: mentioned that it would be nice to have a bit more respect but are you are you happy with with this

Gary Schulte: yeah i think it's reasonable um is the uh setting that baseline expectation i think we're all on the same page about uh post merge sync it sounds like at least with the implementations i just um yeah having that setting that expectation for the community is probably helpful and uh logging bugging weights might be sufficient for that

Tim Beiko: got it um okay and justin i thought you had your hand up but put it back down did you

have a comment

on this thing yeah

Justin Drake: i guess my main comment was that shouldn't we be having different expectations based on which test net we want the we want to target so maybe the the last test net that we do if it's girly for example we're gonna have very high expectations maybe passing all the hive tests but maybe we could have you know lowered expectations for the for the next test match which could be sepolia and i tend to agree with danny uh that just commented polia first you know maybe we can have kind of this keep the ball rolling um and uh and try and do that first with lower expectations

Tim Beiko: right okay yeah so i think yeah that that kind of moves us to the next bit

but i just want to make sure on this thing so i guess yeah one thing i want to make sure the sync issue it seems like there was nothing else there um and then like just taking a bit of a step back you know nevermind eragon basically chimed in about like where they'd like to be at but uh i don't know guess did you have anything else you wanted to add on that front

like uh so obviously you know hive tests passing and we can debate after like how much what test nets bugs fixed and like at least one test net where we consider the code finished anything else on the get side

uh no so

regarding the hive test um i've been working on that progress was a bit slow this week um we [Music] should be be possible with next week to have to have like the missing hive tests fixed or the hive tests that are like where the test is wrong fixed and so that it can be passed by all the clients

got it thanks lucas so

Łukasz Rozmej: i'm not entirely convinced that the tests are broken actually i think that the test might test a decent uh decent scenario it might not be easily

possible on on every client and because it might be um hard to implement but i will get up to you with that in a few days because i need to prove it that we can pass it and i'm not entirely sure maybe your marios is right i need to do

it's possible to pass it uh but it's just it's just uh way different than the spec that we have so we need to implement something completely new completely different than what we have already just to pass this one test that is uh this is not not a real scenario

Łukasz Rozmej: so the test is that network has correct blocks and our cl

send us incorrect block

basically it

um yes but our our cl send us an incorrect block in the past that we that we had to store and so like it introduces a new caching mechanism that we have to cache all the new payloads uh which is no way in perspective for that

Mikhail Kalinin: guys sorry for interrupting you but i would take the discussion to the testing call

um fortunately it's happened pretty soon yeah okay that's okay okay yeah

but okay so

Tim Beiko: anything else on the get side like with regards to just general merge readiness i don't think there was

um so the the only thing uh one thing missing uh in geth right now is the the safe block safe block has like in the json rpc stuff uh that one is uh is missing we have to like the finalized uh in jason if you see it but the other one is not there but other than that we should be pretty good i think there might be one or two hive tests that we are still failing

that i'm not sure

Tim Beiko: got it and for jason rpc it seems like

having finalized is obviously like a must-have uh or at least a very very nice to have um how about uh yeah is are all teams kind of on the same do all teams have

at least like finalized implemented i guess uh we have okay does anyone not have finalized

Andrew Ashikhmin: we haven't implemented it yet but it's we'll implement it soon it's not it's not a big deal

right okay

Gary Schulte: yeah basically it's safe and finalized in pr form right now still it's not yet merged

okay it's merged just some hours before yes and it's also passing the ib tests regarding the things um and okay and

Tim Beiko: there's a couple cl teams here as well so i don't know beyond everything we we discussed um is there anything unlike the cl side that you all are like looking for or want to see um before moving to at least one more test net oh parents yeah i think we are pretty much ready to go

we do have a few um ux related issues that we want to deal with but they're not blocked here on top of that we are spending a lot of time working on med boost so that's not working as well but that's nice to have to have that ready ish but that can always brought in later

so yeah

Tim Beiko: got it

Ben Edgington: um no blockers for tekku i think um we're in testing with our meth boost stuff so uh yeah that's not not a blocker either

Tim Beiko: okay yeah it's the lighthouse is on the same page where it's mostly ux and maybe boost testing got it okay

um so clearly there's like you know oh sorry was there another cl team

oh yeah hey i'm mathews from flashbacks and there are actually some reservations from um from violators [Music] around mv boost so the weight and solid vitators need some guidance on how to run this and they also should be ready um before the marriage

Tim Beiko: sorry i missed i missed the like first part of your sentence can you repeat that actually can you just yeah repeat like a sentence before they need to be ready for the merge

sure so we have we have some reservations from uh validator pools um in particular glado around monitoring uh of of the

extraction of muv and from the cell validators in that they are still not they still don't know how to run mv boost and they also have to be have to be ready

for the match um okay thanks um yeah so we uh oh

Tim Beiko: danny sorry did you have your hand up as well oh i was just going to say um this should continue to be developed in parallel i would even argue that it's potentially a good thing if it's rolled out right after the merge rather than right before

um although just to reduce the potential attack surface and error and issues there so i i wouldn't necessarily put it as a blocker i think that it should come out very soon after if not

if not during them okay um and then there's a comment oh sorry go ahead yeah yeah another on this is that

mev can be really dangerous to what happens on the chain and then we boost actually can can mitigate some of the attacks that can happen um so i would actually consider looking at the research and trying to try to answer this question should we be releasing canvas before the merge or or should we wait um because it's not the

clear cut that it's not so critical that energy boost is an attack vector and i would argue that it actually tries to mitigate um some of the sum of the effects hack vector from um protocol construction i just mean is an additional piece of software and a very complicated upgrade

okay

Tim Beiko: yeah just to not uh dive down the med rabbit hole uh for for the rest of the call um yeah so i guess i'm generally curious to hear from like just eel client teams um given like all of this

uh i feel like there's like a few paths forward one is um we we move to sepolia as the next test net and uh and and use that as like kind of a better run than gordy probably not with like the software that that will end up on main net and and we keep gordy as the final test net for that to happen um or the other way around is like you know we wait that we have something really stable and then ship that on gordy first because there's more users and then cetolia is just kind of a quick sanity check at the end that can happen quickly after um so i guess generally the people prefer to have the next test that happen

sooner to get us another round when we have some confidence that we've addressed these issues but we're maybe not quite at mainnet code or the people prefer to wait longer um and then maybe just have separate happen like really quickly near the merge but gordy kind of being the the test uh like waiting to basically deploy another test until until we're much closer to production

readiness code um i see a hand in like bordeaux and one from danny i don't know if those are like new hands or bordell's been up for a long time uh mine's in new hampshire we did discuss this one week ago on the consensus layer call and um everyone on the consensus layer call wanted

so two things uh answer it i'll so uh after uh two questions um this question was uh my personal preference is that we i mean if we start forking test nets then it would be nice to just if we're on a roll then we should it would be nice to have it finished with main networking because if we fork three test nets and then decided okay oh with us we lost to peter okay let's tweak it a little bit and wait for it so that's uh sorry we missed like part of what you

were saying so when you said like if we start forking the test nets we should and then you start freezing out

okay so when we start forking the test that's it would be nice to finish with actually forking mainnet too otherwise if we just for all the test nets and we never reach mainnet and we just have a two month gap in between then by the time we reach

main okay you're breaking up again but i think i think with if we start working then we should just finish with maina okay

Tim Beiko: so you broke up a bit and i think like beyond the momentum bit there's also the idea that like if we fork test nets and then wait two months to do main nets

there's a bunch of other pr's that get in the clients and and um and that's like would go directly to mainnet um danny i think you're yeah i was just gonna say on the consensus layer call consensual air teams wanted to do so polio first so that we can kind of keep moving in a moderate stakes environment um and then ramp up and make the decision on

gourley from there any el team have looked in here

Gary Schulte: was the

june 20th uh genesis for the sapolia beacon chain was that in line with the somewhat accelerated or or the sapolia first timeline or would that change that works for sopoleo first um assuming that in two weeks we were picking let's see we had kind of circulated ttd and we're picking it

on for devs or something like that well we could do it doesn't

Tim Beiko: it's like the minimum duration right we could have bellatrix and then pick ttd two weeks after that right like and and it would still work right right i just mean it it lines up with like dvd in [Music] in soonish and getting it done i don't think we're going to do sepolia in a

week so it's probably a beacon chain in 10 days and doing sepolia shortly after it works with

my wine

Tim Beiko: okay that makes sense uh and then lucas

Łukasz Rozmej: so i would like to have a bit of time to to progress with all the things we mentioned so uh it all depends on the details of the timeline really so maybe let's discuss propositions on that

Tim Beiko: okay and i guess oh yeah tj rush i was gonna mention the bomb but yeah you wanna you wanna give a quick update there

Thomas Jay Rush: if you don't mind i'll share my screen and go through two minutes it'll kind of set the stage for what i

think we can do are you seeing that yes okay so just to get you where we're at we're at like 14.7 second blocks here and if you look over here you can see it's going to probably be down around 18 to 20 second blocks i think by the middle of july maybe the end of july i'm conservative so i'm going to say the middle of july you're going to be at 17 or 18 second blocks this is another view of the same data just so we get 14.7 as the current time

there's a couple of scripts out there that are actually inaccurate they're underestimating the effect of the bomb so this is accurate data as of this moment so i think there's basically three things we could do we could delay remove or just do nothing each of these has different good and bad points delaying it gives you time obviously it's hard to delay to an accurate place and in either and it looks bad to the community but that's there's nothing you can do about that removing it is also looks bad it's actually good i think because you can make an argument that we no longer need it because this merge is going to happen and it's kind

of a sign of confidence in the fact that it's going to happen but it might also be interpreted really badly by the community the third thing is to do nothing and that obviously has the very bad outcome that you have 20 or 30 second blocks by august but i think it's actually maybe a good thing because the community i think of it as hitting the community on top of the head with a stick and saying hey you know the core devs are a great group of people but the community has to pay attention to the fact that the core devs has the power to make this decision and um we can use that to say you community have to become more

involved in these kinds of decisions it can also be interpreted as a threat so that's a really bad outcome as well so what i would do and nobody asked me but this is what i would do i would simply remove it the next thing i would do after that is do nothing at all and the last thing i would do is delay it now i know that's probably not how it's going to happen it's probably going to be delayed if you delay it what i would do is i would absolutely make a decision now i would delay it for a longer time than you think you need i would make a very loud very public announcement of a date when you're committing to make the merge that gives you like if you do miss the

date that you know if you do this for three months you have some flexibility so the same exact thing happens if you remove it you can make a very loud announcement but i think you need to make an announcement now and i wouldn't try to delay it to a precise place

okay well i guess yeah first

Tim Beiko: thanks for for for sharing um i do have a couple like high level thoughts like first is i don't think we uh should i don't think we should make uh a merge date announcement by far like i think that will just backfire uh there's no way we would get it um i yeah so i don't think announcing the

merch date is like in any way uh uh like possible option um i think yeah it might make sense to just like here i know thomas had put together an eip based on the the discussions on the previous call so like it might make sense to for thomas to take like a couple minutes to walk us through that as well and like the the reasoning there um and that was like for a delay um and then i see there's already a bunch of people with their hands up

so

Thomas Jay Rush: let me make one quick point by making a delay you're definitely creating a date of a merge date because that's what the entire world will think that is

Tim Beiko: sure i i don't i'm not as concerned with like the perception

of it like if we delay the bomb two months four months six months whatever uh and whatever the commitment we make like i don't care if there's some articles that say that the merge is delayed to the end of the year um i do think that like us announcing your merch date is is really bad because it's actually us announcing

the emerge state um yeah okay thank you yeah thomas hi yeah so uh erp 5133

Tomasz Stańczak: was proposed and the calculation is based on the discussion from our last session uh it proposes 500 000 blocks delay for which peaks day

mid august more or less mid-august for the uh for the bomb to activate so as we know the bomb already activated and i think it looks uh worse than some people thought it was that's based on the uh just a what dj mentioned a moment ago um the values calculated based on the scripts that we have show around 0.1 seconds delay by mid august with these numbers around 0.6 seconds so around 40 seconds blocks again in november and then very uh very steep decrease as we see it's now so it actually is all based on we want to have the stable network behaving as

expected that degrades um when we think more or less the merge should happen and mid august was that target for now so please have a look at cip5133 i agree also that's we shouldn't really announce the date of the merge um until we know exactly what the terminal total difficulties because people will make a lot of decisions and preparations based on that date so we have to be absolutely sure of at least that like days or a week when it happens before announcing we are not sure yet um the

removal of the bomb i think that it puts us at some risks in case some particularly uh unexpected event happens where the delay is significant and then we lost stay with the bomb that so far i believe always worked for having everyone prepared for releases and for acting uh so once again i don't think that delaying the bomb delays they merge i think it simply says that the month already started happening we already missed uh the previous estimate over and june uh this one simply moves it to our current plans uh of

midaugust thanks thomas um i don't know who if

andrew or ben was up first um i guess yeah andrew you want to go first and then will you ben

Andrew Ashikhmin: uh yeah sure so i i think that delaying the bomb is the best option i don't think that it sends a a bad signal it actually sends a good signal that we are doing the responsible thing that we want uh to we don't want to rush the merge with the code that uh that is not ready so it we kind of moving the bomb to a realistic date and it does synchronizes our shadow with the bomb and

doing nothing would be actually be irresponsible because then it will hurt the throughput of the network and kind of some kind of i don't see a point some silly political game i don't get it so i would delay the bomb

okay thanks uh then yeah

Ben Edgington: i um disagree with uh your point tim and others are made about not announcing a date um let me uh and i don't understand the logic the let me explain why if what i think we need is a sense of urgency i'm not really perceiving a sense of urgency

about getting this done there are very real costs associated with not doing the merge a hundred and thirty thousand tons of carbon dioxide every day it's nearly a million tons a week every week we twiddle our thumbs that's a megaton of carbon dioxide uh that we're emitting the this is this is a very serious uh issue so where do we get a sense of urgency from well the bomb is a very powerful sense of urgency uh currently um though that the timing may not be ideal uh but i think we owe it to ourselves and we benefit strongly as a distributed community to

um set some other mechanism by which it gives us a target so things happen in this community when we have a target to work too this is my observation uh this inculcates a sense of urgency that make get gets things done and i've seen it so many times uh in in ethereum world so i would propose if we do delay the bomb we also commit at that point to a time frame for delivering

this thing yeah i think i think that

Tim Beiko: that makes sense like i think the challenge is if we commit to like a time frame we can't like set a specific date because there are unknown unknowns like if we have an issue show up two weeks before we want to fix the issue rather than have a theorem go down even

though there's a cost to uh obviously keep staying on proof of work and and not delaying and delaying the bomb so i think it's like sure my opinion is like if we do delay the bomb we should implicitly target some realistic uh you know delay which i think is is what thomas's eft was was like attempting to do when we can discuss you know whether 500 000 blocks is the right amount this should be four should be three should be six um but like for example if we delayed the bomb by like 12 months i agree that would be really bad um i think on the sense of urgency like it's worth noting like on the last call and privately to me like client teams have mentioned that like they do

feel pretty stressed and urgent and that like i think there is a fine balance between having a sense of urgency but just having people be so under pressure that like the quality starts to drop so i think yeah i i guess it summarizes yeah if we do delay this i think it should be a realistic delay like to still maintain this this sense of urgency um but i also want to point out like i there is a point where like too much pressure just pushes teams to like burn out or make worse decisions and that's also not the situation we want to be in um uh yeah thomas and and and tj and then i guess if anyone has like final

comments uh please raise your hand now and we'll do like a final round and then wrap up but then because we're already at time

uh we are going to go over a bit hey oh sorry oh yeah yeah please yeah you haven't done yet yeah we we cannot we cannot fight the race but um uh so

like we kind of discussed a bit just now and we kind of all agree that delaying the bomb is the way to go uh and the but the calculation that uh too much did is not the way to go and we should do some calculation that pushes the bomb by uh roughly two months

okay two or three months thanks um and we can and also we can we can push off the bomb delay today if we want to or like schedule the fog as soon as possible

Tim Beiko: so on on that note there's like some comments in the chat about that and i think it's how you get enzgar you have to comment about we should make the decision today but i think it was a year ago or so edgar you made the comment that we should probably not like include vips and hard forks on like the first call that they're presented to give people time to like think through them and think through like the values so like i i would be pretty against like including

like selling a hard fork uh and like making this this delay like official right now i think for sure on the next call we can and like we we can like discuss exactly what the number is what the right approach if that's what client teams want to do

um um yeah so that would be my my opinion but uh yeah enzgard thomas uh and thomas as well uh y'all have your hands up but uh i think i think thomas rush was first then thomas and then hensgaard

Thomas Jay Rush: um i'll just go real quick i totally agree and understand everything everyone said in response to what i said and i'm

obviously whatever happens happens i think what i'll do is just go back to reporting on what the timing looks like as far as how long the blocks are taking um i do think it's getting a little urgent to make a decision on when to do this fork no matter what we do so thanks for listening

uh thomas yeah hi yes sir

Tomasz Stańczak: i would like to suggest this urgency or even like vote now to announce the fork date on the client when this would be implemented um i think that we already see like the 17 18 seconds blocks at the time when the

fork would be activated if we announce it for like three weeks from now let's say um i don't think we should be waiting for the next local devs and this 500 000 blocks calculation is based on uh on the suggestions from the last goal so it actually is to two and a half months uh just to address what marius was saying uh if there is some miscalculation uh please let me know but uh i'm just posting here like we have around i believe six point five thousand blocks per day which gives around 190 000 blocks per per month uh and 500 000 boxes around to two months 18 days or 2 months 20 days

Thomas Jay Rush: yeah i always say a hundred thousand blocks is two weeks so 500 would be about two and a half months

yeah sorry uh too much i wasn't like we weren't trying to to pick on you but uh it was we were just going off by the comments that uh ansca made that it would put us at like 81 days longer than [Music] longer longer than what uh so that the block time that you see right now would only happen in in [Music] november and i think that's a bit too much

Tomasz Stańczak: i do believe that uh just just blocks happening that's slow blocks as we see now it's already a problem i mean like in in one two three weeks we'll have really degradation of the network so it's already affecting the the mining pools and the users of the network they'll be basically going up so

Tim Beiko: yeah i do think we should just review the numbers offline um to make sure we're all on the same page uh i'm curious to hear so like from el client teams do you yeah do people think like this is a decision we should take now like obviously nethermine does uh

get seems to be on board um but yeah i just want to make sure and then if if the decision is like we do delay the bomb uh by some amount that's roughly two months uh obviously we need to figure out what uh what those things are like what the exact block number uh the delay is async um when we i don't know that we would want to choose the block height today like i think we can probably get that done async as well just to not like rush it um i don't know yeah i i i think we can make a decision about like the general like we do want to delay the bomb by some small amount of months but i i just

want to make sure we don't like rush figuring out the exact details on this call and actually take time to like think through how much and when we want it to go off

um i don't know yeah i guess yeah how do clients yes we would exactly this we we

we decide today they want to delay the bomb and we we do async [Music] do some more calculations and we decide on what to delay the bomb now as soon as possible and we should maybe already um do the timing for the next fork for the for the

uh for the for the delay fork um oh wait we also need the block numbers for that

so yeah so yeah

Tim Beiko: i just don't want us to like come up with like a block number in 30 seconds and it turns out to be wrong

but we don't

Micah Zoltu: we don't we don't need a block number we need a date like i think we pick a date like say you know in three weeks then we can figure out what the block number is for that later but we can just choose a date

today yeah okay i think that's that's helpful um i guess i just want i okay does basu the basu and aragon generally agree with that i just want to make sure that uh yeah i think we

Andrew Ashikhmin: we should delay the bomb and i think uh something in a hot fork something in three three

weeks would be good and that we can discuss the exact block number offline but i do think that we should we should decide to delay it now

okay and base you

Gary Schulte: yeah that sounds reasonable to commit to a time and then back into the block height and the delay sounds quite reasonable uh especially if we do the off uh off cycle acd call time for that

uh so

Tim Beiko: i wouldn't do an off cycle error icd call just to be careful we have the cl call next week and and like uh client teams can show el client teams can show up to that as well if we really need to uh but i think if we have the decision now and we agree to the block number

and uh the block number both for the fork and for the delay async we can just communicate that async we can just communicate that both async and on the and on the cl call but i wouldn't have another awkward apps a week from now just for this um and then yeah the other part is like how when does the delay happen i heard like three weeks uh so that means that like so if we do three weeks just uh the thing i would want is at least two weeks from releases releases being out to the fork happenings and does that mean that like client teams assuming that like today or monday we get like a block number and

and block height um next week you can have a release that's ready for that basically is that is that reasonable

no problem for another mind yeah because i think yeah if we yeah if

Tim Beiko: if we have releases out next week it means we can announce them early the week after

and and ideally yeah so just

Tomasz Stańczak: i suggest 29th of june which is wednesday usually the wednesday releases for the safest um so we can yeah for the fork which means that uh

now off offline we can confirm the numbers but everyone can start implementing the ap like configuration to switch it on uh then we can confirm the number of line and in two weeks on the all core devs we can just confirm that everyone is ready uh accept and then the wednesday after the awkward deaths on 29th and this was 18.

Tim Beiko: yeah we're gonna need confirmation that teams already like before that so my hope would be like we would have a blog post for this by like thursday or friday next week with all the client releases um sorry the one other thing that we we need is like some amount of testing for this

which is not huge but we also need this to happen obviously

bordeaux

so i think thomas said we need to have a confirmation in two weeks that everybody's ready i mean that would be three days before the fork that's definitely way too late so if you want to work in three weeks my expectation is that wednesday this wednesday we have a confirmation from everybody that releases are out

otherwise it's just right

Tim Beiko: i agree we need to have the releases out by like this wednesday at actually yeah the releases should be out and we should announce them by like the consensus layer call on thursday which is

14 utc on thursday

yeah but we need a name yeah we'll do we'll do uh yeah we'll do the names uh async is all yeah

Łukasz Rozmej: merge is close as the name

Tim Beiko: um thomas you still have your hand up did you have a final comment or is it just up from last time

so one thing to reiterate for everyone listening in this does not mean that we will delay the merge the merge will not be delayed we are only uh delaying the the the difficulty bomb and it will also take no uh no like capacity away from working on

the merge this is like a five line change and we can put it out in ten minutes and uh so the like this does not impact the merge at all

i guess contest will not listen until this point so i'll let people know marius that you said that

Tim Beiko: um okay uh we're already about 15 minutes over um there were like some eips that wanted to give updates i i think it makes more sense to move them to the next call so they have proper time i would encourage people so there's links in the agenda

for eip 444 and eip5027 if people want to look at them before the next call because i know yeah they've basically waited through this and they get the chance um there's also two like quick announcements so uh next friday basically a week from now at like the alcor devs time there's a eip 4844 uh call uh because there's already a couple teams prototyping it and uh so if you're interested in that uh the links to the agenda and then similarly uh for uh eip4844 there needs to be a kzg ceremony um and there's a bunch of people working on that and they're having public calls i believe every two weeks now so if people just want to

join or watch the recordings um those are linked in the agenda as well um and yeah just to reiterate the decision about the bomb is uh you know we're roughly in agreement uh we're in agreement to basically include uh thomas's eip still some bike shedding to do about the exact number to make sure that it's it's correct um and um and uh the activation number which should probably be about three weeks from now roughly around like june 29th um and then there's some comments on the chat about we need to keep the test nets uh merging as well like i think that's that's that's correct uh but obviously

like a bunch of client teams have mentioned they do need probably more than two weeks before they're ready to pull out a release for these um so i suspect it's like in the next two weeks we get we get obviously this pretty sort of bomb that happens in parallel um in in parallel uh we keep working on sepolia and um that'll merge uh sometime sometime probably right after or close to the difficulty bomb being pushed back on midnight any closing uh oh yeah there's another comment about like

uh continuing shadow forks uh we can discuss that on the testing call uh next week

uh so one quick thing we still have shadow fork minute shadow fork 2 running laying around and we would like to deprecate it but before that uh we would like to run sync tests or i would like to run thing tests i'm i'm doing the sync tests for guest and i think uh it would be really nice if all the other clients could also run their sync tests on manual share of work too that would probably be interesting

Tim Beiko: okay um cool let's wrap up here we're

already 15 minutes over appreciate everyone staying on um and uh again apologies to the folks who had eips to present uh i'll i'll make sure to to get to those on the next call

cool thanks everyone bye thank you bye thanks bye [Music] so

[Music] [Music] [Music] so

[Music] foreign

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.