All Core Devs - Execution #142

Transcript

[Music] so [Music] so [Music] [Music]

[Music] [Music] [Music]

[Music] [Music]

[Music] [Music] [Music] so

[Music] [Music] so

[Music] [Music] [Music]

[Applause] [Music] [Music] so

[Music] [Music] [Music]

[Music] [Music] [Music] [Music]

[Music] [Music] [Music] so

hello everyone uh welcome to awkwardev's number

Tim Beiko: 142. um a couple things on the agenda today uh first just going over greg glacier which happened a few weeks ago already um then a ton of merged stuff around sepolia robsten latest vale hash gordy and mbv boost and um finally i i don't know if henry is here but we might have a a quick update on eip4444 um and yeah i think that'll get us through the hour

and a half um i guess just to start anyone want to give a quick recap on grey glacier and what we saw otherwise i could just go for it quickly did we see nothing we saw the block times go back to 13 seconds um so if you look at the block time chart it's just pretty clear it worked uh because this is uh like an update that changes how the difficulty is calculated you can see on the first block that all the clients are in consensus so like uh there's no like weird potentially creeping issue uh that that can hit us um so it seems

like everything went went well i don't know if anyone noticed any any issues

oh

Martin Holst Swende: one thing yeah i'm really paying very close attention but i'm looking at the pokemon now and you can see that there were 10 blocks mined on the old chain the one that open ethereum is on and the land so the timestamp on the last of those is four days

ago it was mine by exa pool i

Tim Beiko: i tried to reach out to exopole uh so there were two there were two entities that seemed to be mining blocks on the old chain one was like in the non

like a single address so like not like the the extra data was like 0x um the other one was exa-pool i did some searches online for example it seems like a project that has like shut down or something so i it's not clear my impression after like spending 30 40 minutes of searching on google for it is it's like it seems like a project that was doing a bunch of blockchain things that shut down and maybe someone somewhere forgot to turn off the mining rigs um so if anyone is listening and is in touch with exopol you still have some miners running on the uh

yeah the pre-greg gray glacier fork um do you know martin if uh are all the new blocks on the old chain still being mined by exit pool or is it still a mix of the two

coinbase addresses

Martin Holst Swende: i do not know i i mean i could just shake it up but i don't have it at hand

Tim Beiko: and i i also i sent them both on chain message telling them i'm not sure how much attention they're paying to their addresses anything else

oh i think okay so yeah one one of the of the miners the one who is not exopol um since i've sent them an unchained message they've mined a block on the uh updated chain which is which is good and then the second one which i believe is exopole it seems like they've also mined some blocks on the on the main chain uh just just today oh i'm not sure that one is except polish anyways two two of the miners uh that were mining on the old chain

have have migrated but yeah it might be worth uh looking into exit wall some more anything else okay um cool so i guess um next up on the merge we also had a fork uh earlier this week for sepolia um i know perry danny and a few people have been looking into it uh does anyone want to give like a a quick update about this polio merge

Parithosh Jayanthi: sure um i can go for that one

so it's a boy of um much yesterday if i'm not no day before yesterday um around yeah doesn't matter uh almost immediately after ttd was hit rather about two epochs after we noticed that a ton of participation was dropped and we were seeing a significant number of missed proposals we traced that back down to config issues from a number of different validators on chain and most of the contract issues were patched relatively quickly once the config issues were patched we did see proposals go back after like and at the station go back up to about 94.95 um the remaining five percent are

missing keys so we would never hit a hundred percent on this network until they leaked out um we did notice that there was an invalid um signature that was propagated through the network we're still figuring out why that happened but the network self-healed as expected and besides that uh i think since then we did notice that there was a bezu i guess the world state issue that was triggered on a couple of notes but they already have a fix for that and i couldn't link the pr and maybe someone from the basic team could talk about that i think that's about all the issues that we saw on the network um in general almost all of the issues were

config related which is great news um but that also means that for future for future um fox we're gonna have to work a lot more on communication and we probably create like a guide of common pitfalls as well as maybe working on a basic script that just validates if uh if notes are properly set up if the right ports are used if the right ptd is used

etc

Danny Ryan: i think that's it i also worth just stating that the the con fig issues resulted from us essentially doing a strange ux something that we've done on multiple proof of work test sets where we essentially defensively set a

very large ttd on the command line and then are supposed to remove that override or set that override to something lower um that is you know from my understanding the the cause of a lot of the issues here that is a strange ux um to set and then unset whereas on mainnet it would likely be only reaction against some issue to set and not necessarily and then not unset um so i think it's definitely a concern and there's like things to do better here and to communicate better about but i wouldn't suspect we see this issue on maintenance there's all sorts of things you can get

wrong and how you configure the setup don't get me wrong but i don't think you'd see this one

Parithosh Jayanthi: definitely i also don't think we can see this issue on gaudi because scully also has a relatively predictable difficulty so we'd likely not have to do an override on gurley and possibly not only that as well right yeah

well right

Danny Ryan: if the but if the main if the override happened on mainnet it would be not defensively and then needing to unset it it'd be to accelerate because hash rate was dropping so the ux and like the experience of doing that would also not have the same type of error that could happen here

Tim Beiko: got it

anything else from sepolio so aside from this config issue basically uh which uh which was then which was then fixed a couple last keys um and and the basic issue that we have a fix for there were no other client level issues that we saw is that is that right yeah i think those were all the issues that we saw yeah and we still need to dig into this invalid signature but um yeah that's not something that affects like the overall health of the chain um

oh okay and uh yeah uh gary from basically has some mic issues but he's saying that regarding this this world state issue uh the fix uh there will be a fix for it part of a basic release that's out in a few hours and the basu version number is 22.4.4 to address that and also just to follow up on that i don't think that's a much related issue and also didn't show up around hpd but rather something to do with that database and as far as i know it was unrelated

got it

Justin Florentine: that's correct all right this is a this is a rehash of the issue i spoke about last

all cordobs cool

so

Tim Beiko: yeah that's that's really good um obviously like the configs we we need to do a better job of explaining but yeah the fact that everything kind of went uh smoothly from the client side is is really really good news um anyone else have comments just on sepolio okay um i guess marius had something and he he couldn't make the call but um he wanted to talk about choosing uh basically merge blocks for both robston and sepolia now that the merges happen i don't know if anyone else had the

context on this i know it was discussed a bit earlier this week as part of a testing meeting

Łukasz Rozmej: um so the four blocks would be the first block that is finalized

after their terminal book

Mikhail Kalinin: no i i guess not because we want the fork id to be correct and we want it in the future i mean this block and also we would like to coordinate and upgrade

this is what i understand

ah okay it

Łukasz Rozmej: needs a hard fork in order to have a fork id in the future to disconnect potential clients that do not okay

Mikhail Kalinin: exactly i think it's just unfair with the people who have been much

Tim Beiko: and are we i guess is there enough value to do that before say shanghai or can we just wait to shanghai and retroactively add like a merged fork block there

so i think it's uh it would simplify

peering process with 16 notes that's the biggest group right

Tim Beiko: at the cost of like having another hard fork basically

Łukasz Rozmej: yes yes i think we can potentially do it after girly or after or after maynet i don't know but uh we shouldn't like uh take too much time with it or before i need to just test it okay yeah before minutes so it's it's okay it can be considered as a part of the testing process too as a [Music] right it's a small thing but it can be considered as part of this

Tim Beiko: and

i guess the good thing is now that these um these networks run on on proof of stake uh when we plan a fork we can have like high certainty of when it happens so what we could do is you could imagine having the the release for the gourley merge containing also the robston and sepolia merged fork blocks and we can just like scatter them so they they happen each like a day apart or something like that um and because it's proof of stake we we have really high assurance that this happens um on that day um

one thing i assume this this would be an el only fork right so it's like you still trigger it on a block number

Danny Ryan: um so you could trigger it on a block number and have pretty high assurance of when it's going to happen block numbers don't always come at every 12 seconds in the el right but you could also trigger it on time stamp which is there's a debate there on future hard forks as to like how you do the coordination i don't i think block number is probably fine here we don't get into the debate i would i would do sepolia first well i don't know

i would potentially do separate first because it's like a very limited set of people running nodes and so i think it'd be easy to see if it's working correctly uh where's robson's i think at least a bit more open

Parithosh Jayanthi: okay we can also make the argument that technically robson's deprecated now we only have to support sephoria as well as um girly so we don't have to go through the extra coordination effort for austin that's true actually yeah we don't need to

Tim Beiko: yeah that's a good point if the aim is yeah i think in that case my suggestion

would be we um we only do it on sepolia as a nudge for people to move away from robson and it's a very soft nudge because basically um basically they uh you know they can still use it they're just maybe get worse peering um and then yeah we include that as part of the gordy release um so just like for people it's just like one release that they need to download for both the sepolia merge block and the gordy merge does that make sense to people okay

no objections a couple um i would personally prefer to do it on a block just because yeah it seems like it requires no work from the cl teams to do that and the the um yeah the uncertainty we get by like missed slots seems quite low um so anyways we can discuss this offline but yeah they got a high level we we just do it for sepolia uh combine it with the gordy release and see how that goes anything else on that okay

um [Music] latest valid hash uh mikhail or potus do either of you want to walk us through the latest stuff in latest hash i'll share the uh the pr here um let me defer it supposes as is the proposing this change

Potuz: i was of writing in the chat please make how you do it all right so uh i just put out a clarification uh about latest valid hash uh in spec

and it turned down into a discussion that it seemed that it wasn't really clear a clarification so i just want to state something and i'm not sure if people would agree or not but the way we're using the latest valid patch when we receive an invalid block is to prune our fortress uh in the consensus layer side and to prune report the purchase branch that is invalid we need to know uh which were all the invalid blocks so currently uh what we're requesting is that the latest valid hash mean two things it's actually a valid block it's an ancestor of the invalid block and is the last and every descendant from that valley block leading to the invalid

payload is invalid so that means that we can do two things we can remove the whole embodied branch and declare valid everything that is left out now we do understand that there are some clients that cannot comply with this and so there's a there's there's a cabinet that we can we can allow for a client to say well i just cannot give you this but i can tell you that the last block is invalid in some scenarios yeah so that is correcting this correct so there are some clients that can tell us quickly look this block is invalid i have no clue what the lab the latest valid hash is and we're just asking okay so give us null but don't don't give us the parents because if you give us the parent then we're going to make a wrong

assumption about it

Danny Ryan: right the worry is if you if you give latest valid hashes just the parent you the consensus layer now thinks the whole branch is valid which is not not the case so it's best probably to make it clear that you don't know

um yeah

Martin Holst Swende: can i just get it out yeah yeah i'm just i'm just i just want to get that clarified one more time so the the what what you want to avoid is that an el for some reasons say the latest valid hash is this invalid block

Potuz: that's what you want to work with the problem is that the ea might say this latest valid hash is this thing that uh is this thing that is the parents of the one that you gave me and i just don't know the validity status of this one i haven't executed it yeah and this is

i think that makes sense what you're saying so yeah that may happen yeah sorry

Mikhail Kalinin: yeah so i wanted to discuss the scenario so for example we are syncing right and we got payload and we know that it's invalid because for example uh i don't know it's hash that doesn't work out or it's uh yeah the header values

are malformed

Danny Ryan: those are easy to assess

Mikhail Kalinin: yeah or the base fee change in the way that it's through them to compare to apparent which we also for example got a new payload and we have uh and we know that this is wrong but we cannot really say which one is not valid because we are still syncing so there's no way of saying which one was this file so this is like a problem for consensus uh execution layer yeah so in this case you should return no uh a spare pr

yeah i think that that's the reasonable scenario a reasonable fix

yeah and lucas the that

Potuz: if you allow clients to do this then there's the risk that uh the el might be lazy and always return and it's going to be spec compliant so we can just add test for checking that right

Łukasz Rozmej: so the other risk the other risk is that el is forced to sync an invalid branch that's known as valid which it might not be able to do because the network won't serve the data for it and you have a deadlock why let's say there are more invalid

ancestors in this chain so all other clients already like discarded those blocks and you won't be able to sync

Mikhail Kalinin: yes but i don't think it's like related to this exact

vr

Łukasz Rozmej: uh yeah it's kind of like uh so the the workaround would be if you required us to give you this valid cash uh then we the only thing here we can because we cannot give it to can only say syncing right and we need to try to sync to get the latest what it has right that's the only way around it uh if we cannot not give you the latest solid hash and like i said this might fail uh because of

the network

right you mean that without the ancestors too old so there's no state is that what i mean

Łukasz Rozmej: yes for example yeah

right yeah but if

Danny Ryan: this branch is not available cl should normally rework to the one that is available there's nobody will vote for valid branch or for not available branch is your point you need to do the null thing because otherwise there's a date availability problem on even attempting to continue to sync the thing that you saw this was invalid are you saying the data availability

problem results in this change from this change

Łukasz Rozmej: um i'm not sure if that was the question to me so the problem is that we might get an invalid payload that cannot be saved so we cannot get that solid hashtags yeah so we need to be able to answer involved yeah

Danny Ryan: yeah okay i'm just that was thank you for clarifying yeah

Justin Florentine: so what do you mean by exquisite

flag value well i'm just worried about null being a little overloaded is all it sounds like we're picking out a pretty specific use case where all the other fields that are in there we know that they're invalid based on the state of the block and it's really the relationship to the parent that's the problem because we're mid sync right um so you know why not flag that as a specific scenario instead of just saying okay

Mikhail Kalinin: no i think no makes sense here because it means literally nothing what's the latest about no not that the whole response is null yeah we may argue about encoding it in a different way but i don't think it's like worth to spend time here

where it's better to spend time in the pr comments

Tim Beiko: okay andrew uh yeah

Andrew Ashikhmin: i think uh this change makes sense but i would like to argue that let's can we use jason now or neil not this special like 0x value that's my only only comment

Danny Ryan: yeah let's i'm happy to take it to the to the repo i don't have a strong preference on coding

Tim Beiko: okay and so it seems like everyone

agrees that allowing the option to return null for the latest valid hash um is good yeah in pretty pretty certain scenarios i think we need to be careful how it's specified and uh make sure we have a few hive tests to kind of show the difference between lazy behavior and correct behavior here

yeah

Andrew Ashikhmin: oh yeah sorry and why like but jason has new right so like what what's i mean can we use just jason neo type it's a minor thing but why like i'm confused about this null and quotes because

it can be without quotes

jason

Danny Ryan: knew i think the the one argument against that would be in the event that there were a few special scenarios and you wanted to say that in in named flags um otherwise nil i think is totally fine

Tim Beiko: yeah so uh regarding this this vr can we then uh assume that the latest valid hash is valid

Danny Ryan: if we get one i mean that's that's the intention of the specification and i think as it should be and we should make sure that

the test scenarios cover that i think if we clarify that and allow nil in certain cases that i believe everyone can now

conform

Tim Beiko: okay um [Music] so aside yeah aside from like neil versus null um does anyone else have concerns or thoughts about that

Micah Zoltu: i do but i posted them in the pr

Tim Beiko: thank you

and what is i guess is this something we can like get merged in the spec early next week um so that clients can like start working on on the final like implementations for it some sometime next week is that realistic

Danny Ryan: yeah i think so my unknown is how long to get a test and hive but we can circle back on that on monday with mario and others right

Tim Beiko: yeah yeah but yeah it feels like at least we have this pr set and clients the clients can start implementing uh the actual behavior in parallel to uh

testing teams writing tests in hive and you know optimistically by end of next week early the week after we have the behavior correct and all the clients

i think behavior-wise uh read the code correctly guess where it has this behavior the difference is that we return the column well the zero hash if we instead of no okay so that's it we just changed

Tim Beiko: no unfortunately not zero hashes for enough purpose is when you have to uh let's one cl have to invalidate starting from a transition block

when there's something wrong with the terminal block or with the transition block itself so we use zero hash in this case okay okay um anything else on this okay uh i guess next up i wanted to talk about gourley and first like just a quick there was a quick question around gordy about

uh from infera about um do active proof of authority signers need to do anything after the gordy merge uh the assumption was like as soon as gordy has finalized um they as soon as gordy is finalized they can basically shut down their poa signers but is there a reason why that's not the case

Danny Ryan: um so if they're if they're running the actual merge software so if they're running gather aragon that's merge enabled and knows about the terminal block hash they'll just stop producing blocks um if

i understand correctly or their blocks at least will stop being gossiped i think they'll stop producing though um and so they would just stop producing and they can turn them off can someone validate that if i were running a minor or a clique designer against that it

was done yeah yeah yeah okay

Tim Beiko: and then i guess if for whatever reason it did not stop automatically once they have confirmation that like we finalized on the proof-of-stake side they can shut it down manually but they shouldn't actually have to do that

Danny Ryan: right and everyone else in the network's going to be rejecting those blocks

anyway

Tim Beiko: but you can imagine there's a world where like their signer instances run

separately from like other nodes that they may want to run so like it requires like they might they basically don't need to upgrade those nodes and they could leave them on the old version up to the merge and um yeah right

Danny Ryan: the network will reject them and yeah they'll be sequestered in their own little zone and they can turn them off

okay

Tim Beiko: okay that was my my rough assumption but i wanted to make sure that there wasn't a weird edge case we weren't thinking about cool um i guess so the second note around gordy is um

on the last couple calls we've talked about wanting to transition gordy when we have code that's pretty much feature complete for mainnet so that it can be like a good basically dress rehearsal for all the validators on mainnet and everybody else running a node um the people like obviously there's like this change to the engine api we need to get in um aside from that i'm curious from client teams like is that like the last thing people feel that uh they want to have in before moving to gordy or are there other things that um are still missing that we'd like to to finish before we we start looking at

merging gordy um and i'll just pick on random kind teams i'll start with aragon because i saw you all past the hive tests earlier this week

Andrew Ashikhmin: yeah thanks to julia we managed to fix them

Tim Beiko: so is there anything on the aragon side that you think still needs to be done again except this like engine api change before we we moved to gordy

Andrew Ashikhmin: uh

well um i would like to test our sync performance um with and especially i'd like to reiterate i think it's very important to enable checkpoint sync on the cl site and on the main net it should be the default otherwise if you join as a new node and without checkpoint sync then it will be prohibitively slow so yeah more testing of sync performance

okay

well um yeah there isn't really much else we didn't get out of bug reports lately so

i mean depends i'm not sure honestly okay

and nether right now

Łukasz Rozmej: so i think we have still a few bugs iron out and i've just passed um might be good to actually think about scheduling the at least on one of the networks above this work id change um yeah so we are close but i would i would like to have a bit more time to finish everything

marek do you have anything to add

yeah generally i agree with you okay um basu yeah

um we have one hive test left um to complete so we're working towards that we're pretty optimistic on it we still have some concerns with our bonsai implementation and some curing things but nothing merge related and so we're feeling pretty good

nice and i get uh so it appears i don't know i guess team almost called i've been away and training through europe for the last week so i don't really reconcile

given our status no worries um yeah okay and uh i see there's like uh there's obviously uh uh prism tekku on the call uh uh nimbus like yeah anyone else um yeah

Terence Tsao: yeah so this is unrelated to um execution layer but i do think it's like quite important to test um because this is their client and the med boost interaction right at merge especially on a test setup because this is kind of our last chance to do this before the

main event so i posted this pr by alex essentially he he outlined this much transition delay and uh i do think it's important that we merge this or we come we come into some sort of sort of agreement and then put that into defined code and basically test it yeah

Tim Beiko: okay so it would make sense yet to test the mvv boost stuff on gordy um yeah so we've had to run at it before magnet um any other cl teams

Ben Edgington: i think we're good to go agree with terence it would be um good to test that transition but other than that all good

Tim Beiko: and anyone else i missed sorry there's a lot of folks on the

couch

Mark Mackey: lighthouse also wanted to test the transition uh with um an av boost but we had an issue we were actually doing that during the last test network we had an issue um as you know but yeah we've planned to do it rally and we're gonna go for the okay

Tim Beiko: yeah that's actually a good point are we running muv boost on the shadow forks

i have nothing and

Mark Mackey: it's been fine um so far but we just haven't done it

during the during the merge

sorry perry you're going to say something as well yeah uh we have shadowflop 9 scheduled for next week if the relay infrastructure is in place then i can give you some keys and maybe we can test that next week yeah i think perfect that'd be really cool if we can do it yeah

Danny Ryan: yeah i'd love to do it on a subset you know with a noticeable subset like 10 but not 50.

Tim Beiko: yeah okay yeah that's that's uh that makes sense so i guess um you know there's still like some small amount of work to do on all the el salve on all the els um and then on the cls it seems like it's mostly just testing with mev boost um i guess we have the cl call next week we can see how things have gone since then and if uh if we feel comfortable uh picking a ttd for gordy then we can do it then otherwise uh if we need a bit more time we can do two weeks from now uh on on this call but it seems like we're um yeah we're almost ready to pick something for gordy does that generally make sense to people

um i'll take this as no objections oh yeah alex

yeah i was just gonna say since we're here uh if any still teams are listening if you could take a look at the pr tears mentioned 38 on the builder specs uh and just be prepared to discuss on an excel call that would be good

Tim Beiko: cool um and yeah i'll add like i know we've been doing this for a while already but yeah just making sure that all the el teams have someone on the cl call next week as well so if we make a decision for gordy or if there's any issue with your client you can let us know and worst case if if no one from like a cl team can make the or from an el team sorry can make the call just leave a comment on the

agenda if you have like a strong opinion or blocker um cool i guess uh next up uh on the agenda we had one more point on that if i could just follow up on that

Parithosh Jayanthi: of course um in order to make communication a bit easier and give people more time to update nodes uh whenever we decide that you should we also decide main at the same time and we can rather use girly as an abort as in if we find something really we abort the mainnet ttd and we change that or um otherwise we go ahead and we get the benefit of easier communication

Tim Beiko: i think from a communication standpoint it's actually more complicated um and i say this as the person who writes the blog post for all these upgrades and they're all they're pretty complex i think i think what would simplify communication the most is if for gordy the ttd is chosen like the real ttd is chosen um basically from the get-go and that clients have a binary which contains like everything right like where there's no ttd override that has to be done and there's no like you need to download a first version with the high ttd and the second version with the low ttd i feel like if we can run through this process um

and then also like do some stuff around like uh the the staking launch pad and like making sure that that's all up to date um so that it's like smooth for users i think that's probably the best thing like community slash communication wise um and then shortly after that you know if it goes well we can pick the ttd for mainnet um and similarly like and i think for mainnet we also want to pick it pretty close to when it actually happens um so if we were to pick them both at the same time and say there's like you know two weeks before gordy and then two weeks before main net you're like picking it you know at least four weeks in the future i feel like for maintenance if we could pick it on the

order of like three weeks or so in the future that means that there's just a lesser chance that the hash rate uh changes a lot uh in in that period um yeah i know i feel like keeping them separate is better for like a community communications point of view but well maybe other people disagree

Mikhail Kalinin: um but what about bilateral book for the mainnet should we like we could probably pick it at the same time as like early

Tim Beiko: yeah the question for cl teams is do you want to pick the mainnet bellatrix epoch before having seen mev boost work

on gordy which might be possible

Danny Ryan: well there's also that would require also users to upgrade twice because ctd would not be baked into the cl clients oh right they'll have a super high ttd i which is not the end of the world that's not the same thing as them setting that manually on the cli so again it's not the same like ux that might fail here but it is two points of coordination instead of one i would try to just do one if possible

Tim Beiko: yeah and i think i think it's quite possible to do just one because if basically if someone can double the hash rate on mainnet and like make this ttd hit twice as quick they can effectively

51 attack proof of work um which is like something we assume is is not possible or probable today um so it seems like yeah if we if we have a large enough gap between belatrix and ttd you can combine it in one release at the risk of just like hitting ttd maybe a bit later than you that you would if then you thought that the hashrate starts dropping but from a ux perspective that does seem better

Danny Ryan: that's my current preference i think it would have fewer end user errors

yeah this would make sense to

Parithosh Jayanthi: try the exact same thing on girly because the hash

rate slash difficulty is pretty long early as well we can just have uh have valid checks happen maybe one week before ttd supposed to get off a few days before um and then we can change just adjust that tolerance yeah yeah and i think algorithm can be much closer right like you could imagine doing like bellatrix on like a tuesday and then like gordy on like and then ttd on like a thursday or something like an order of days um and then maintenance you probably want like order of a couple weeks yeah i think we just have to be a bit aware um in communicating that because i would also imply people don't have three

weeks to update their node or they don't have up until gtd but rather up until the ballot is hard for yeah which is actually easier to communicate because we know when bellatrix is there's like a timestamp development yeah definitely yeah

Micah Zoltu: so while i can appreciate the ux improvement of having a single release that includes both bellatrix and the tpp um what makes me uncomfortable is the precedent the sets where we're designing switching from designing systems that are resilient against basically you know state-sized attackers to systems that we hope work as long as no one big shows up to play and while i generally agree with the sentiment that it's unlikely i i really dislike the direction we're

taking where we're saying well it's okay don't worry about it like historically we've always worried about every scenario and now we're saying and it's an unlikely scenario and not unlikely in like the cryptographic sense of finding a hash collision like which you know we just put a bug fix in to address a potential hash collision with contracts and eoas and now we're saying you know like that the order chances of that happening are like you know if someone has all the bitcoin hash power for a year they can do that and we fixed it and now we're saying well if someone has 50 percent of youth hash power they can break or merge and just like this direction makes me very uncomfortable i agree

Danny Ryan: it's very unlikely but it just seems like the wrong path to it it's also but contextually this is a

singular event and we're making an assumption about the adversary during a singular event and not a perpetual adversary which i think is certainly a different uh threat model than an adversary in perpetuity in the long long term so i am more comfortable making an assumption that the hash rate cannot double in the next three weeks um as opposed to making an assumption that such an adversary would never exist which i think is the difference in the scenario um i understand i hear you argument

though i

Micah Zoltu: i agree there there is a very valid point that having a persistent defending against a potential future adversary versus a defendant

against an adversary is going to show up in the next few weeks it's certainly different i'm a little bit worried that there's a it's a very blurry line between those and i'll think on it maybe i can be convinced that there's actually a bright line there um but it just it feels like this is it feels like one of the first times we're making this call where we're saying we know there's an adversary out there that is very realistic like someone you know some government could have this much hash power this is not out of the realm of possibility this is very much within the realm of there are potentially actors out there who could do this and we're just kind of ignoring it this is the first time we're doing that i think yeah and that's that's where i'm uncomfortable is this we're crossing a line that we have not

previously crossed and i'm uncomfortable crossing new lines without thinking really hard about you know is this a line that we're okay with crossing again and again and again and again in the future

Tim Beiko: i think i think that's that's reasonable i think uh yeah and and you know there's like two you know two other alternatives one is like yeah you just have two releases and like we did for ruffstein you can pre-warn users that there will be a second release right so um so like from a community and ux standpoint it's less worse to have to download two releases if when you download the first one they tell you you need to then download the second one i think the second one is like if you

want to absolutely have one release that you can play with like the delay between like uh between bellatrix and um and uh and hitting ttd because you can imagine in the worst case it's like that delay could be even longer than it would be with two releases um but yeah i i think yeah you definitely have like a valid point there and it's something we should we should think through

uh more

Micah Zoltu: yeah and to be clear my argument here is almost entirely philosophical like we can set the the difference the time difference between the two so large that it's not just 51 you need like you know 99 kind of thing um and so we can make that number very big and we do have

control of that by separating those two times a lot again my argument here is primarily philosophical because i feel like we're crossing a line that we've not crossed before and i want to make sure that we're very intentionally crossing that line we understand that precedence are strong and people will look back at this and say you know oh we did it before it's okay now like i don't want to make normalize this and you know if we're going to cross this bright line we should set up a new line that is very clear that we don't cross in the future

Tim Beiko: yeah i think i think that's that's a very reasonable stance anyone else have thoughts on that

Danny Ryan: um yeah if we went the other route i

just want us to be extremely careful and explicit and simply maximally simply so on how the ux for users goes because it's order magnitude more complicated

Tim Beiko: yeah and i think it there it's like saying from the very first release like you will need to download the second release um and also probably making making it easy to verify when you've downloaded the second release that your current ttd is actually correct

Danny Ryan: um yeah i guess i'm most worried well i'm worried about a lot of things but i'm

worried somebody then you know updates their execution layer and doesn't update their consensus layer and their consensus there doesn't know about new ttd and things like that just those types of scenarios

Micah Zoltu: yeah don't we have a check between the el and ceo that if they're out of sync with the ttd they'll talk to each other and notice and do something

Danny Ryan: they say they should have an error that goes to the logs but i'll continue running

Marek Moraczyński: we are not doing anything with this

Tim Beiko: so maybe um in the next like week or two

danny and me can just like scope out the pros and cons of the two options and uh yeah share that on like the next cl call or ideally before but i'm at least then sounds good

cool

Danny Ryan: yeah i want to think through the where the ux can fail to decide how bad it's worth yeah one way or the other

Tim Beiko: and then similarly um yeah what's like the failure like what's the attack scenario and you know yeah how different it is than our current

like the current things we're willing to tolerate basically um and i feel also that like whatever we end up using for mainnet we should probably use for gordy so that stakers have the opportunity to run through the entire process once on as it will be on maintenance anything else on this okay

um then we have uh leo from uh the flashbots team to give an update on mev

boost

um yeah i am here um and i mean a new place so let's see how this goes um i have a bunch of things to say so i made an agenda on the pm issue um i will go through them and then at the end we can focus on the things that you find interesting uh most of these things are linked to repositories around so for the things that we don't discuss there are places to continue the discussion

interesting and i'm planning to attend these meetings uh all the time from now on so the updates will be shorter and if you have questions that you want to bring here we will be around to answer them um [Music] so well yeah i'm leo for the ones that i haven't met and i'm coordinating things from flashbacks around the merch so i will be responsible for the things that we will be shipping and i wanted to start with where we are uh so what we have delivered is a permissionless protocol uh for

validators to participate in mvb extraction and a lot of people participating on this specification for this so thanks to all of those uh this is big this is a super big change because in proof of work we have the allow list of miners and that's a severe limitation that's something that we wanted to avoid and that was the first thing to achieve we're there already we have a stable relay that can connect to multiple builders that also enables another avenue of experimentation on the centralization we have a block builder that's optimized for nav extraction and then um

the validators don't have to focus optimize for this they can connect to the relay to the builder and participate on the network immediately uh and we have two consensus clients ready for testing and the others in progress um so this week uh on our sig meeting we were like overwhelmed by all the things that we have in the backlog and all the ideas that come and then we stepped back a little and we checked on this and i this is huge uh we were calling this milestone merged ready uh because these were the basic things that we wanted to have

for the merch and we are already here so yeah sometimes we just like lose the context uh there's still a lot of things to do but i would like us to step back and celebrate because so many people were involved in this i took a look at all the work that went into the consensus clients implementing this and it's a lot of commits a lot of work a lot of trials um so thank you everybody and well now we spend the weekend celebrating uh what comes next uh since we have these two uh consensus clients ready uh we want to publish the

code protesting and start moving our communications around this to get more people to execute these things provide feedback to everything on the stack we spent the week shaking a few issues that were there making a list of things that are expected to be weird on the locks reporting them but i think it's now good enough for people to just give it a try so we have a testing guide uh in the wiki the week is open the code protesting is in a pull request so if you want to take a look change some things uh we are welcome of course

um that's what's immediately next like now in an hour or so we will be making this public what we were planning for the next months goes around three areas first is boost adoption make sure that people are using a boost as a way to mitigate med and super relevant make sure that the big validators that's technicals that exchanges are following this protocol and not trying to solve this on their own way opaque way uh

that everybody is like openly participating on this mev mitigation and maybe spreading so an important thing that we will start working on now is data transparency now that we have the relay and the builder working we can start collecting numbers from what's going on there and making those numbers published public and the idea here is also to uh not play this role of mediators like currently when when searchers feel that miners are cheating they come to us and we do the investigation and then we are in charge of punishing the miners somehow

this is very uncomfortable not a situation that we want so with data transparency we will enable people to identify when somebody is deviating uh exploiting a hole when something is not working as we designed it and then everybody together can figure out a solution not uh using flashbots as a mediator in there and we will be restructuring our community around this two uh the idea being that this shouldn't be a community of searchers and miners as it has been now

this is actually in my mind a community of researchers so all the roles that we're playing all the data that we're collecting all the experiments that we're running uh ultimately are for for decentralization for our handling of meb and we will be working on trying to to design the workflows on everything that goes through our our projects to be supporting that so that's like what i will be coordinating these three months and we will be uh trying to do that on in the open welcoming everybody to participate

um so that's so short-term cultural testing long-term like is restructuring of transparency and community of researchers but then on the process itself the plan what to do next we currently have one relay and one builder uh we want multiple builders in here and we have this issue open to see who's interested in running a builder uh in here we are being careful like we are not super sure that the builders can be permissionless that anybody can be a builder

uh but the the uh specification the protocol enables us to experiment a lot uh so we want to see how to approach this issue uh playing with one more builder that's independent same with 10 more builders that are independent seeing what limitations we find in there and we are writing a post that should come out next week about what are the possible conflicts in here uh where the incentives break if an organization plays a couple of roles in here like what

happens if the searcher is a builder what happens if the validator is a builder uh what happens if the relay is a builder and this is flashback's case in this moment flashbox runs the relay and the builder and it works only because it's trusted right uh we want to to come up with a lot of ideas what are the possible scenarios and what are the risks and then this system should be designed in a way that it incentivizes uh independent parties to be running the roles that could be potentially in conflict uh a lot to dig in there uh that i think

will be the work in the next three months about who are the partners who are the competition or the adversaries how do we put all this energy into something that gives us a decentralized protocol um and um there are many many questions to get there so uh if the merge is happening on september like being realistic with the resources we have what we can achieve is a relay monitor not like permissionless builders not like permissionless relays but

some tool that will the validators will be able to connect to to uh get some reputation uh measure their risk when they are communicating to uh to a really uh yeah but then you see this is like what was overwhelming then the monitor is centralized and we have to decentralize it and it's wow so many things uh but we're getting there uh we need a lot of ideas on this so um on this issue for the agenda i also put a link to the open questions these are like the big things that we're researching uh

that are are [Music] defining how do we move the protocol forward uh please take a look there uh any thoughts any ideas will be super interesting to hear and yeah if you have an interesting one decentralizing this let's do it together let's contribute to answer those questions and then have multiple release multiple builders without this uh this mediation by flashbacks um in the issue sorry i just there's a couple people that have their hands up so i just want to

make sure we can get to them

Tim Beiko: um yeah i don't know if do you want to quickly just wrap up sorry yeah i just want to make sure we can get the yeah to the other folks as well yeah

i'm almost done uh just mentioning that on the issue there are other things that are not like copying research questions but helping practical things uh some are more complicated than others please take a look on the meeting last week we talked about uh sealed options or open options and that might require changes by the consensus clients so those kind of things we need to discuss them early to give time to all the consensus clients to agree and implement or take a an alternative uh yeah and just to close

um a question for you like i want us to test my boost on the early like be ready for the early merge but i wanted to hear your opinion um and a crazy idea came out of all this brainstorming intense work uh that is maybe some proof of work and that sounds like super fun super interesting that will give us a lot of testing that we need like making sure that the systems we have will be ready for the merch but we kind of discarded it because we cannot do both this would require work on map gifs and require like

some uh communications with the miners uh but i just wanted to throw the idea because it was fascinating for me i got super motivated just with this possibility it would shake things around maybe somebody would get interested and want to take it push it forward contribute something there or it's just a crazy idea that we will not implement because we are doing a lot of things already uh yeah and that's all i wanted to mention yes i don't think we have time for many questions so i'll be here

Tim Beiko: awesome yeah yeah thanks for the extensive update

uh mika

Micah Zoltu: i think you were first

when you said you only have one builder um what do you mean by that you mean there's only one person who has come forward saying i'm going to build a block builder or do you mean there's only one person participating in

desktops

no i'm i mean that there's a flashbots builder uh and the touchpads really is currently communicating to the passwords builder uh on this issue uh where we are asking people to come forward and declare their interest there are already like three poor people interested also through our network we know there are more people interested in running a builder uh just we want we want everybody who's interested in this to to participate

openly say and contribute

so

Micah Zoltu: so if i understand correctly then you're just saying that currently flash spots the relay is permissioned and you only have one person that is permissioned at the moment is that correct

uh yeah that was that was the merge ready uh milestone and now we will add more builders

Mark Mackey: um it's dreamer hey so in our testing of med boost so far uh we have this mock relay that sean anderson built that just like hits the execution engine so we don't actually have any d

but we can at least test the code pass through the consensus layer um on lighthouse but um is it possible for you guys to build a main net relay so that we can like use have actual mev on the shadow forks

um yes it's possible um why don't we take this shadow park that will come next week as a way to see what's required this this lack of mev to test is what we will be tackling now with the

community and testing work streams like yeah we are we are sending we are asking the relay for blocks and the relay is talking to the builder for blocks and the builder is talking to the mainland to the to the to the mempool for transactions and then we are testing all the pieces but we're not testing this end to end so we need searchers to be involved in the testing and we need sources of order flow uh so the next step on this work stream is to talk to our top searchers to talk to uni swap all the main sources for for meb extraction

and start sending some some significant significative uh transactions into these test nets so we have the idea of doing a capture the meth uh incentivized testing situation so i don't know we highlight the smarter meb extraction and this gives a reward to the searcher that did it the value data that participated uh this requires some coordination but if we want to be safe by the merge i think we need like more realistic uh usage on these networks

we'll be working on

it thanks uh terence

Terence Tsao: yeah so um thanks for the summary dio i just have two high level points i think one i think one of the points is that if we do think we're gonna emerge in like september october it's quite important to come into a date where we say that we are gonna freeze the api spec meaning that there's going to be no more changes to the builder and then the computer api so that we can also total freeze on our end too so yeah just just a point on that the second point i have is that um we have a blogger where

we send 100 registrations that it comes out so that is blocking us from testing on robson and also sapodia but we can also take that offline so yeah thank you

uh yeah so the like materials entries are working on the resilience of this relay and builder uh right so we are starting to hit it hard now with some of these test nets and that tells us what do we need to do to to make it responsive um i know they they yeah so these problems and i know they are working on them i

cannot comment a lot more so i will let you talk to them uh there were some strategies to to reduce the load and to increase the capacity of the machines uh and this is a super good experimentation to figure out what are the requirements for really a builder to be actually stable uh your other point about the api press like yes in my mind uh the api was freezed after amsterdam uh frozen after amsterdam but unexpected things come so if it worked for me i will just face it

now uh all the things that will appear will be dealt post merch as as as our plan was calling this merch ready uh but that means some limitations like this open for closed auction we will have to deal with whatever is possible now so i don't know maybe insist more on taking a look at what's open what we want to happen for the merge and come to the agreement that all the other things will not be possible because we will press the api then we focus on testing on resilience on this on adoption

on measuring the numbers uh and if nothing crazy happens then okay we're good to go for the merch if something deadly appears on the testing okay we will have to think again what does that mean do we have time to iterate once more on the api uh will that affect the merge or not um yeah so this is actually up to you if it were for me we close this and we start playing with all the other topics but we will not make changes until the merge happens any other questions

all right thanks a lot for the time i insist thanks a lot for the amazing work so many hands going on in here so step back celebrate because what comes next is going to be intense again uh i'm super happy with being part of this

yeah like

Mark Mackey: i guess there is something i want to bring up um i i think we were thinking that it might be best to uh disable uh mev during the merge until finalization um it's i mean it's just it's a preference but i thought it might be something i want to discuss just to

you know eliminate the things that are going on during the merge

Danny Ryan: um i believe there's a pr up for this and that it's on the agenda for the call in one week the consensus there and i've not heard any uh anyone against it yet

yeah we will follow whatever your preference is uh we can uh not answer on the relay until we should answer so that's another safeguard uh i think that's why i want to test in barely we can prepare for that we can see how that goes we can see what are the

risks by not extracting mbb on the first blocks how much do we want to wait uh well yeah sounds okay

Micah Zoltu: i can give you a sneak peek of a potential argument for next week's call against it and that is that mev extraction is still likely to happen it's just going to move somewhere else and that somewhere else may be engineered more poorly than mep boosters and so we're just moving people out of the light thing that's engineered and tested by people we trust over to things that are engineered by people we don't um right but that's not clear that is

it

obviously

Tim Beiko: well the the win is there's less interaction with the with the client software so it's like you're willing to take more on-chain and obviously off-chain craziness um yeah at the cost of just or i guess for the benefit of just having less stuff

touching the nose so

Danny Ryan: 20 of validators decide to use some crazy system and they go offline that's that's probably okay

but yeah i we can talk about this yeah

Danny Ryan: we'll we'll slot some good time for it yeah

uh like my purpose is to to go live

from the first block i understand the position of like there are so many things going on the first block uh but yeah like are we opening like by not mitigating this up front are we opening the door for other crazy things to happen uh we will follow you yeah happy

too good content

Mark Mackey: that's just a command the command line override you know defaulting to disabling until after finalization but if you want to pass dash dash enable dash dash dangerous dash don't do this then fine all right

there's

uh one more thing i wanted to mention so we have all these

open questions and people are coming like wanting to implement the relays and this is like fascinating but when we talk to them we realize that they don't understand mev and they don't understand the risk or the responsibilities or the temptation of like going rock bro if your relay becomes used so my idea would be to insist on these people who want to run a relay to first from a builder then get an idea of what's happening get practice get our support while uh getting their their builders strong and then yeah we cannot control who runs

a builder and which validators go to which builders but at least we would like them to be informed so if you know or you want to participate on this running a builder running a relay please talk to us go to the issue tell us and we will be here to support you uh and also like to hold you accountable for like what will happen and ask for your collaboration answering these hard questions

Tim Beiko: cool anything else around any v boost um

okay if not uh we were going to have a four fours update uh today but unfortunately henry couldn't uh make it uh there was an issue that was including the agenda um and uh i know that uh alex and and like client you have like a bit of context on there i don't know if there's anything quickly you want to share

Lightclient: otherwise we can just uh move all of this to the to the next call yeah i don't have ugly okay i i get i can just say quickly that henry is still working on this implementation that can import and

export this data this circle data out of depth and shortly he plans to take this format that we've sort of discussed in a few channels and put it into an eip so that'll probably happen for the next all core devs so be on the lookout for that i'm sure we'll post it on one of the discord channels any questions

Tim Beiko: comments thoughts about that okay um and then if not uh there's uh just two quick announcements uh the first is this is the last all core devs calls on the

friday so we agreed on the last call to move them to thursdays so the next awkward devs is thursday july 21st and not friday july 22nd uh so yeah please uh update your calendars we've updated the protocol uh call calendar as well um oh yeah last awkward devs on on a friday today and then uh final quick announcement next friday um next friday we uh have another merge community called uh so for um application developers infrastructure providers and whatnot who have questions

about the merge and this is the place to come and chat about it uh if uh client teams uh want to send a couple people to show up it's always useful to answer some like much more in the weed uh in in the week's questions um yeah so the link for that as well as in the uh agenda for all core devs today um and yeah i think that's it anything else anyone wanted to chat about before we wrap up

wasn't four four four four on the agenda

Tim Beiko: yeah we just that's what i like clients just covered so uh henry was supposed to give a full update but he's not there but um basically oh okay it went nice it went

quite that fast yeah yeah um and i i just the last link in the comments is is like the longer i guess

progress up there there it is

Greg Colvin: okay yeah i looked at that all we said is we're gonna talk about it later

Tim Beiko: yeah yeah and henry you should have it i've already read this okay yeah and there should be just like a proper eip for like this specification

of the data yeah i already i've already studied this so thanks of course anything else

Tim Beiko: okay well yeah thanks everyone and we'll

see you in thursday on two weeks from now thank you see ya thanks everyone bye thanks bye thanks bye

[Music] so

[Music] [Music] [Music]

[Applause] [Music] [Music] so

[Music] [Music] [Music] you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.