All Core Devs - Execution #144

Transcript

[Music] [Music] so

[Music] [Music] [Music]

do [Music] [Music] [Music] [Music]

Tim Beiko: okay hi everyone uh welcome to all core devs number 144. um we have a long list of merge related stuff to discuss um and then towards the end um also wanted to touch on the executable specs uh work that's been going on um so i guess um just to start with like a quick announcement for everyone um we discussed this on the last call and i believe two calls ago uh but sepolia is gonna be having a post merge upgrade to set kind of a

merge block for for peer management um this is scheduled uh for august 17th uh it's there's a blog post that went out yesterday with all the client releases um so if you're listening and you run a note on sepolia you need to upgrade the upgrade basically just changes peer management it doesn't add any new features or remove any other functionality um and also the client releases that were advertised for the gordy uh proof of stake transition also work uh for the sepolia upgrade um and lastly this upgrade is only something on the execution layer so if you're on a node you only need to update your execution layer client um not

necessarily your consensus layer client um so again people can check out uh blog ethereum.org for um just the client the client releases but um yeah please uh upgrade that anything or anyone on simpoli upgrade if not um we had an upgrade on crater just a few hours ago so uh getting it ready for the um the merge um i know it seems like overall it went uh quite well i don't know danny perry do either of you want to give a quick update on what happens on

prayer

Danny Ryan: i think perry has a bit more detailed view

Parithosh Jayanthi: sure so the network was roughly at 90ish percent attestation performance pre uh better tricks for people uh yeah before beta6 and after we dropped down to roughly 81 82-ish percent um it seems to be stable and holding that at least according to the tags on beacon chain i think that one client team might have a subset of their validators that need to be updated but besides that everyone looks perfectly fine i guess the missing attestation rate might just be from um

communities on validators that haven't been updated i haven't heard of any issues and in general proposals look healthy there aren't too many unexpected missed proposals or anything so i'd say it was a it was a as good as we could hope for upgrade for an

uninside device test net

Danny Ryan: yeah that's what it looked like to me as well

Parithosh Jayanthi: it would be awesome if any validator is running on proto who hasn't updated if you could update and get the participation rates back up it would just be nice to have a bigger buffer when the merge actually happens

Danny Ryan: and i believe uh one of these one of the client teams didn't update their execution layer client um so just a warning you have to do both i know that uh it seems like configuration errors are increasingly you know the primary things that we're battling with so we'll continue to work on guides but make sure if you're listening to this that you work extra hard on getting both sides of that thing upgraded

cool

Tim Beiko: anyone else have thoughts about uh bellatrix operator

Adrian Sutton: it's probably worth noting that we've

discovered that teku and prism no longer talk to each other uh on the network um it's a multiplexer config thing in the latest rc prism and will be fixed it's not actually mode related but hard fork made it quite clear as we watched all the upgraded nodes disconnect and all the all the non-upgraded ones also disconnected because they didn't have bellatrix so it was actually really good to see that working in that we quickly disconnected peers that hadn't updated that were formed and at the moment there's something like i don't know must be about 40 of the network between teku and prisms team nodes um that aren't in any way directly connected and

they're routing fine around the gloss through other clients and it's all just still working which is actually a really cool experiment um so that's kind of good

Terence Tsao: yeah i just want to confirm on that as well we fixed it right away we were sending the right dubai code so yeah it should be out in our next week's release

yeah

Adrian Sutton: thanks for the quick support on that by the way that's awesome

Tim Beiko: sweet anything else on dielectrics or predator electrics okay and then

uh one more network upgrade we had gordy shadow fork six uh literally hours ago as well um you want to give us an update on this

sperry yeah

Parithosh Jayanthi: uh so belly shadow fox 6 hit ttd earlier today um just a primer on the network it's a girly shadow fork and roughly 30 percent of the network was running mev boost it was planned to test mev boost through the transition um we were at roughly 97-ish percent at a station participation pre uh ttd and drop down to about 94 post but it seems like the drop isn't really related to the merge but rather a couple

notes to stand out of disk space and uh one or two nodes was still syncing up the head um but otherwise we look great there on the mev boost front um no real issues noticed at least from a network standpoint we did notice that one of the nodes that was farther away so it was in india whereas the layer was not wrong in the us um there was really high latency but it could be related to the machine itself like the the network card just looks weird we're not getting great throughput on it um i'll look into the node itself but yeah otherwise i don't think we've noticed any other issues um you can check for on each stats to

have a live view of what's going on

there awesome thanks

Tim Beiko: any thoughts comments on the shuttle fork okay so yeah um you can see they have done well across the the various upgrades um [Music] mikhail i you had a whole bunch of things uh you audited the chat about so um i think yeah we can start going through them uh you had two uh requests for comments on the engine api spec um i'll share the first one in the chat here

um issue 270. um yeah you want to give some quick uh good context there

Mikhail Kalinin: yeah sure thanks tim uh first of all both of these rfcs like just a discussion around proposals that we will have after the merge not for the merge definitely and this one this that have been dropped in the chat is about removing invalid block hash it's basically about replacing the invalid block cache status with invalid status plus the latest valid hash set to new uh it has been this opportunity has been opened recently by a residentiation to the engine api

um the other one is about like it's like more uh involving uh is payload status is worth reworking it's actually not like complete rework of them but making them uh the payload status is more clear to allow for cl to distinguish the two states of el it is communicating with one state is when el have to go to the network pull some data from remote peers to validate a particular payload and the other state is when it has enough it has all data required to do the

validation locally but it just needs to to make some computation basically executing like a bunch of blocks to validate the payload in question uh as for now current statuses that we have like accepted and sinking uh they're a bit wig uh with respect to this like uh distinguishing between these two states and this is probably what cl's uh would like to have to start like utilizing this uh i mean to start making a um making a difference between accepted and sinking statuses so i'm just uh yeah i just want to engage ciel and the alkaline developers to look into these two rfcs with their comments

um i don't you have like a lot of things to do and this is like post marriage but anyway um would be great if more people take a look on them

so

Tim Beiko: thanks um any any thoughts people want to share now about either of these okay um and yeah it seems like the second one already has some uh comments from some cl uh developers so um i suspect we'll be talking about that on the ceo calls as well um

okay i guess the next up uh you also had a pr about uh checkpoint sync in the beacon

apis

Mikhail Kalinin: yeah this pr uh proposes to introduce two endpoints um uh like uh yeah the the idea of these twin points came out of a discussion discord awkwardest channel with mike adrian and paul probably some other members of the community but the um the idea of these two endpoints is to allow for the the following separation of concerns there are state providers that

provides a state at a finalized checkpoint within big subject date period but this finalistic point is on on the choice of the state provider so it may be like the most recent finalized or the previous one it can be updated on the face that the state provider choose and the other endpoint is for trust providers so they expose the tiny api to verify that the block route from this finalized checkpoint state is matches the block route that all trust providers expose all dress providers has so we basically

request the state first presumably it's like the most recent the state of the most recent finalistic point then you take a slot from this state from from the block header that is in the state take the slot ask as many trust providers as you have about the finalized route the finalized block route in this slot uh if if they all agree uh on this route and if this route actually matches the one that you can compute out of this uh state that you have just pulled off uh then you're free to start sinking from this like checkpoint yeah the idea is to

make this uh checkpoint sync easier for state providers and increase the lag the number of state providers and networks have increased the adoption uh of of the checkpoint thing from from that standpoint um and yeah if entry and or anybody else can want to elaborate on that

pretty chiming

Danny Ryan: yeah i agree this is good i agree that it's the most sane way to do kind of multi-verification when you're doing this bootstrapping and we can take it to the api repo i

guess my one concern is defining it in the api repo where that api generally is not expected to be exposed publicly um and then having a name space that is i just want to be very careful when we're making that decision if we make that decision to kind of couple these things just because it might result in more users exposing accidentally exposing endpoints that are easy to be dosed but nonetheless uh i like the i like the pattern a lot

Micah Zoltu: is the trust endpoint easy to be dust or just the state endpoint

Danny Ryan: well um i guess what i'm saying is the rest of the api in the beacon apis

those are user apis you know which all of them are easy to be dosed you know it's stuff that you just don't i think in such a way that it's like hardened like a p2p interface so just having it in the same kind of definition makes me a bit worried but or you could also put big bolded if you're using this name space this is you know be careful

Micah Zoltu: yeah i think i think that's a reasonable argument for putting it somewhere else and in particular the um the trust provider like you really want as many people exposing that as possible and so making it very easy for an operator to expose that without exposing everything else feels like a win

Adrian Sutton: yeah so i think everything is being said spot on um but

also i think um it's useful to be able to see those two apis as kind of a separate thing as well so like you're not not expecting to provide the whole api they are kind of a concept in themselves um and some of the details of how you'd implement them as a particularly as a state provider might differ to what a a beacon node would expose directly because you could add caching or you know not pull the state as regularly that kind of thing it doesn't have to be the latest finalized um so i think that's just a matter of making it clear in the description and i need to

to read that a bit more carefully but generally it looked good and i think the apis make an awful lot of sense and should work really well probably the only catch with it is that it does bake in the idea of being able to start from a single state which is possible but it is a bunch more work for people for clients that that haven't gone that far yet and are using a state and the block that goes with it a couple of clients have some extra conditions on exactly which finalized checkpoint needs to be used and that kind of thing as well so i don't know how much that's a problem for other clients or

whether they're planning to get further along and make it easy and this api can push them along or whether we need to also provide access to a matching block or something like that

it means

Mikhail Kalinin: so they have to pull the block with body right

Adrian Sutton: yeah so i i'd much prefer being able to just start from state i think it opens up a lot of simplicity for observing it and a whole bunch of options but i do appreciate is a bunch of work

Micah Zoltu: is there any compelling argument for not having this long term or is the argument here just that

not all the clients have this feature yet and it may be a while before they have it

um i

Adrian Sutton: it does have some impact into your database a bit um so there might be some arguments there uh i don't know i think once you've done the work it's disappeared in tekku and you know it's not something i ever think about again but yeah i don't know how that plays out in other client architectures necessarily and whether that becomes more of a burden longer term to keep supporting this one case where you might only have a block header that said if you wanted to you can get started with just a block header and connect to the network and request that

full block to to make the problem go away it just depends again on how that fits into the architecture

Danny Ryan: should we um take you to the issue and then maybe bring it up again on the next consensus layer call as we've thought about a bit more and maybe done a bit of refinement

Adrian Sutton: sounds good

Micah Zoltu: can we get the same feature in the execution

plan i don't know

Adrian Sutton: if you can shrink the state by about

a hundred times you do stay with you don't need it at

all you just you start with the final you can play and then go from there okay yeah oh sorry

Mikhail Kalinin: yeah you've mentioned yeah you've mentioned the um probably publishing it like yeah it's already in a separated namespace which is called checkpoint uh but if there is an idea to put it like separately from beacon apis i don't know if it makes sense i mean the complexity and maintaining like separate it as a separate api but yeah let's just discuss this probably in discord

and then

yep sounds good

Tim Beiko: um and then last thing uh mikhail you had another comment about uh basically competing uh ttd blocks um based on some conversations on the discord this week uh you want to give some quick background on that as well

Mikhail Kalinin: yes uh this is like generally to check to double check this one of assumptions uh um yeah let me explain why it uh appeared so we have we recently we learned that some um execution layer clients uh do not uh instantly process uh blocks that are

not from like the canonical chain so if if yeah in particular it was the issue with the nethermind it received like terminal block a with a total difficulty uh reaching the terminal total difficulty barrier and then we have like the terminal block b which also like a terminal block uh in terms of uh terminal ttd but it had either lower total difficulty or the single difficulty as the previous one and in this case nethermine and aragon they don't instantly process these blocks so they just put them into the block tree and that's it yeah we had the

test cases for for for this and we had the fixes in these clients if like the transition block build top of like the block b uh will uh appear then these clients will just execute all they have after this transition block and then execute the transition block to prevent the cl from getting stuck at the merge transition block because of safe slots to import optimistically and one related question to uh i i assume that every uh el client do propagate uh every preferred block in spite of like

ex executing it or not like if in this case if a terminal block b hasn't been executed it will be propagated after verifying the proofwork seal if it's correct it's propagated this is very um yeah this is also critical for the transition and yeah my assumption that every el client does this i just wanted to double check it if any if any client does not do it like it does not propagate a block that it hasn't processed let us know

or you yeah

Marek Moraczyński: so i think uh cli what clients are doing is

they are propagate the blocks without processing but only to small fraction of the peers if this is correct and after processing they are propagate to all peers and i have to verify another my behavior here and i'm a bit worried that we are not working uh in exactly in this way that we described but uh i will check

Mikhail Kalinin: uh but if it it it propagates a block to a small fraction of peers but other peers will should receive like this new block hashes message isn't it

Marek Moraczyński: after processing right yes

yeah not after the sponsor no but

Marek Moraczyński: no no let me send links linked to uh specification

Péter Szilágyi: so um the way block broadcasting should work is that when an execution client receives a block it will i mean a pre-merged block it will um do a proof-of-work verification if that passes then broadcast it to square root of appears that actually imports the block locally and once imported import succeeds then it actually announces it and the reasoning is to actually import the block locally

before announcing it is before because if i announce the block to my peers and i don't have it yet imported and then when my peers actually request the blog i will not have it yet in my local chains i won't be able to serve it so we do need to actually import it before announcing it to the rest of our peers but uh doing the square root broadcast already propagates through the entire network it's just these degenerate links where you have some star topology or some many nodes hidden behind some nat gateway only those are actually requiring this uh announcing mechanism but and as for the

as for the ttd um split at least in gas we have when we have transition we have two markers one of them is that we've left uh proof of work and the other is that we entered proof of stake uh entering crop state means that we receive the finalized blocking proof of stake so give or take 32 blocks and we only disable block propagation when we actually finalize the prostate so within those initial 32 block transitional time period we will still broadcast all the ttd blocks so if i have 10 different dtd blocks i will keep broadcasting and

announcing them

until the pos now for final ends thanks um i'm curious to see i base you aragon

Tim Beiko: um yeah how does it work on your end

Andrew Ashikhmin: uh i think currently we have some problems some flaws in the logic around multiple terminal power blocks so i look into improving the logic and uh yeah just making sure i'll check how we got ship to coming up your w blocks and try to improve the logic

got it thanks and uh basu

yeah um so

Gary Schulte: we have a similar behavior to disable block propagation uh after finalized i'm not certain i'm gonna have to get back with you about our behavior for gossiping uh non-processed blocks though non-canonical blocks

okay got it yeah the

Justin Florentine: other thing i would add to that is that it is the second finalize that we stop after

yeah i

Marek Moraczyński: i want to add that for another mighty it's the same with stopping block propagation when we finalized

got it um okay why the

Micah Zoltu: second finalized

i guess yes for base you yeah

Justin Florentine: sorry micah could you repeat that

Micah Zoltu: why do you wait until the second finalized instead of the first finalized

uh that's how we interpreted the spec actually

Mikhail Kalinin: um okay so let's not go into this details around this person second i mean yeah so as long as new blocks are sent to square root of number of beers um yeah it should be fine right we'll have those blocks disseminated across the network

unless there are some synchrony issues

okay okay

Felix Lange: uh by the way can i ask a quick question do you feel like it would be necessary uh [Music] at some point you just specify this uh the behavior around the merge in the network spec or is it just sufficient the way it is now i mean we will have to remove the propagation from the

network second here

Tim Beiko: you're breaking up a lot

felix but i think we got the gist of what you were asking like should we add the merge uh peer-to-peer behavior to the network spec directly

uh mikael

Mikhail Kalinin: um good question we have this network section in the in the eip um which is a bit odd because it rather should be like a separate eip um on the network track yeah probably probably we should deprecate the um block announcements and block new block propagation in this back after the merge

i'm

Péter Szilágyi: not sure i understood felix correctly maybe uh the way i kind of understood him was he was asking for it but we want to spec out this network behavior during the transition period for the merge and uh i mean honestly it seems if we want so that would it might make sense if we wanted to support this merge transition uh for multiple occasions but that doesn't really hold too much we transitioned all our networks already or transitioning and we will transition may not too so from that point onward the question is how how long we will do we want to maintain this capability to transition

one network into the other my two cents would be that long term we should try to deprecate supporting non-transition networks and if we can somehow make some small tool to enable spinning up an already part already transition networks for private networks then that would actually be the last nail in the coffin and then it would allow us to actually remove quite a lot of code so my two senses that we should try to push towards [Music] not supporting non-traditional networks anymore which would also entail not

having to document it

Tim Beiko: it does seem reasonable to like be able to drop this at some point especially if there's like more like pretty significant changes that happened the execution layer so you know things like vertical tries and whatnot like it seems like if we can have a clean slate to start that work on that'd be valuable

for clean slate

Péter Szilágyi: i don't know if we ever can because you still need to import the chain history so clean slate would also entail somehow dropping all the ancient history so that's right discussion but still pre-merge and post-merge at least

synchronization is quite different right and it would be nice to support both and also the problem is that currently all the clients need to support this transition but this would be a mechanism that will never be tested anymore so it will be we just test it out for the test test now we will do it for mainnet and from that point onward we will never ever test it again live so if anyone else relies on it i mean it could always be like this lottery that doesn't still work we don't know we don't care it's probably not something you want to have around for too long

Tim Beiko: um okay i guess just yeah coming back to

the the ttd block gossip issue it seems like the next step is for like the different client teams to look into what their actual behavior is and potentially make some changes uh to ensure that they are gossiping at least a fraction of uh what would be non-canonical blocks but that are still valid uh ttd blocks um before things are finalized does that sound right to everyone

Micah Zoltu: why only a fraction

so the spec the spec let's say basically that's the current um period of respect like unless you import it you only gossip a fraction of it i guess if they can gossip to all their peers

Micah Zoltu: i see that's

even better so you're saying some clients some clients already don't gossip all blocks and so that's it's okay if you continue to do that behavior with the ttd blocks that correct

Tim Beiko: yeah that was my understanding but they don't gossip to all their peers until they've imported it locally but if they never import that block locally um they still gossip it to like the square root of their peers and that would allow the block to propagate on the network

yeah

Péter Szilágyi: you guys were talking about different things as far as i know tim was talking about uh not propagating every block so every block would get propagated but not very peer and mike was talking about uh certain blocks not getting propagated at

all

Tim Beiko: right you're right yeah sorry so yeah yeah i was just back to like mikhail's original issue if you hit ttd you get competing ttd blocks which you don't import because their difficulty is lower than what you see as the as the your kind of canonical tt block you should still gossip those to at least the square root of your peers uh assuming the proof of work cl is valid and that and even though that doesn't propagate them to every peer it should be good enough in the case of multiple conflicting ttd blocks is that is that correct

yeah

that's great okay yeah um so we want all blocks all all terminal blocks disseminated despite of they've been possessed or not so yeah that's important yeah

Tim Beiko: and obviously it would be ideal if clients sent that to all their peers but that might just be more complex and if like the current spec if just following the current spec the percentage to like a square root is good enough then that that's that will work

um you really don't need to split your objects so that's just

it here never worked like that so we i think we never ever broadcast the blocks to everything up here oh

Tim Beiko: got it okay does that make sense for everyone okay sweet um next up uh we have uh chris i believe from the flashbacks team to talk about uh the latest on the mev boost side

um there's a lot of people on the screen so oh yeah chris okay yep yeah awesome hey

Chris Hager: um let me start with a quick update on the release from today's curly shadow fork uh everything went well we were just seeing some uh connectivity issues from servers in india um sending the validator registrations um like if there is one thousand but the is sent at once it's about half a megabyte of data and servers with slow peering might not get that data across to the relays within the default two second timeout that seems rather edge case with really bad connectivity but still something to keep in mind that

we will broken documentation on setting the patch sizes and improving or changing the request timeouts which is by default uh two seconds but everything rockwell under really here i really want to announce that we are going to open source the the relay that we are running and we are waiting for some more updates we have to stabilize some interfaces and and clean up the database structure and do a source code audit but we'll probably release it under the agpi license please chime in if you have any opinions the idea is if http that

people that change it or redistribute it in bif changes would need to publish their changes as well which might help lead to a active open source ecosystem and we believe that our relay has been pretty successfully stress tested and this should be a good starting point for other people [Music] yeah so we expect that to happen early september it needs a few more weeks but we are committed to opening up and uh on the point of opening and open source software i think

many people are not aware that we do have an open source builder and relay that anybody can run okay you know what i will during this call later i see some chats about my licenses and i will just create an issue in the math boost repository and post this here in the chat in a bit where everybody can chime in with their opinions about licenses um yeah open to discussion here and the repository just posted the boost gap builder this is a builder and really implementation so if anybody is interested in just running a a simple like experimental development builder and really this is a repository that you can use that implements all designing

all the apis and the block production i would probably not use that as a production [Music] relay because it cannot handle it's like a single process it cannot handle external submissions it's probably not can cannot handle a lot of concurrent validator registrations but it's a starting point for anybody that wants to play with the code and this is the best that we have until we release the proper resource code any any thoughts or questions on

us

Tim Beiko: very very cool to hear you are going to

open source the everyday so that's yeah that's pretty good news

yeah

Chris Hager: we are very happy about it too

Tim Beiko: um anyone have questions thoughts okay and yeah i guess we can keep an eye on the the mvv boost repo for this uh licensing discussion because uh as you said there seems to be some strong opinion in the chat um sweet um okay and then uh one last thing i had on on uh on the merge um is uh basically the dag size um so

um for context that is like how much like data i guess is required uh in ram to mine ethereum um and then every time it exceeds some specific amount it makes some hardware that's mining ethereum uh obsolete because they just can't store that amount um and so we're about at five gigs on ethereum magnet uh right now and it's expected to exceed five gigs uh basically two weeks from now which will lead to like some drop in hash rate you would expect on the network um like any any kind of uh machine that uses a five gig uh card

won't be able to mine um and i guess this kind of affects how mute we may want to choose a ttd for mainnet um just because you know if we have this this uh this drop in hash rate um it affects how long it takes to actually hit the ttd um so given i guess you know the the the the rate at which things are going um and like the the fork on gordy that's happening in the next week or so um the next awkward devs happens basically the day after this this dag size increase so we'd be able to see the impact uh on mainnet hashrate um would people feel confident potentially setting like a ttd for mainnet right

then um once we have kind of this number uh and and can estimate the hash rate um setting it before might just lead to like bad estimations because we don't know how much we're gonna lose it might be one percent of hash rate they might be five um i would doubt it's something massive um you know on the order of like 20 that's five gigs um but yeah i guess just curious about people's thoughts on that like um assuming we we doing we do exceed kind of this threshold right uh right before the next uh courthouse calls um yeah oh and there's a good question in the chat how much did the hashtag drop

at the four gigs um i'm not quite sure but we can probably see it on ether scan um and if we can't see it it might be a good indication that the drop is minimal also it was block 11 million 520 which either scam does not give me on the hashrate chart so 11 million uh yeah 11 million that was okay no that doesn't make sense i think there's a zero missing here

sorry okay so that was december 25 2020 uh that we exceeded the hash rate and so if you look at so on december 2020 the hash rate was going up already so we were like in a pretty kind of uphill trend um and so it seems we had like a pretty minimal drop that was kind of back up in the in the past few days but it's um yeah it was in the world where like the hashrate was was going up quite quickly um

so in yeah i guess this is like a bit of a different situation hashrate's been kind of stable for the past month or so and it's definitely gone down in like the past six months um but you know based on that it wasn't something like a 10 20 drop so it should be like a a minimal impact um which means you know we could potentially choose like uh a ttd before and maybe we would just hit it slightly later um yeah so i guess i'm curious yeah from client teams like does like around you know two-ish weeks from now seem reasonable assuming that things go well on

gordy

Ben Edgington: um

tim uh what what lead time would you anticipate i mean working backwards you know if you wanted to target um day x of september then you know would you want to decide four weeks ahead five weeks ahead three weeks

Tim Beiko: yeah that's a good question i would like four ish um and the reason you know we've talked it on about like these potential mining attacks where like you know if there's a ton of my hashtag that comes online um you hit you have like ttd hit before bellatrix so it seems like having something where it's like you know we we choose the we choose the the the bellatrix uh epoc and the ttd uh we give people you know a bit under a week to put out a release and then you would

expect bellatrix to hit like two-ish weeks after that and you could aim for a ttd that's like seven to ten days after bellatrix is hit um it's it's just hard there's like high kind of error bands on on the ttd depending on like how far you look at the current difficulty and what assumptions you make about the rate um but yeah you could think something like two and a half weeks before belatrix which is effectively when everyone has to upgrade and then another like seven to ten days i think before ttd gives us um enough like margin so that uh we're quite confident we would not hit it before belatrix and okay yeah so there is a comment also

from youtube on the a6 so saying that five gigs is actually quite a popular size for asics so um the drop might be more significant there um but yeah i guess yeah so that means back to your question then like if we chose something in two weeks it means clients would need a release and you know call it like two and a half weeks from now so like the week of september uh the week of august 22nd um that would be ready for maintenance and is that something that's like realistic and maybe okay maybe i'll wait to flip this

does any client team feel like this is not realistic um or that it's just like too early to tell we can obviously discuss this on the cl call next week as well but yeah does anyone feel like this is like completely impossible to hit

now

Péter Szilágyi: honestly i think we should uh if everything's fine girly we should bite the bullet and do something

Tim Beiko: okay um and and gordy by the way is scheduled um i think right now it's gonna hit next wednesday so it might happen before the cl call next week it might not um so we'll i guess we'll see by then um

one thing we can also do is um you know we we can probably uh set like an epoch uh for on the cl call because that's that's quite uh fixed in time and then um yeah wait a couple more days and agree on the ttd um if we wanted to set it even like before the awkwardest call we can do it async on on the specs repo once we've seen um yeah once we've seen gordy stabilize for a few days after the fork because i don't think we'll get that yeah i think on the next cl call we'll have had gordy live for like less than 24 hours okay

then

Adrian Sutton: is there anything we should try and do before that call so that we're ready like a lot of the times we have these conversations about and we decide on a call that we should set a ttd and then someone goes away and works it out

then we should get questions yeah so

Tim Beiko: yeah so we have uh we have mario uh have all on uh on the protocol support team at the ef he's been doing a bunch of estimations for different ttd's and like based on historical hash rates i think it probably makes sense to have him come in and talk about that on like the cl call next week and kind of share what like some numbers could be assuming things are looking good and and what the rationale is is behind them um and he can probably share that publicly before then so like people have a chance to

review before um but yeah we can definitely share share those estimates so that like on the call everyone agrees at least of like what are like the parameters we're looking at and what's like you know the refrain which we're going for and then even if we select the actual number a few days after that um people should be on the same page

about it

Adrian Sutton: yeah i mean i'd almost be keen on next week's call to pick the numbers we would go with once we're happy with calling then it's just a break yeah so particularly if golly hasn't actually merged or it hasn't been long we can give it the time but be close to it anyway

Tim Beiko: yeah and then the only the only potential risk of doing that is if 5 gig a6 are significant part

of the hash rate it kind of changes the calculation right like if we lose 10 overnight it means we whatever ttd we set we could have said 10 lower and kind of hit at the same time but that we won't know until basically august 18th yeah

uh yeah endscar yeah i

Ansgar Dietrichs: just uh wanted to briefly double check so um it's basically the idea that because of this five gigabyte issue we would not want to because i think right there was talk about scheduling like an irregular acd call right after yeah

that's really fun

Tim Beiko: right right yeah so yeah yeah i feel like given that the the the

drop would happen literally on the 17th keeping el cordevs on the thursday is probably like the best because if it's like we'll have just more data between the wednesday and thursday to make a call about the actual ttd if we want to set it there so i would lean right and if i want to ignore yeah sorry as if we want to ignore the five gig issue we can probably have a number on the cl call the thursday before right

Ansgar Dietrichs: right yeah yeah i mean i'm just um i mean i don't know it almost fits early two two two two i could just put about like a couple days back and forth but i mean i think every day right we have like a and i don't know what the exact

number was but i know so like some two double-digit millions of dollars of issuance so it's kind of like if even if we just you know uh uh a week quicker that's still a couple hundred million dollars that we save so um are we because looking at the week right that would we would probably not if we if we keep on the regular thursday or whatever that means we won't be able to have releases out by the end of that week of course so that does mean that we kind of shift everything one week to the right um so i'm just wondering if basically this uh five gigabyte issue is enough of a cons it doesn't to me it doesn't seem like enough of concern um necessarily to to kind of recur that extra one-week cost but also maybe i don't know we've

waited for like two couple of years we might as well wait one more week

Tim Beiko: yeah i i agree with you though like like you know days and weeks for sure matter um one thing we could do is potentially set like agree and set some values on the call next thursday um and then don't have client releases like kind of schedule client releases for the the thursday after like the basically the awkward devs day and that'll give people like around 24-ish hours like to watch to observe if there's like a massive hash rate drop and if we want to lower this uh this ttd value um and then if we did arguably you could like lower it on all

core devs like you know agree to you know if the hash rates come down by 10 just like run the numbers with the new hash rate and you know maybe clients if if they're just literally waiting on that number to make a release um can can release quicker um so that might be an option as well you just sell it you assume that like the dag size is like a non-issue and um and then if it if it is an issue like you kind of wait until it's exceeded before you actually put out the release and then if it is an issue um we we updated on the

awkward ads um then

Ben Edgington: i think there's value in never mind the ttd i mean that's kind of background stuff we could we can argue about but i

think there's value in setting um a target date expectation um which i think will motivate users of ethereum and infrastructure people in exchanges and whoever to plan uh and get everything ready allocate engineering resources and so forth so if we um irrespective of the actual number of the ttd say you know we will do the merge on x of september or plus or minus you know variance but uh that's our target uh that that might help

Tim Beiko: yeah and i think one one way i agree with that i think one way to do that quite easily is on the cl call you set the bellatrix epoch right because we can set that to a specific time stamp and

any significant like basically any validator and node operator should upgrade before bellatrix hit regardless of when ttd happens so i think for sure on the cl call we could set belatrex and that's like the date by which infrastructure providers and users should have upgraded anyway and then the time between that and ttd you're just kind of voiding around because we don't want dtd to hit before belatrix um but i think we can probably set that on the next we can probably set the the electrics epoch height on the cl call does that make

sense if governance

Ben Edgington: if governance allows then uh that's good

with me

yeah oh i'll let you chat with me

Danny Ryan: yeah i said i think that makes sense um assuming goes well and we feel the same way about dates as we do today

Tim Beiko: yeah yeah and obviously yeah would there's not all the cl teams on this call so like i would you know el team seem like pretty on board and like the ceo folks we have here as well but um want to make sure that all the el teams like don't have something that's holding them up or what not but okay does that seem reasonable that everyone any objections

Adrian Sutton: so to clarify that means that

next week we are assuming goalie has actually merged we are making a go no decision and setting a bellatrix epoch

Tim Beiko: correct and i would also set a ttd yeah and like call it an optimistic ttd um and then you hold the client releases until we've actually exceeded like the five gig dag size if we see a massive spike in hash rates downwards because of that we reassess on awkward devs if not we assume those releases are good to go and um that ttd is basically the mainnet ttd and we'll make sure to like share kind of some of the the calculations or like estimations that we've been doing

on ttd publicly before the cl call so people have a couple days to review

them okay

Adrian Sutton: sounds good

Tim Beiko: sweet anything else on the merge that people wanted to discuss okay yeah that was that was a lot for an hour um okay in that case uh there's one more uh thing we we uh had on the agenda um so just to share some quick background um there's a pretty big difference between how the consensus and execution

layer handle specifications for ethereum today um so uh we have eips and the yellow paper on the execution layer and there's a python spec on the consensus layer um and with the merge happening uh it'd be nice to like potentially have a process that's a bit more uh similar on both sides um and so sam wilson and a few others have been working on an execution spec for the execution layer as well um and it's getting kind of ready there's like some questions about if and how we'd want to use this as part of the network upgrade process um yeah sam do you want to take a few minutes and kind of walk us through

where things are at and like what are the open questions uh right now

Sam Wilson: sure so uh we've been building python specification if you haven't seen it go look at it it's pretty cool um we're up to berlin we're doing a little bit of refactoring and then moving on to london and we're hoping to have a maintenance parity in time for shanghai uh and a couple of us you know most people working on the specs and some of the eip editors want to make the python specification part of the official eip process for chord changes and we wanted to bring that up here to see how people feel about it and whether

or not you know that's you know in line with what core devs want or if it's not um tim and i have a proposal on how we want to structure that change it's linked to the agenda if you want to take a look at it uh that's basically where we're at right now

so what you can ask again so the question here is should this uh executable specification be a part of the e-process in that you'd have to [Music] change

to the execution executable spec in the eifp so you

Sam Wilson: your audio is really bad so

we're reading it

Felix Lange: kind of different in the agenda where it was more about like should this be replaced

Tim Beiko: okay so i think what felix is asking is would this replace complement the eip process so maybe sam do you want to take a couple minutes to talk about like how how that could work like what would the process be like basically if you know if you had your way like what was like the ideal process

look like yeah

Sam Wilson: sure so there are a

couple different options um the the biggest open question about that is where to put core eips and their associated code changes but uh putting that aside for a sec i guess my ideal vision would be to have core eips um contain it like the motivation the abstract everything except the specification section in a marked uh markdown document or or similar inside the execution specs repository and then alongside each uh one of those documents you would have in the same git commit or same branch you'd have modifications to the code itself so you you would still write like a human documentation describing why you want to

make a change and then you'd have like a a commit that describes the technical portion of your change

i'm not sure if we still have felix yes he is in the chat yeah

Sam Wilson: so there are obviously some downsides to this approach and there are obviously some upsides so the biggest upside is that there have has been a lot of ambiguity in eips that have led to incompatibilities between clients and if your client isn't gpl you can't exactly look at geth and figure out what

they're doing so um the execution specs is kind of trying to be like a neutral ground for implementing these kind of changes unambiguously and uh it because it's an executable spec you can also use it to fill the tests so uh you don't have to necessarily use geth or any particular client to fill it you can fill it with a patch that you're writing for the eip

Tim Beiko: um yeah trent has a question in the chat about integrating the elcl process um i guess you know the it's worth doing the cl

process is basically the opposite right now so the cl process is prs to the spec without this like markdown uh you know eip i guess english portion that we have on the execution side um one thing that would be nice is like adding that to the cl process but that they can be kind of different processes but having something that's like that's where somebody who wants to write a change that either goes across both or a change in each doesn't feel like it's two completely different processes even though you know there might be quirks to one of them like they don't need to be 100 uh aligned

and i'm sure

Danny Ryan: there's an edge case here but i don't immediately see a reason to like integrate the build and integrate the software of these two specifications and instead if we wanted instead you kind of release them as packages and allow the consensus layer all those functions if you wanted to run them in tandem but i don't think that although i think uh the consensus layer respects how they're formatted and how we handle branches and handle new features and that kind of stuff i think we can converge on something i still think i'd keep them as separate repositories

yeah

Sam Wilson: yeah that makes sense

i mean we could probably you know migrate to using the same tooling for both um that way you don't have to learn two different sets of tools but keeping them separate or together it doesn't make much of a difference

Danny Ryan: and you know i will say very biased if you've been working on execution uh executable specs for quite a while i do find it really really nice that when i'm specifying things i'm also writing tests and when i'm writing those tests they also generate tests for client teams to run and it's just the a being so tightly coupled to writing of specifications has been i think really valuable to our process um but you know

it's a different process

Sam Wilson: so greg is here and he's definitely one of the biggest opponents to this change uh so i don't want to speak for him but some of the big things he's brought up are do you want to talk right because you're more than welcome

too um opponent it's

Greg Colvin: not really the right word um i'm very much in favor of there being an execution spec um the cl process is what it is um and they've been running basically running pretty quickly with with this single client that they're working on and the el the el layer has been working differently for a long time

a lot of different clients proposals come in at different levels of doneness uh from people working on different clients in different languages so i don't see that we can pull all of that together right away but i think there should be an execution client and a team that maintains it and that team should do what they can to keep it aligned to the eip process so you can look at the execution client and say yes these are the changes that were put in to support that eip uh work with the eip authors but it's

not until the eip goes final that you really could have those things aligned the the el later doesn't work such that everyone making a proposal is going to know how to make changes to the python client it just doesn't work that way

Sam Wilson: so is python familiarity a problem for a lot of core devs

it

Greg Colvin: it just isn't like for me it's not worth it the parts of the vm i care about i know the yellow paper very well and i can reason directly from the yellow paper for what i'm doing

and learning how to how to put things into the python client it's just a whole another level of work that's not worth it to me if i'm going to implement it i'm going to do it in edm 1 or in gaff so it's actually used we'll just somebody on mainnet and so i have numbers for performance that means something to me

um [Music]

Tim Beiko: andrew has his hand up

Andrew Ashikhmin: uh yes so about the yellow paper um because i've been updating it uh for uh

well for a number of releases but i'm running out of steam and i'm i'm not sure whether i'll be able to update it to london and probably not uh not for the murder the merge to paris so i'm just thinking that yellow paper unless somebody uh picks uh that mentor that yellow paper will become obsolete

Felix Lange: can i ask a quick thing so i i'm sorry for the audio it's better now the issue maybe the the [Music] question would be what area of the spec is covered by the

executable spec for example the evm it has been very stable and uh the most complicated thing is things like the gas rules and like exact pricing of codes depending on state and things like that these things are very tricky and i think it would be nice to have an executable spec for that but then there's a question like how much like is that is that all is that what it's going to be about or like where does the responsibility of the executable spec end

Sam Wilson: so i'm not like currently and i'm not talking like theoretically but currently the executable spec you give it a block and it will tell you if it's valid or not so it does

you know all of the the gas rules all of the the uh hashimoto stuff it does everything along those lines and then it doesn't handle reorgs it it expects to get the canonical chain um one block at a time

and doesn't do not working or anything

Sam Wilson: no

yeah

Felix Lange: i don't think it would really be possible with the networking but the question would be more like so this is basically the like the state transition or

Sam Wilson: yeah exactly

the stage

Felix Lange: transition and then also yeah well i mean this is this is pretty good like this is the the part that that i think can realistically be specified uh in this way anything that goes beyond that like for example i know like most of these things are now

anyway taken care of by the cl so yeah i guess it's pretty good

Tim Beiko: right and you can imagine stuff like sync obviously like snapsync or whatnot is not going to be specified there right

no yeah it's not it's not a fit i think for the executable spec format yeah yeah

Tim Beiko: and even like json rpc and even if you look on the cl side um you know they have the they have a separate network spec they have the beacon apis as well so it's like yeah it's really about this score state transition

Felix Lange: yeah so i'm i'm i'm in pa i'm in favor

of this i'm in favor of like trying this like if we can get is what's what's missing is are there resources missing to to get this spec find like the executable spec finalized or is it

uh on track

Sam Wilson: we're on track i think um i think it's mostly just figuring out if we want to use it as part of the governance or not and i think that's the the big open question

Tim Beiko: i think on that front like one idea that had been floated before is that for shanghai we can maybe run both processes in parallel so we get like a call it like a test drive of the execution spec um and and like it's and it might be you know it might look

something like people have eips already and like we just like copy them there and like you know kind of mimic what an eip in the executable spec would look like um and see you know how we like that um yeah um and then okay so yeah trent has a question about eips for cl and no eips for el so this is i think the if you accept that like we do want to move to something like an executable spec um then the biggest question is like do we keep using core eips like in the eip repo and simply link the executable spec from them so you could

imagine like an eip as it exists today but the specification section is just replaced by a link and you could imagine doing this on the cl as well so like we keep corey ips and they are where you describe the rationale motivation and whatnot but when you look at the specification section it's literally just a link out um and this has the benefit that um basically you could you know the eip process is well known people kind of know where to go to find the ips um and also what's neat is you could specify a single eip for a change to the el ncl so you could imagine something like eip4844 um the actual eip

has um all the all the rationale and whatnot and then for the spec there's a link to the el and the link to the cl specs which implements the different parts of 484 um so that's quite nice the downsides i guess with the eip process is one you then need to sync up like these different repos together um so you need you kind of have this uh this uh change in the eip repo that can affect kind of the change in the in this both specs repo and vice versa um and then there's also the concerns around like just you know the eip process being very high friction uh which uh adrian mentioned earlier in the chat um yeah and so i

yeah i think that is like probably the biggest question if if we move for micah if we try this to win how we do it um and the dark in the agenda has this section kind of with the with the different pros and cons about that

Felix Lange: yeah felix yeah sorry i must speak one more time yeah and uh for me i don't really understand uh the the difference between again why why you're putting these two ideas the eip process and this new executable spec as sort of alternatives when it is isn't it more like that the executable spec gives us

just another tool to describe changes sure and i feel like that that that should be its role for now we could keep everything exactly the same as it is now but just have this additional tool to be able to express uh changes to the specification and then we can see if it's so much fun using it that we feel like the the e process is not needed anymore or it's becomes less important to to to you know

yeah

Tim Beiko: so does when you say you keep it the same does that mean you have an eip and then you simply link out the executable spec for the implementation details is that what you mean

yeah

Felix Lange: i would rather just include

the relevant parts in the erp so one nice thing about erps in general is that they are standalone documents so you can always judge the eip on its own when you just read the thing you don't necessarily have to go to another place and that has always been a goal with eips i feel that you'd be able to i don't know print this document and hang it on your wall and you'd be able to look on that in five years and be like oh yeah that was this idea and it's like fully fully self-contained in this document the description of the idea and that's something that you lose when you when you link to to another repository where you have some diff

in some code

Tim Beiko: right so i'm not sure what i understand what you're proposing

Felix Lange: uh i'm not proposing something i'm just i'm just trying to yeah i'm just i'm just basically interested in exploring the the the the options here because you guys have been portraying it as like you draw that like basically i feel like this whole executable spec is mixed up in this whole other debate of what should the governance process be like yeah fair enough and i feel like these are just separate things

Micah Zoltu: so i think the the argument is that if you have the eip

process and the executive executables process we now have even more work to get a change through and the eip process is already frustrating enough for most people and having to do the vip process plus something just makes it worse

um i see yes sam has his hand

Sam Wilson: i don't mind waiting i have a different topic to talk about

okay i'm sorry sophia like sorry i cut you off no i would just made an acknowledging noise um then crowd was your comment about this specifically at all [Music]

Dankrad Feist: yeah i guess like i would say that the current um

process doesn't really do a good job of actually specifying exactly the changes though like um it's like very cumbersome you always bring in this like reference to bring in the past so i would actually like why can't that be a div i agree that the eip change log should be there but it should be more a diff of the executable spec rather than this ad hoc re-specification of what is happening because it's nowhere properly specified uh then saying what changes you want to make it and all of these like little scripts that are in there never being properly checked and yeah being all kind of incompatible and everything like that seems terrible at the moment

Felix Lange: well that's the good thing with the

executable spec you can you can have a tool then to have these these snippets that you can actually check i agree and i feel like we kind of voted in the executable

spec no sam

Sam Wilson: sure so um yeah i think the problem with doing diffs in the current eip process is when you get into um merging things together to make a hard fork so each eip would have its own diff section and then like let's say we actually use like a patch file then you would end up running into problems when you try to put them all together in the end if there's any

kind of conflict um and if eips depend on one another you'd have to apply them the right order and i think that just gets really out of hand um so in the the proposal that is is linked in the chat uh we we kind of have a uh a branching structure so we use git to handle merging all of those things and but that would be kind of a large change from the eip process

Tim Beiko: so i guess just like from the general comments it seems like everyone is kind of interested in seeing how this could work um obviously there's a bunch of like kinks to figure out

um but it probably makes sense oh sorry andrew

Andrew Ashikhmin: um oh i just wanted to say that maybe um if somebody creates an eep and also and on top of that creates uh um pull requests to the execution spec then it might be considered as a bonus as an an extra argument in favor of that eep so like that person has done the extra work so we should maybe like yeah like err on on the side of accepting that deep and that would be a nice motivation

Tim Beiko: that could also backfire so yeah yeah uh yeah i think

i think it probably makes sense to like at least try to run a parallel process in shanghai you know and see what this would look like um in the executable specs um and like also for shanghai most of the changes we do we've already either like considered for inclusion or that are still kind of pending are already sort of specified as eips so like you know realistically we're not gonna ask people to like throw out all that work and then move to something new but if in parallel we can like make a copy of that eip as a potential change to the to the executable spec i think that would be good um yeah does that make sense to people

and i guess the other thing in this is maybe more something for the cl call is like if we do want to think about how this process could eventually be like a bit more in line is like how how do we have something resembling the like english portion of an eip on the cl side and whether that's like some change in the cl specs repo or something that's i don't know trialled in parallel um i think that could also be interesting to test for like the capella portion of of the upgrade

um yeah so

Sam Wilson: before we before we move on i just want to bring up a few of the other concerns that people have had about the

execution specs so one of the big ones is that you have to pick a particular algorithm when you're implementing something for the execution specs you can't leave it specified in like math notation or or as a description of the outcomes and that might influence client developers so that's kind of a negative uh some things are much easier to to specify in english than they are to specify in code uh so you might lose some of the clarity you would get by writing it in english um yeah and i think there's the big ones greg if i missed anything else let me know but i think those were the two other ones that we haven't talked about yet

Greg Colvin: yeah that is a big one i've got one thing that four sentences of english makes it quite

clear but it's a whole page of code to actually validate it and if you read the code it'd be very difficult to extract the four sentences

um yeah

Felix Lange: but we can can't we just accept the way it's basically always worked so that like we have eips as this platform where you can publish your ideas and uh i don't know get feedback on them or whatever and then eventually once once the idea is like reasonably accepted we like put it to the spec and then like the spec can be executable that's really nice but it has i feel like very little to do with i know

Greg Colvin: no i'm with you exactly i think that's

exactly you know right now the executable spec is what it is and it runs in parallel and at about the same time that everything goes final the executable spec is ready um like you say it's another tool for all the clients to communicate and get on the same page

um yes

Felix Lange: it's just a way to form like really like precisely express what you mean when you're talking about consensus changes and i feel like that that should be all

the stars yeah

Tim Beiko: yeah so yeah but just to be clear so that means you could imagine an eip which the eip itself does not contain the actual specification right because

it just links

out yeah you've read

Felix Lange: that after once you've yeah right yeah you can't put it or not but it shouldn't really be like a it's yeah but it's just like it's it's it's not if you can if you can express it in this in the spec language like if it's a change that like i feel like many changes can be expressed in this back language if we have a working spec language where we can be like oh yeah i want to make this tweak to the gas cost and then you just write out the the the the new formula and you can have a bunch of text that says why you want to make this change and it's going to be all together but then once everyone kind of feels like this is good you can apply that to the actual spec and then yeah it's

going to be in there i feel

and then

Sam Wilson: i think the problem with that is that every change eventually needs to make it into the specs so that you can build on top of it so why not make the spec part of the process

right

Danny Ryan: you can have back implementation dvd and if you actually go and implement the pr it helps eliminate what's going on it helps show that it builds properly but maybe some it's delayed some of it happens quickly depending on kind of the style of the process of because i guess if it's a complex change and you want to get the english language out there you probably want to be able to say stub execution so i can just like get

initial feedback i guess

Tim Beiko: yeah and i think the question is also what's the canonical so imagine you have like a gas cost like a complex gas schedule change right then you you sort of or like a complex algorithm you know like greg was saying you specify something like in in more of a math notation in the eip you link out to the executable specs for the actual implementation if they disagree with each other in some weird edge case you know which one is treated as canonical

Felix Lange: so i feel in this case the execution like this the spec should be canonical okay and also for somebody this this also solves something that that we have right now where if you have

multiple eips applying to the same kind of thing then they sort of override each other or start interacting with each other and this is something that you can't really specify in the eep or in a related scenario what if you create the eep and everything seems seems fine and then much later during like hard fork testing we run into an issue and we find that the spec needs to be changed in this case we'd have to now right now we'd have to go back and like change the eep but i feel like in this new world where we actually have the spec we wouldn't necessarily change the eap anymore we just keep the eep as a sort of historical document that started the idea but then just like make fix it up in the specs so the edge case is

removed or something yeah i it just creates a bunch of new possibilities that we haven't really had yet just having the spec like i feel that's that's like the milestone we should aim for right now is just having the spec in the first place because we don't have that right now once we have that we can see like oh how we yeah like we can just check how we integrate that once the spec actually exists and works and

[Music] yeah

Tim Beiko: that makes sense and i think yeah it's like yeah all these things we need to agree to if we before with this but i think okay and we're kind of wrapping up uh in terms of time um but i think people

agree that this is like a really interesting experiment it's worth continuing um the link in the agenda uh has a link to an eighth magician's thread if people want to comment about like the the specifics of of how this happened um but i think yeah it's pretty clear that people want to see this play out at least in parallel with shanghai um so i think sam that's that's a reasonable next step does that make sense okay um last thing i guess just as we wrap up uh there is a merge community call next friday uh at 1400 utc so friday august 12th um if client uh developers want to show up that's always useful um and there's a few flashbacks

people on the call as well i suspect if a couple of you want to show up as well and some people might have questions about mev boost um and it would be good to have like experts to answer them um again this is linked in the agenda at the bottom um but yeah next friday 14 utc and right before that we'll have the cl call as we discussed uh to talk about the bellatrix epoch and a potential mainnet ttd um yeah thanks everyone this was pretty good

thanks so bye thank you thanks everyone thank you [Music]

[Music] so [Music] so [Music] so

[Music] [Applause] [Music] [Music] so

[Music] [Music] [Music] [Music]

[Music] [Music] [Music] [Music] you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.