All Core Devs - Consensus #93

Transcript

[Music] [Music] [Music] [Music]

[Music] [Applause] [Music] so

[Music] [Music] [Music] [Music]

[Music] [Music] [Music] [Music]

so [Music] so [Music]

[Music] [Music] hello welcome kansas lair

Danny Ryan: call 93. issue 594 on the pm repo and yes now we have a 6x6 grid this is very aesthetically pleasing um okay so today we are going to go over

merge related items including the recent gorilla merge a number of um minor discussion points and um there are some documents circulating for mainnet bellatrix epoch and tdd which we had pre-committed last week to talk about this week so we will do that and then a couple of issues related to mev boost maybe also a general update and then there are some discussion points for 4844 that came out of a recent community call if we don't get to it we will certainly get to it at the next call we will see how time plays okay let us get started

way to

Danny Ryan: merge

uh it happened approximately 12 hours ago the network currently looks healthy although there was much in the interim perry would you give us the high level and then maybe we can go as you're as you're giving the high level if there are additional points on any of the points from a particular client team please client team jump in at that point so we can discuss those points um just so we can kind of all get on the same page and understand how to move forward

perry

Parithosh Jayanthi: yeah hey everyone uh so like danny said we had the merge on the goli network about 12 hours ago one of the things that was peculiar about the merge this time was that we

had two terminal blocks so essentially one block that was produced with the difficulty of one and another one with a difficulty of two and some amount of chaos is attributed to that um essentially we did have some so we were looking at about 90 participation before the merge and then we dropped to roughly 70 and at for a couple of epochs below 7 below the required 66 so i think we didn't finalize for 405 epochs um essentially a couple of nodes had gone into their own forks or were offline for another reason i have elaborated those under in the in the issue tracker linked in

chat um yeah besides that one of the biggest things that helped with participation over the last 12 hours is that there were a couple of nimbus nodes as well as load star nodes that had wrong conflicts either they were unupdated um execution layer nodes or they were they weren't configured with the required jwt token once that was fixed we noticed participation rate increase up to the present 81 to 84 percent that means if we were to remove config issues it would have potentially been a 90 to 84 drop which is actually not that bad at all um

yeah and besides that we have uh two or three client issues that we've noticed so the first one being a netherland issue shortly post ttd and that had to do with nethermine's processing of pre-emerge and post-merge blocks differently essentially they they had a bug where if there were multiple terminal blocks it was hard to hard to fix the fix was known we had the fix deployed on a couple of nodes as as a test and i think it will be included in the next release and one thing to be noted was that we already do have a test for this on hive tests so if the hive tests are all passing then we wouldn't have

noticed this bug we had an eragon stall before itself healed and essentially aragon had built a transition block on top of a side four and once it got stuck on that side fork and once 128 slots were done it kicked into optimistic sync and fixed itself we had a issue with a rocket pool node using an invalid block and the node was lighthouse aircon if i'm not wrong and we're still investigating that one but potentially updating to the latest development fixes that i'd let the client team talk more about that

and there was an issue with bezu incorrectly invalidate incorrectly replying to the canonical chain to tekku but i would let the bazoo team elaborate on that one as well if you look at the issue tracker there's also a status for most of the issues that we've mentioned and yeah i guess that would be updated over the span of the next day yeah i think that's about it for me

Danny Ryan: thank you and um the nimbus and lodestar nodes are online as well are online now those are the nimbus team and lotstar team nodes

Parithosh Jayanthi: yes the nimbus team and lots

and we knew it wasn't necessarily a client incompatibility because the ef was also running those clan combinations and we didn't notice any issues on

unknowns

Danny Ryan: got it okay um are there other would any client team like to go deeper into some of the issues that they solve uh you know the steps that they're taking being taken to resolve them if there's additional tests that need to be brought in here just any other comments

Andrew Ashikhmin: yeah i'd like to talk about the issues in aragon so we did have a bug in the in our previous version with the

transaction indexing but uh i'm still not sure what happened uh with the rocketpool because um the person the operator says that he actually uses our latest version and we saw in the log that an old transaction made it into our transaction pool so still not sure what exactly happened there and uh whether this uh bad in embedded blog was due to that bug or something else so still investigating and i would also i'd like to ask about this um 128 block maybe mikhail or somebody could explain

uh ruthie to me what what why this needs to wait 128

slots

Mikhail Kalinin: um shortly this is like the protection from the fork choice poisoning that can occur if you're uh trying to import the block with the parent that that is not available like uh yeah the the merge transition block built on top of uh a parent that is not available or just randomly generated parent hash in this case it will the syncing response will never resolve and cl will just if cl switches to such a block it will get stuck there

and if like uh if like a lot of cl nodes and where their nodes get stuck and syncing they will not be able to do their well there is duties build the honest chain this is why we're waiting for 188 blocks uh before optimistically switching to um to this block because if this should give enough time for honest chain to progress

Danny Ryan: and yeah and no these are for actual unavailable chains um i think the aragon issue is that it was not executing um the these side chains that were available

and if the side chains that are available during the transition are executed then this issue is would not happen um it's primarily to avoid the case of someone's fork choice being poisoned for with unavailable blocks if the blocks are available and they are executed um then this doesn't happen so i believe that there was a pr up in aragon to do this partially but if it had any sort of um parent depth into the proof of work chain it was not actually um executing and was returning either accepted or syncing and thus getting into this deadlocked state for 128 blocks

Andrew Ashikhmin: okay i need to take a look at

how is this take a look more into the logic around the terminal pow block

Danny Ryan: right i guess it's it's after the merge transition you know if you don't execute side chains immediately and just return accepted even if they're available that can be handled and is fine it's worth the additional load during that transition to actually execute these to not enter into this status

Andrew Ashikhmin: right i see

Mikhail Kalinin: yeah and this is only for the um transition block once it's imported this timeout is not applicable anymore

merrick

Marek Moraczyński: i want to rise one more thing because we notice it on our internal node another might tackle and we started discussing it with adrian and the issue is that we are returning that we are in sync uh as nevermind but our logic is that we are downloading the history um we are trying to reach the head and we are returning that we we are in sync but we are still downloading the uh history of the chain so all the blocks alt receipts and it seems like tech who

is shut down because of that and we are wondering with other and what we can do because the solution could be to change if syncing logic or or adjust cl client

that's a question for adrian from tecu

Marius van der Wijden: so what's what's the what's the issue why why why is this breaking stuff it's is it because tikku is asking some old blocks because they say okay

yes yes okay okay yes and i'm i'm wondering marius you are

returning sinking only if you have all history or [Music]

Paul Hauner: uh with lighthouse we were tracking using is syncing on the els to try and um help us understand when the el is thinking but we just found that um it wasn't reliable enough for us we found that there's a few funky occasions where geth will tell us that it's um i guess it isn't thinking i can't remember what it was so we just decided to um to give

up on it and we don't we don't track that anymore

maurice

Marius van der Wijden: yeah the issue is we turn we return not not thinking if we didn't start the sink yet i think and that's like some some corner cases you kind of cannot really rely on on this thinking because um i think so i think the more correct way to say is if you don't have all the history and the state then you are still thinking and you shouldn't return sync

Łukasz Rozmej: yeah so that's a very good question

about the semantics because what we are doing is if thinking is a question as if we have the head we can we can process the next block so we return sinking through when we don't yet cut up to the head when we have some head that we can rely on and we we know that the network isn't further then we return sinking holes while we still can download ancient blocks and asian receipts so that's the current nether mind and potentially we might want to change that based on this discussion

Adrian Sutton: um so take who needs this historic blocks

for other historic receipts for for the for the deposit contract right yeah in this this particular issue that we're hitting is slightly different to just the deposit receipts um i suspect the next thing we'd hit is that we'd be searching block ranges right for log events and i think we get a json rpc error back if nethermine doesn't have blocks for the range we asked for um so i think we can handle that and we'll just kind of keep retrying that but the the code that's causing problems is our search for the minimum genesis block the first block that meets the conditions

um it's probably not but it's not required now that we have main genesis so we can pull it out um it's just really complex and we haven't disabled the ability to run and actually calculate genesis from the execution layer yet

uh which which genesis that that's beacon chain genesis

Adrian Sutton: yeah beacon chain genesis and it's it's the first block that could have triggered that if there were enough deposits which on main net i think was the block that triggered it um yeah

part of it i think is that a while back like way back we actually added code to handle i think probably never mind that that didn't have all the blocks back to genesis and so as we were kind of doing a binary search to find this block we'd get blocks that just didn't exist in history so we assume well it must be after that because you didn't bother syncing those old blocks um but when you are still in the process of thinking that's now leading us astray because we're trying to find a block that's way too recent and then it's failing the check when we try and confirm it really was the right block so there's a few tweaks we can do it

sounds like it's only affecting tecu as well so other clients have either skipped that logic or done it in a slightly different way or for whatever reason we will be able to work around it um it is probably oh no it'll be all right if if the requests for deposit logs are ejected we'd then fall back to a different node anyway all that post merge it doesn't matter as much anyway it's a really interesting case because saying you're in sync and we technically uses that we kind of expect that you've

got that history um and i suspect there'll be some other surprises in different cases from people getting block history in that case

Danny Ryan: yeah i do think the semantics here are much more about i have the state with respect to the head and can process blocks built upon it and that extending the assumption beyond that's probably dangerous because everyone can have all sorts of different like historic modes and stuff and or even pruning obviously there's not a lot of like most blocks are kept um but even 444 could like change the pruning depth and stuff like that so like the assumptions there shouldn't shouldn't be about the depth should be only be about the head i think

Adrian Sutton: yeah i i think there needs to be some indication there though that you know how far back do you have um because we're kind of changing some of these semantics now

Danny Ryan: it's yeah i mean i but i would i don't know if i'd couple these head semantics with depth semantics you know we're using the eth apis are much more reliable with respect to that um there might be rules about how deep you should have um but i don't think that like sinking verse valid in the head should in the engine api should dictate those depths to as logic premises

Adrian Sutton: yeah so the the problem i think then we're getting is just that we're we're asking for a block we can't

tell i don't have this yet versus i'm never going to get this and so we're handling it as in we're never going to get this because that's what it used to be so the sinking is kind of it would patch over it if you were still saying sinking because we would just treat the node is offline but we might be the other option here might be that if you are intending to download a box but don't have it yet return a json rpc error instead of empty because then i know hey you're just not ready for that that would also fix technically i i suspect we'll change teco as a result of this um it's just i i can certainly imagine a bunch of other applications having surprises you know someone like infuria would see

the node is syncing they wouldn't go and check all the odd blocks so there's got to be some indication to say actually this node is not ready for service yet um otherwise it kind of gets thrown into the json rpg rbc pool and does all kinds of weird things

Danny Ryan: right and i i agree with that i just i think the conflating of the engine api semantics is not going to give us what we want here um do we have enough information to move forward and can move on to the next issue any other points on this one

Mikhail Kalinin: yeah i just wanted to point out that it's very important for uh from cl perspective when el validates and processes the emerge

transition block if the terminal block is has been received previously by this el so all data that are required to execute this block are available otherwise if it's returning syncing or accepted the uh cl will stall for 128 blocks also uh there was like from what i understand there was a bug in the test that checks this particular scenario and we are going to fix it and this is what is important to be fixed in your clients before the mainnet merge if it's not already implemented as

expected

lucas

Łukasz Rozmej: so i would like to ask other execution uh engine clients to how they are implementing the cth syncing in terms of that can be offline sorry if needed but i'm really interested if we should change it or not

Andrew Ashikhmin: uh in aragon we say that to like uh if

we have anything missing we re reply with uh thinking we like we don't download anything in background if we do that

requires the thinking

Łukasz Rozmej: yeah because you don't have the state sync right you have to have all the blocks to execute them right i think that's fairly obvious for your for your syncing

method also basu uh downloads everything before in the backward sync before saying that in sync i believe it's the same for death

um and also because the like history is way easier to think than state at the moment you should already have the history when you when you finished uh no our snapseed can take like three hours while downloading all the receipts can take a long time

Marek Moraczyński: okay and maybe we should work on our snip yeah we are always thinking state faster than all history

Łukasz Rozmej: so

[Music] it's way faster maybe we should work on our

history but it's way faster

Danny Ryan: okay i think we should move on to the next um discussion point and if there's additional discussion here let's take it to discord okay um other discussion points from perry's

summary that client teams would like to go into um you know discussing an issue that arose that is going to be patched maybe some additional test factors that should be put in anything else on the gourley

discussion uh paul here

Paul Hauner: i just wanted to say that i had a good dig through um fork choice just before we finalized uh the merge transition there were a few forks floating around i found nothing none of them to be particularly interesting so yeah i didn't didn't really find anything to be concerned about regarding consensus say a fork choice just looked like a kind of bit of an unhappy network trying to sort itself out and it did

thanks paul other girly discussion points

Marius van der Wijden: so one thing i i saw was that some of

our notes uh lost a lot of peers uh like most of them states didn't lose any or stayed mostly the same and then a couple of notes that went off and probably went up on the wrong foot i think it was never mankind combinations um they almost lost all of their peers on the execution layer or consensus there or both i think i think it was on the on the consensus layer so maybe the peer scoring is a bit aggressive yes

perry

Danny Ryan: some of the nether mind was

restarted so they had to reap here um i'm just gonna parry said that saying that out loud

neither my execution had a problem right yes yeah which we've

Marek Moraczyński: we've discussed but um if the mario's if the the node restarts was actually in line with also the pure drops would be good to know

Marius van der Wijden: um i think at least from the the time that i looked at it it was more a gradual decline

okay

Parithosh Jayanthi: yeah i'm just looking at the graph now i think they went from 50 peers to about

30 and then

back up after a while

Marius van der Wijden: so when i looked at it it was more like 50 to like five and that that's why i was like that's why i'm a bit not really concerned but i think the scoring might be a bit too aggressive right but not really yeah you know it's it's good

no

Danny Ryan: and that i mean if the pure scoring is aggressive that could also account why some of these forks were three four blocks instead of just one and quickly resolved if there's like kind of um

transient network partitioning going on there um let's on the consensus layer side can investigate that a bit okay other girly items

mikhail tbh

Mikhail Kalinin: yeah does any el client team want to share their progress on the terminal block hash

override implementation sorry go first

Andrew Ashikhmin: uh yeah we still don't have it um it's we we only have the the the override on the networking layer but that will make us only appear with peers that have this specific plot and so we should i'm not i'm not 100 sure how it works but like if you specify this then you are guaranteed

Danny Ryan: right i'd i mean at this point i think having a credible path to do this in an emergency is more

important than having it top down fully implemented um it's my my take andrew you had a comment

Andrew Ashikhmin: uh we haven't implemented it either

thank you

Marek Moraczyński: for us it's still in pr because uh we want to focus on today's issue and uh it's thinking and probably um we will review it later a bit because it could potentially introduce some regression so we need to be careful with

merging it right

Danny Ryan: i back to my previous comment i might not

even merge it i might have it there proof of concept and ready to use you know in an emergency not affecting the code base at this point okay thanks for the update mikhail anything else on that one

Mikhail Kalinin: um oh thanks everyone and agreed with the above comment from danny okay uh paul specking the placeholder ttd in engine api

Paul Hauner: yeah so um when when the ttd is not specified yet um we still want the el and cl to be able to

exchange transition config and do that successfully just so that users can set up their stuff before the merge and see that things work there is a standard for this on the consensus layer about the value that we use when none is defined it's a very high value there doesn't seem to be anything on the el side it seems that i think death returns a none and i've seen never mind return zero my proposal is that we take the standard that already applies to cls and also apply it to els so basically um you you copy and paste this very large number from um my pr and then put in your code base and if you don't know what the ttd is then you just return this high number um yeah i think it should be fairly easy to

implement i think we've already got a bit of buy-in from people um just wanted to raise it and see if there's any objections

Danny Ryan: so there's a bit of a race at this point paul depending on the next discussion point as to whether there will even be client releases between now and when we have a ttd um does that affect your desire to get this

merged

Paul Hauner: um yeah so it might be that we figure out the main entity before just before you know before people get around to implementing this i would i don't know i'd be tempted to just implement this um it's not a big deal if

you don't but it's kind of helpful because you know there's other people running other tests that's elsewhere like like noses and stuff that haven't decided there's yet and it just makes life a little bit easier um but yeah no no one's going to die or nothing bad is going to happen if you don't implement it but if you can spare the cycles i would i would try to do it just for cleanliness future future sanity

i

Marius van der Wijden: i think we already i already implemented something like this i'm not sure if it's merged or not right because the agreement

Danny Ryan: last time we brought this up was it wasn't necessarily going to be spec but people were going to do it i know that if something's not spec it's less likely

to be done um but let's i could go either way on the spec but um if people get this out that would be great other comments okay if this issue is relevant to you and you care please go to the um engine api execution apis pr that was linked and comment um if we're going to do this probably makes sense to complete it by maybe end of the day tomorrow so if you feel strongly about the other please chime in okay um on the last awkward dev call we

had discussed picking a mainnet bellatrix epoch today and a tentative ttd today to then potentially be overridden next week in the event that we see a precipitous drop due to the five gigabyte threshold on miners on the dag size and to do client releases shortly after can we trying to here yeah tim why don't you give an update on dag size uh just so we can understand the timing there

Tim Beiko: yeah so

um i don't have like a perfect understanding of this but i think it's it's good enough for our purposes as i understand it when uh when uh we say the dag exceeds five gigs next week it's actually the dag plus the size of the block hash um that exceeds it um so it's it's it's right under uh five gigs and then um which means that if you store the block cache elsewhere you can keep using uh you know those those miners and you can you can kind of fix that through a firmware update um even when the dag exceeds five gigs it seems like there's at least some ways where people can use uh five gig cards um and like

probabilistically uh get like a a valid bag if the only query kind of the subset that they have in ram um and the more i dig into this the more it seems like there's a a long tail of strategies of increasing complexity that you can do if you want to maintain that hardware um which leads me to think that like the odds we see an extremely high drop right when this hits are low um historically we haven't seen kind of really big drops when we've hit a threshold we've seen some drops but you know they they're like on the order of noise in hashrates um

so i think um you know we it's fine to pick a ttv and we can permit next week um but i and and it's probably safe to like you know wait until we hit that and see you know do the like uh what share of the hashrate did the network who just run like the vanilla firmware and do no optimizations represent and is there a falloff there i suspect like after that you know there's an increasingly complex amount of tricks you can do to keep maintaining that hardware and so um different actors uh will be able to do that um yeah so i i guess all this to say is like we should probably wait just to be like extra safe or at least probably

reconfirm next week just to be extra safe um but i don't think like even if next week is when we hit it or when we when the dag actually exceeds five gigs uh excluding the block hash um we'll see like a massive drop

Danny Ryan: thank you tim so potential information in about a week but actually unclear if we'd see like a discrete drop um if depending on how people are programming their hardware and how the hardware is actually constructed good to know um in such a timeline um we would be aiming to on probably tuesday the latest not this coming tuesday but the following to

do a blog post with client releases thus client releases would need to go out thursday friday saturday sunday or monday quick turnaround helps us make sure that we kind of keep the timeline rolling and some of these more exotic attacks around accelerating ttd we kind of have our eye on and kind of can keep things tight um tim do you should we go over bellatrix stock and the uh quarterly i'm sorry in the end the tcd dock or do we want to have other types of discussions around strategy first

Tim Beiko: i guess

you know under the el and cl side like are people comfortable with that timeline right like that's probably the main thing then we can it's like yeah is are people comfortable with those timelines and if so we can look at you know specific epochs and ttd and based on all that but it's yeah

uh hey

Terence Tsao: i just want to confirm so we're looking at the week of august 22nd is that right

Tim Beiko: correct

Terence Tsao: yeah so um so um this is prison so i spoke to the team and majority of them has no issue on releasing before august 22nd yeah and

that's from our side

terence

Danny Ryan: you do sound like a robot but we could understand you thank you yes

Tim Beiko: yeah i guess yeah i cares to hear from other teams as well or i guess maybe another way if people don't think this is possible this is kind of the time to step up

Andrew Ashikhmin: yeah i think for aragon we would like more time because uh we kind of we still need some work to do around terminal pow blocks potentially optimize uh unwinds rewinds especially with i think it's the case with taco but maybe with other

seals as well so from my point of view i would like more time

Danny Ryan: so i think one strategy in getting more time is to actually do the releases then but to also set the expectation that there might be el strongly suggested el update releases after bellatrix in which we'd have a follow-up blog post and kind of thus yells are feature complete and ready to go um if i did if you don't upgrade your node's probably fine but in the event that uh there are strongly suggested releases you do kind of another wave rather bellatrix to say hey these are the releases that client teams recommend

i think that allows us to have a little bit more flexibility and play in getting final releases out but also at the same time gets us positioned uh in the coming week and a half

onscar

Ansgar Dietrichs: yeah it was just on that phone i was just wondering um my understanding would be that for mainnet basically the minimum time between having new releases out and being confident that everyone upgraded to something like two two and a half weeks or something so wouldn't that mean that we basically have to have this minimum two week two and a half week period after politics before we do

the march

Danny Ryan: no what i'm suggesting is that client releases come out in that plus one half a week from now and their main net ready you know they

work they work as they did on gourley today if not better but in the event that there are additional releases we would do a wave after bellatrix to say hey these are the recommended releases upgrade if you can thus you don't necessarily have the lead the sufficient lead time for everyone upgrade but if people do want to patch with the hardened releases they can

matt

Ansgar Dietrichs: okay that's reasonable

Matt Nelson: yeah no i think the big act is cool with that timeline especially given that we could again put out another release my question is when do we anticipate the i know kyle mentioned some bugs with hive can we anticipate those sometimes soon just to make sure that if we have extra time to

like double check and button everything up before the release is on the 22nd would be ideal but if we need to push some of that back i'm presuming that the tech we understand the bugs and the tests and they'll be updated pretty shortly

mario maker l what's the status here

Mikhail Kalinin: from my understanding it should be like an easy fix but it's better to ask mario about it

sure if he's on the call

Danny Ryan: do you mean the uh the gossip testing or the recent wave of el tests i believe that found a few bugs in corner cases

Mario Vega: oh yes there were some for nethermine but uh i think they were already working on it the new release for this distances or remesh so um there's no not a substantial increase for for all the the other clients i think one test forget and that's it i think but there there was no

no substitution substation

Danny Ryan: and these are merged and in public hive and teams presumably have seen them and are working on them

Mario Vega: yeah yeah yeah it's it was much just today actually so the latest run should reflect the current status

of this

Mikhail Kalinin: so the um the test has been fixed right and already

if it was even buggy that was my impression um which specific one yeah the test when there is like the terminal block uh from from a site fork and uh client response thinking um he is some issue with this test another mind passed it but uh it didn't work on the

girdy

Mario Vega: yeah definitely that that one is is fixed i think that was the specifics since the last release so that should be working uh even now without the latest update

thank you mario lucas

Łukasz Rozmej: uh so

um i think we are passing most of those new tests i think i have two right now failing in my pipeline um but apar about those general tests and and the on this this transition test so i i gave mario feedback that it would be good that the blocks and hive tests that we are using for for those tests would have some unique stage changes each one of them because this is what what we have issue with that on hive the all the blocks have same state route but it would be good for all the blocks to have unique state routes on hive tests

like in all the tests probably

that's

Mario Vega: yes definitely definitely i we need i need to tweak um some of the uh the test cases to include more transactions in the merch in just the mesh merge block so we can have a different state route and yeah but that yeah that's uh that's that's work in progress

Danny Ryan: okay um other input on the suggested timelines on the suggested release strategy

Nishant Das: uh what's the expected date for

bellatrix that's what we would get into now

Danny Ryan: um actually adrian can you adrian and i believe tim was doing a couple of additional estimates this morning can you all talk us through and share the document

Adrian Sutton: yeah let me just get the document shape um so there's a few options uh i share your document to the shout outs i can share my screen if that's helpful we're gonna share yours yeah just trying to move things onto the right screen so i can not have it all too big i think he's got like eight screens over there yeah yeah that's the only way to monitor

you know right so uh this is working on the basis of essentially what we just talked about that these dates become in range um with the idea that we would have releases out set around august 19th with tweaking what kind of sounds like we want to have that weekend up our sleeves as well so we're probably looking at the later range for these dates um first september would be just less than two weeks away from the thing it now gets to more like week and a half um so i'd say we're kind of looking at the monday tuesday type time frame um i think the first thing to set is are

we happy with these kinds of dates like we can pick specific numbers easily and we can make them pretty and i can show you the slot picker and that kind of thing but i i'd be thinking either monday fifth or sixth about activating bellatrix based on these conversations

Danny Ryan: i believe someone commented on the fifth that that is a national holiday in the u.s and that even corporations might be off so pushing into the sixth i think does make sense as for consensus layered teams just in some

conversations over the past day that i believe that there's no objection to moving forward on that but please correct me if i'm wrong yeah i agreed one four five zero zero it's really easy to eyeball that you got that right instead of like having a copy paste error or something

so

Adrian Sutton: if we're going to choose six that's probably a good one it is very late for europe um it's 10 p.m it's not 4 00 a.m right

that's it's probably 11 or 12. oh because utc oh okay yeah i did fudge that one

Jacek Sieka: a little um is any of these epochs on a what is it 8192 slot boundary such that we get a uh such that we get an even multiple for the block roads table to be fresh for the merge those two are not um if somebody wants to find one they can i think let's see

what's sorry what's the benefit of that well we have these um block routes and state routes and his above all historical state historical routes tables uh lists in the beacon state and they rotate every 8192 blocks

Danny Ryan: so um then you can have like the rule set be the same for all batches the next one i think is one four seven four five six which i think is moderately substantially later

Tim Beiko: um if someone before that would be one for if you wish so i do think it is yeah either too early or we delay by another couple days just to hit that which would basically be two days right or uh more like five days because like between one four three and one four five you get about five days um and so between one four five and one for seven you'll get another five days um which seems yeah

Jacek Sieka: ah that sounds strange it's 27 hours per cycle oh slots

Danny Ryan: sorry i was doing it i did i did math which is wrong um someone oh someone said that should be yeah yeah yeah it's easy it's easy to bump um my apologies on that let's redo that

Tim Beiko: so is there is a for the beacon chain is a multiple of that better than a

neat like epoch boundary basically we did so on altair right okay

Danny Ryan: um i you know i i would i think i would rather keep with that for the potential use cases there for the cleanliness there than keeping a round slot

um i could be convinced otherwise but i i don't see a reason not to i might apologies on getting that math totally wrong but let's run a quick couple numbers on that and i guess sorry please go ahead

Jacek Sieka: so it's 256 epochs that we need to be

round to

Tim Beiko: okay so we can we can figure out which of these epochs we like the most and then find the closest uh multiple from there right

Jacek Sieka: yeah exactly and it's actually quite convenient when they line up i mean it's kind of like a

clean slate where um we get this unique number for the state that corresponds to the purely one rule side or the other orc yeah and not only that like but it actually ends up in all historical beacon states for for for the future because of the historical roots thing so we'll get like that particular hash will be in the beacon state forever

Mikhail Kalinin: does it make that much sense uh given that the tpd is unpredictable and we will not be able to so it's the bellatrix rule set

Danny Ryan: right but then you have a transition

during the bellatrix rule set but it's it's the same code whether ttd has been hit or not there's just conditionals on the code whether you're doing x or y so i i i see what you're saying but i think that from like a consensus layer perspective that's when the upgrade happened and that's when you're running the new logic even though the tcd isn't happening yet

ah seriously

Paul Hauner: yeah i would be um cautious of fulfilling that 8192 rule though if it's going to land the update at some really awkward time for a majority of users i would think that if we want this to go smoothly um doing it when users are online to update their notes is probably the most important thing um not necessarily you know making it so it's nice round number

10 or even fulfilling that 8192 goal but that's just nice to have both

but

Danny Ryan: can we sanity check the a192 just real quick then

yeah

Jacek Sieka: so i think we'd be looking at one four four eight nine six which is that time that i hope you can see and it's highlighted there um so it's still on sixth it's pretty reasonable for everyone except la

that's like plus

Danny Ryan: was that plus how much time from the previous so that's a little maintenance for the nice round number

oh

Jacek Sieka: okay okay so that was like minus 10 hours and and what does la's at 4 30

Danny Ryan: we've

definitely had forks at 4 30 on west coast because i know terrence has been on them pounding coffee

Adrian Sutton: it's 1am here now so i'm not entirely sympathetic

[Laughter]

Danny Ryan: yeah i hear you um i'm i don't think that that's like bad for users um i'm fine with saying that is a good tentative number and that we will kind of circulate it in a pr and put thumbs ups on it sounds good to me

Adrian Sutton: okay sounds good to me

Danny Ryan: great

moving on to um ttd which mario hobble has been working on a dock here on predictions mario

Mario Havel: ah thank you danny so yeah i'll be sure to don't quit thank you tim um so uh yeah thanks then for sharing the docs so should i share my screen as well yes please let me know if you can see it yeah thank you good so um yeah so i i um i would like to present uh the proposed

ttd value for the merge um which um uh this dot explains the strategy behind choosing the number and i'd like to get feedback on that and present the values itself so uh just to give you a quick overview on how the number for the dtds picked first it's using polynomial regression uh um uh from past few weeks so basically data on difficulty in the network from past few weeks uh extrapolated in next two weeks to predict uh what value of total difficulty will be reached at given time it's uh done using the prediction tool which you might uh might

noticed if you followed both wtf it's pushing the it was it was uh uh used for predicting the test that merges before so uh with this uh we establish what is uh what is uh roughly the the difficulty the total difficulty to expect at uh the given date but the problem is that uh the accuracy of this uh is purely purely depends on how the hash rate will evolve on the volatility of hash rate so uh we have to understand the uh what hash rate levels are we working with looking at the current uh trends in the hash rate we saw in may that it peaked at all-time high and then in june

dropped significantly and currently we are under 900 hash on average it's like 880 hash of course price has a certain influence on this because uh the hash rate depends on the profitability of mining so um here is charts since january on uh uh where just the price and the hash rate is put together so you can see that uh the hash rate somewhat reacts to the price changes as well and um uh white located hashrate uh so we can because the the total difficulty is basically just the cumulative difficulty the we can we can exactly calculate how what is the average hash rate in the

network needed to achieve a ttd at a given time so um uh here it's visualized over a bigger time frame with an example of uh the total difficulty value which will be reached during september with current numbers and what this shows is that like to achieve this uh total difficulty like during uh during august for example you see it's uh it's a huge number like up to five but the hash and then it goes it drops fast and then it declines slowly so it is the trend that we are working with and the red part is september so uh this is where we are on the curve uh and uh

we're using this script you can generate um you can run it just with uh uh ttd value or the total difficulty value to to generate uh the charts with and and and uh you can create charts which are below uh but uh how so uh why do would we consider here with this with this trend is that what you can see that in the first days in the first part of the month basically it takes uh more uh more hash rate to achieve the value and then it declines more slowly so we have like better protection against the uptrend rather than the downtrend and uh

so i was i was pessimistic with the uh with the estimates because uh of also the deck size issue and um and uh i rounded down the numbers as you you will see so the first uh the first scenario was counting on the battle trees happening on 31st august or maybe maybe first september which would uh which if we need like two other two more weeks to achieve dtd we would be around 15 september so first here we get roughly the number total difficulty which will be reached on 15 september and i round it down uh this this because also we want to have a nice number

uh which is a bit easier to remember and uh this number is however um it's not not much difference to these it would be like less than 12 hour difference with the current hash rate so and these these are charts which you can produce with the script above and these show us uh basically how much hashrate is needed to achieve the dtd over the time frame over the period of september so uh with this with this dtt value you see the the red dot line is the currently something under 900 error hash with the current current hash rate it will be hit around the 15th of september and uh if the history goes up up to

let's say 1000 here uh it would be it would be like 10th of uh 10th of september and if it drops to 600 terahash it would be by end of the beginning of the august and uh here you can see it's in percentage so like what percentage change of the hash rate we can we can expect and this is a table summarizing it and comparing it to uh all-time high uh history all-time high and uh and the current value so you can see that to achieve this immediately after bellatrix in the first days it would take a lot of hash rate which is probably not possible at all and then uh by the end of september uh even if uh the hash rate uh for

comparing the current number could go down 30 we would achieve it by the end of september um and here uh are other uh examples of ttd value which uh give us more space for uh history drop to be achieved by uh the this uh drop by the merge is meant as the 15th of september so uh if hash rate drops by five percent uh this value could be achieved under 15 right and uh you can you can use these values with the script to produce these charts if you are interested in details um and the second scenario is uh with

september 20 uh where the ttd values are a bit higher so uh here i rounded down the number again just a bit more it doesn't make much difference but uh the thing is that we have uh maybe less room to achieve the ttd here before the uh before defcon so um uh here you can here i'm here i'm uh um counting it uh by the uh maybe october seven so the numbers are here and uh yeah it's a bit less room maybe but but still reasonable uh yeah so i guess that's it uh if you saw the dog if you have any comments any

feedback please let me know

what

Danny Ryan: you think about the values about the approach when we say um alter alzheimer needs to be reached obviously it's for the entire time period is that time period starting today or in plus one week from today

Mario Havel: uh i'm sorry what do you mean the

all-time high

Danny Ryan: yeah for yeah to hit those like in that chart is that saying all-time high from starting today or all-time highs starting in plus one week uh this this is meant so this is just compared to the all-time high which was uh ever achieved on the

in the network

Mario Havel: so like that's like the the maximum uptrend we can come with but it's saying it's saying if like literally now

we hit all-time high hashtag this is the date the ttd would be hit correct yes yes

Tim Beiko: literally now or literally one week from now when that would work when i did a

dock

Mario Havel: yeah yeah okay it's from monday so the numbers now would be even higher yeah yeah okay yeah yeah

Danny Ryan: and you know the next week being kind of the defensive selection of ttd when probably an attack would actually occur because we would observe and adjust between now and then um that makes the number even higher

all

Mario Havel: right yes okay

yeah yeah basically the chart would stretch and you can you can generate a chart and see by yourself uh to to get the exit number for the current moment because uh it would be hard to update this all the time but uh yeah okay uh basically the the the closer we are the more it takes to achieve

Danny Ryan: okay so one of the things we'd also be observing over the next seven days would be an increase in hash rate such that one of these kind of timing attack scenarios might occur and if we didn't then we're in even better position with respect to the timing attack scenarios and what it would require starting a week from today um

[Music] so mario none of those so uh there's been a bunch of chat going on uh it seems like there's certainly a compromise and a bit of a debate as to how far after you do this um with respect to what is likely a september 6th bellatrix we see two weeks we see someone arguing for even less than a week and but we also see at least from the people who've been participating in that something like a 10-day compromise which i think adjusting for the numbers that mario gave us knowing that we would have good really good visibility on them on such an attack on plus one week from now um 10 days is

probably in that like safe spot with respect to all-time high which we can crunch those numbers a bit more um that's at least the synopsis from what's going on in the chat i'd now open it up to discussion points on plus x days from bellatrix

onscar

Ansgar Dietrichs: right uh so i just wanted to basically say that just because like any such attack we would see like well in advance as you're saying basically i mean some of these calculations say like basically the the the hashtag would have to have spiked since monday already um or despite business or whatnot so so we would see this well in advance so i don't i think basically kind of talking too much about

the specific kind of delay in days kind of gives this impression as if that would be a time period in which we would have to react which is the at that time it's it's already too late right so like the time period to react would be between now or whenever the kind of the attack would start right and bellatrix and so i personally would be very conf comfortable with even like a relatively small um kind of delay in days between bellatrix and ttd just because that's not the time period that would have to to allow us to react

Danny Ryan: so i personally think that the weak is that's plenty so there are two i think two things to to balance here one would be in the event that something with bellatrix goes wrong um a sufficient amount of time to

resolve those issues um and then another is um if we're going through the strategy that we will make it very clear that there will likely be hardened el releases after bellatrix some sufficient lead time in posting that blog post

and um we can also have yeah releases like sure it could be two days before yeah right yeah yeah so that does

Danny Ryan: um so you know at least i i don't think i would go less than a week i think a week with the error on ttd means it could even be five days four days

just on my gut so yeah other takes on the plus x days from bellatrix um i'm for 10 10 days at least

Marius van der Wijden: and i think we also have to consider that a lot of people did not follow this discussions they need to have time to set it up and they would set it up in the last possible moments so if we give them a couple more days to set it up right

that is a good point

Danny Ryan: there are two types of well there are many types of users there are validators who will have this set up before bellatrix almost all of them someone will forget but then there's actually all other types of users which some of them actually seeing bellatrix happen might be them to prompt actually setting up their nodes because they wouldn't they wouldn't actually be knocked off the network before then so there's a bit of an argument that there is a type of user that would be actually waiting until that last window to really be kicked into gear

Micah Zoltu: i would argue that that type of user is likely very common um because if you um

if you want to upgrade once and you don't want to do anything you're going to wait until the like you know day before or two days before you know enough time you can make sure your node syncs and turns on but if if you know like the hardened releases are going to come out later you don't want to deal with that you might want to do it all at once and so i suspect there will be some users who wait until a very large portion of users who wait until the last minute where last minute is those hardened releases and tricks to the stable and everything looks like it's healthy

Jacek Sieka: so other um [Music] other weighing ins on this uh one question is on the ttd built-in

into clients do we expect clients to have the dtd built in or do we yes force users to set a ttd flag

Tim Beiko: just so that no we would want the releases we would want the releases to be ready with the ttd and this is why we would you know tentatively choose it today client teams can pull out a pr you know with that value doesn't need to be merged and then we reconfirm on all core devs next week and then you can merge that number um so users don't need to use a flag except in like the emergency scenario where um we do a tpd override for a reason or another

um will we be doing main net shadow forks after bellatrix and do we need extra time for those

Danny Ryan: um i think the main net shadow forks will continue until the market we can do that before electrics as well yeah we'll have definitely have them before and um shoving one and or two in the middle also does not sound like a bad idea especially if client releases are still rolling out

Parithosh Jayanthi: yeah the plan is to have them like every week with the latest

recommended releases

Micah Zoltu: and so if we want at least one so that means if we do them once a week uh do they take like is there a minimum amount of

time um we need in order to get at least one shadow fork in after bellatrix oh

why should i need a shadow fork after bellatrix

Danny Ryan: because your clients might have hardened releases around them additional releases around them

Adrian Sutton: i mean i i just think i think if we're setting a ttd we're making the decision to ship so shadow forks are useful to keep testing out but i don't think we need to condition on it because we've made the decision to ship oh yeah i'm not by now we shouldn't be sending this

Danny Ryan: i wouldn't be suggesting conditioning on them but um they are valuable tests and if somebody has a regression last minute they can we want our users to be

ready and willing to ship and upgrade um and if we found a regression and patch something then it will be good to find the regression and it'd be good to patch

Tim Beiko: and then in practice um like you know we usually have the midweek we have the electrics happening on a on a tuesday north american days usually shadow forks happen on a wednesday or thursday and then we'd have another week from like that shadow fork before hitting ttd on main net um we could literally have a shadow for the day before mainland if we really want to yeah so i think we have plenty of time to like have a shadow fork fix massive like last minute issues from it uh do a ttd override if we found

something absolutely terrible on that shadow forge and still not be hitting

Danny Ryan: okay so um it does seem like people are generally okay converging on a compromise um there's a bit of a call for a september 15th number which i think was advertised in mario's document 10 days would be that september 16th are there other things we need to discuss right at this point or can this show up in a pr and a suggestion and we can circulate yays or nays there yes we'll be sending you a mic

marius

michael just some timing

Micah Zoltu: i feel like i feel like thursday is better than friday for if we're thinking between the two

i'm just because no one was working

Tim Beiko: yeah we're not though we're picking the illusion of the time right sure

Micah Zoltu: like if we target thursday we're more likely to hit not the weekend then if we target friday we target friday it's very likely we'll hit the weekends right now yeah we can try to not hit the weekend

Tim Beiko: oh yeah i agree yeah okay so yeah it seems like in the chats the original number suggested by mario of uh five eight seven five and zeros all the way to the end um has rough consensus

so we can i can open a pr on the the execution specs uh i guess a little bit of pr for belatrix as well on the cl specs um and can probably merge that pr on the cl specs and just reconfirm all core devs this number in case we see something funky um on the network or if for whatever reason the calculations are wrong so people basically have a week to just uh sanity check that number but the assumption is if if it is actually the right estimation and we don't see anything uh weird on the network with regards to hashrate we would merge that and expect client releases pretty quickly after

Danny Ryan: okay

any other comments here before we move on great we have about 15 minutes left um i don't think that we're going to get to the 484 discussion today i would i think me boost stuff is a bit more pressing with respect to shipping the merge um first of all is there a general kind of immediate mov boost status update and then we can get into these two items of discussion and it's okay if there's not i don't know if i have anything to add

i think i see leo on the call i'm here sure uh

but well

the general status update is the same as this week testing and focusing on liveness issues for mitigating extreme cases and working on the relay monitor so i think just talking about the monitor seems appropriate here unless you have questions

Danny Ryan: okay great um so the first point which was brought up by alex is clarify local building should happen in parallel uh alex can you talk about this please

Alex Stokes: yeah so uh there's two pr's that i'd like to discuss the first one

should be pretty straightforward let's see i'm gonna try yeah let me drop it in the chat so this is really just a clarification in the builder specs and it concerns how proposers should be using this external builder network that exists bmf boost and the way the spec was written is i think you could interpret it as uh so basically there's this timeout where if you call to get an external block or remote block uh you basically give that network like let's say a second before you should say okay they're not going to like help me now and i should build locally uh

without this pr i think you could read it as i should not do anything until i know that decision whereas it's much better to have started building ahead of time and that's all this pr does is clarify that you should do both things in parallel

Danny Ryan: which i'm in very strong agreement of there's a point of no return when you sign that like uh blinded block but before that which is also called need to be

ready yeah

Alex Stokes: right so yeah this one's more informational just be aware of this if you have some spare cycles which i know we all have tons of time over the next few weeks but

uh if you have some spare cycles just double check that this is the behavior that your uh beacon node and or valerie clients are following

Danny Ryan: thank you and then alex i believe you have the next item as well which is the uh circuit breaker which was discussed i believe a couple calls ago um any it was a status update here

Alex Stokes: yeah so we discussed this on the last cl call uh just to refresh everyone this is the idea that uh regardless of this external block building network or not uh you know we could still get in a case where suddenly there's a liveness

you know a series of lightness faults even a lightness failure on chain right so things can get quite serious and the idea is okay it could be possible that this external network is doing this and so the sort of most paranoid and or most sort of secure thing we can do is just stop using it at least temporarily so i had this like a sort of sketch of a proposal last time i've refined that into pr so yeah i suppose i would like to get feedback on that i'm not sure if you guys have had time to review the pr uh i guess i can just at a high level go through it and the idea is that there's some rolling window so basically you look at let's say an epoch of of slots and

that's a rolling window over the current head oh yeah thanks terence so the pr is here uh 47 on the builder specs you have this rolling window and then basically if you have some number of missing blocks within this window you say okay circuit breaker on and i do not use netboost when that condition reverses in the sense that within the rolling window i don't have that many missing blocks then i'm free to use it again if i want so [Music] then the question is like yeah how do we pick these numbers i have a suggestion here basically saying like you know if there's around half of the blocks missing in a given slots for epoch number of slots then we trigger this

thing uh i think we can probably talk about these numbers and or yeah if people want to take a look that would be nice but yeah i think i guess first are there any questions right now from what i've said or something that was unclear

Danny Ryan: so there's probably two design points here one is to do rolling and be able to turn it back on in which we could probably have a smaller window as you suggested with 32 or we don't have rolling and it's just you just turn it off it needs manual intervention at which you'd want a much longer window with an assumed adversary size so that they couldn't you know with random shuffling they wouldn't be able to the adversary of size x wouldn't be able to do that and just uh cause the shutdown

Alex Stokes: right so i don't know if people have any

of the you know if the cl devs have opinions here um the idea of the rolling window is that it basically makes it such that there's automatic recovery so uh the thinking is that it makes it a bit harder to game uh just because you know you couldn't trigger this somehow and then take off the whole block building network for like you know a half a day or something while people recover um instead it would be this like automated recovery

Danny Ryan: i guess there's a complexity argument one is every time you're just querying your state because you can check empty slots from your state and you're just it's a dynamic quick query the other would be i guess a query along with maybe a database flag to turn it on and off um my intuition is actually the former with

the rolling window is less complex but i could definitely be wrong there

Alex Stokes: right and so there's also kind of segways into like i don't know if there's a second design point that were going to call out but essentially uh wherever this pr suggests that clients actually randomize their values within this window and so the idea is that if you don't know exactly what your client is going to do then it's much harder to game right you sort of just get this like herd immunity across the whole network because you know this node is doing one thing this node's doing another so it becomes much harder to like find these boundaries to try and try and trigger this thing maliciously

Danny Ryan: um you do still get like a profitability distribution um which often is enough for an attacker

to do something but i yeah think about it

uh so in lighthouse we actually have an implementation of this rolling window and we check for like if there's been eight skips in the last epoch and it's it was pretty simple to implement

um yeah open to making it like a random number of skips in an epoch or something like that so yeah prism has rolling windows too just that was easy for us to implement we haven't decided on a number yet so

we're still talking internally and i want to point out one thing we're doing differently is um instead of counting the missing slot uh it's like an orphan block we count the missing slot as the block that has never arrived so they kind of give us a little bit of client directivity there too so i thought that's nice okay so you're not just doing a direct state read if you actually had a block for slot n even though it didn't make into the canonical chain you're gonna count that as a hit interesting something that i believe lighthouse has is the ability to use a configure the number of skipped slots i think that's nice because it gives us

Paul Hauner: the diversity as well um

and i think it's also useful as well because i think there's going to be some users out there that are kind of just you know hardcore um i just want all my mav and never to stop um and and generally of the opinion that we probably want to try and like allow them to run safely without creating forks and trying to manage their own repos and stuff like this so yeah giving users the ability to tune that i think is helpful

Danny Ryan: so we could just say in the builder spec recommended value and by that it's tunable i mean it's not a consensus value so you can't enforce it really anyway

Alex Stokes: so we would just apply one fix like you

know pick a number 16 out of 32 blocks or slots and then also say by the way let the user configure this

Danny Ryan: yeah i would pick a number that we kind of deem safe under different considerations and then say that it can be configured locally

Alex Stokes: okay so another consideration here is that it would be great to have this rolled out before the merge or as soon as possible right so uh i think taking that into consideration this is hopefully something that's like pretty straightforward to implement and it

isn't to uh to a hairy so i guess if no one has any comments on the general design then yeah we could take the specific parametrization to the pr

Danny Ryan: okay great um i don't believe that starting this 4844 points of discussion there are many points of discussion is going to be fruitful in the next four minutes so i would suggest we defer into the next call given that okay 10 please

Tim Beiko: oh as i think that's reasonable there's nothing that's like urgent for today

Danny Ryan: great so let's um are there any other final discussion points for today excellent thank you everyone i will talk to you very soon and uh talk again on awkward devs in one week thanks everyone

thank you everyone [Music] so

[Music] [Music] [Music] [Music]

[Applause] [Music] so

[Music] [Music] [Music] you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.