All Core Devs - Consensus #80

Transcript

[Music] [Music] so

[Music] [Music] [Music] so

[Music] [Applause] [Music] [Music]

[Music] [Music] [Music]

Danny Ryan: okay stream should be transferred over if you're in the chat on youtube let us know i just dropped the agenda in the chat um first half of the call we will talk about kintsugi merge things second half the call general consensus layer things if anything comes up there is a pending release with the last little bits being worked on so we can talk about that then there's been a number of cool testing things um devops things that we can talk about and then leave some time to talk about our uh our next steps the release that is coming uh is based off of the discussions

two weeks ago in the discord about some of the semantic issues with um how the engine api works and how execution layer clients generally work and has been talked about on the last two calls so shouldn't be surprises here but it is coming it is a breaking change and uh we'll need to to engineer it um i believe on the marius knocked it out in a couple of hours so at least on the execution layer side it is not a deep change it might be a little bit um more to deal with on the consensus layer side i'm not 100 sure um

generally happy to open up discussion on these two um or on these these items it's really uh this pr 165 which is listed uh by tim here the authentication pr 162 is also coming um it's kind of in the final little discussion points ironing out points um i believe and then the optimistic sync spec which a lot of you guys have been working with has been merged i think there's a tiny little semantic change to get in to match the adjustments in the engine api but that if you've been following it has been shaped up really for the past

few weeks but finally will be released anything on these items um again we've discussed them a couple times um if you i'm sure there might be some some feedback once you'll start doing some engineering

Paul Hauner: i wanted to mention something about optimistic sync

Danny Ryan: yeah

Paul Hauner: um so there's one so that the api needs a little bit of thinking about i shared a link on the merge channel um earlier this week about it the problem broadly is that we

need to figure out how what we want to do with optimistic blocks whether we want to serve them over the http api or not broadly there's kind of two ways and we could probably settle in in between somewhere um one way is to never send them out over the api which means the api only returns information about fully verified things um and the other one is to continue to return information about optimistic blocks via the api um the reason we would want to continue to return optimistic information on the api is so that validator clients can continue to follow an optimistic chain they can keep up they can know their duties but not sign messages about the chain but just know

their duties and keep subscribing to subnets and things like that so that's kind of the reason why it would be useful to serve optimistic information over the api so hopefully someone i don't have to chat here on my phone but hopefully someone can link to that document um i shared in the in the merge discord channel earlier uh so you just need some some thought about that um yeah if you get a chance just maybe bring it up here or raise it in that merged discord channel

okay

Mikhail Kalinin: yeah i have a question with what paul have just mentioned do we need to serve optimistic blocks to validate our clients to allow for a validator client to get

assignments

um the duties

Paul Hauner: yeah i guess you you might argue that it needs to serve out head blocks and and um survive the the um events api events the server-side endpoints server suicide events endpoint but not necessarily not necessarily blocks but just more information about the head about the shuffling of optimistic blocks things like that

Mikhail Kalinin: um because i was like thinking a little bit about this change and from my perspective like this assignment

look ahead is one of the main things why we want to do this and if uh what we likely will just share a minimal optimistic information that is required to satisfy this uh look ahead then it's it should be fine but do we need to account for anything else do we need to consider anything any other reason for sharing this optimistic information yeah and also it's it's it's probably um valuable for debugging purposes it's also

considered

Paul Hauner: yeah it'd be really key to hear anyone

that's using the apis and will say you know if you serve any optimistic information then you're going to break our thing or kind of looking for broad feedback there

yeah because our like

Mikhail Kalinin: previously our our line our general line of thoughts with respect to the api was as follows so we're thinking and all optimistic blocks we assume that that they are not yet fully processed so we just don't serve them and i think it's reasonable and we don't want to like change the direction yeah but but also uh allowing for making look ahead is

it would be reasonable to share the information that required for this book has as for me yeah this this is all that i just wanted to share

yeah thanks

Paul Hauner: that um link i was talking about before has a link to a lighthouse issue number 2946 that's where i've kind of started to sketch out i've enumerated all of the api endpoints and then listed how they could change so that we only share the minimal amount of um optimistic info in order to keep the value of the client's work it has a couple of little open questions but um yeah that's an issue if you when you want to compare all these things

right

yeah i

Danny Ryan: mean i see the simplicity of just adding a flag uh but also i think we want to make sure one of the big things is probably that end users don't make bad decisions um by being served such data uh bad decisions as in assuming things are canonical or getting mixed up with the wires there uh i i could imagine if you if we do serve it there's probably some work to be done on consumers so like if you are a block explorer you certainly have to decide um what are you actually serving here um whereas if we don't serve the data from the api it is much clearer you just

continue to follow the blocks as they become canonical and fully verified it it is it is weird to not have the data available at all it's kind of like funny to have the this relatively critical process going on and not to have some insight so certainly on the debugging purposes like i see the argument i did share a link um paul would love some more feedback and input there uh anything else on this one today not for me

okay great um it looked like perry had some testing updates and so pair you can get us started um i guess give us an update on shadow forking and then we can talk about we'll talk about testnet stuff in the next steps but yeah let us know how shadow forking's going and then if anybody else has testing updates we can talk about them

Parithosh Jayanthi: sounds good um so just to give everyone a brief update at some point last week marius rafael and i were able to shadowfork early that means we've essentially merged girly while not touching the canonical chain so it's

just a fork that we're running ourselves because it's a fork we still receive all the transactions and inherit the state from gurley as long as the state doesn't deviate too much we just continue getting all the transactions we'd assume that this is great for sync testing especially under load and if we want to do non-finality tests with actual state then this is the place we'd like to do it uh that being said it's still a real open question as to what all we want to test so i just created a small document to try and write down everything if people have ideas please comment or make changes there and we'll figure out how to get the tests in action um i

think that's about it for shadow for me and also we need to decide how often we want to do that because we now have an easy way to go through the merge transition with state really easily

basically

Danny Ryan: yeah that was gonna be my question as well so it's it's relatively easy to push a button or even schedule at this point

yep

Parithosh Jayanthi: and also gurley has the really nice feature that the difficulty just increases to every block so it's very easy to time out

right cool

Danny Ryan: yeah i i mean it seems like we certainly should be doing this weekly for the foreseeable

future um with updated builds just to as kind of a continuous live integration environment um there's also there's work to do similar uh things around the transition within hive um so that would that could serve some sanity check building as well but i i think it's pretty valuable um have we considered doing it to mainnet

Parithosh Jayanthi: not yet but we wanted to first maybe standardize or automate some of the tests and then apply it on mainnet the main reason we don't want to do it on mainnet right now is it's a lot more difficult to figure out when the transition will happen and also to sync up a note will take like a day

Tim Beiko: why is it difficult is it just because the rate of the difficulty increase

Parithosh Jayanthi: yeah because it would depend on the hash rate at a certain time hopefully it's very easy to estimate

it's just too bubbly right

Tim Beiko: i think once we have gordy like more automated it makes sense to actually get practice runs of estimating total terminal total difficulties on magnets and seeing when they actually hit because once we have to actually do it for real it would be nice to uh they have like tried to estimate the data a couple times and and seen how close we got to our predictions of when it would hit so i yeah i don't think it's the most urgent thing but i think

it's worth kind of going through that inconvenience a few times um before we actually

fart from england

Parithosh Jayanthi: makes sense

Mikhail Kalinin: yeah also uh the question with respect to the mainnet shadow fork is with a deposit contract will we have to emulate it or what we have to what what will what are we supposed to do with the deposit contract on the main

ad

Danny Ryan: so you don't actually have to make any deposits because you can just start with a genesis state with validators um you can use like an erc20 variant of the deposit contract to gate deposits if you want to actually test that functionality but

it's probably not the most important thing that we're testing

Mikhail Kalinin: oh yes so we don't want to start another beacon chain to launch it like from the main head we just want to launch it like as a side chain

um quickly like independent from the main ad and then yeah i get it

Parithosh Jayanthi: yeah just to give you an idea of how that works currently um i just changed the fork version so all the deposits on the main contract would just be deemed as invalid on my chain and all of the deposits with my fork will be deemed invalid on the other chain of course we wouldn't do this on mainnet because then you're wasting a lot of gas but for girl it doesn't really matter

Danny Ryan: are you actually making the deposits on

gourley

Parithosh Jayanthi: i've just made one so far if i'm not

wrong

Danny Ryan: okay but when you're popular you're just pre-populating the genesis with validators

yeah mainly in this population exactly

Mikhail Kalinin: we can make some deposits after uh the transition on the shadow main net but if it if it makes any sense

Danny Ryan: yeah well and and um again there's an kind of erc20 mod of the deposit contract that if we want to do that type of thing on main net we can it'll still cost us gas but it won't cost us 32 eth

other testing related items that we'd like to discuss today

yes i

Parithosh Jayanthi: wanted to talk a bit about if it's beneficial to have a non-finalizing test net for now it seemed like we were able to trigger quite a few bugs on kinsugi and it might make sense especially with sync testing if we have a non-finalizing test net also for tooling i think it's relatively unclear to a lot of people as to when to show updates what is deemed as optimistic how to deal with stuff so i'm just not sure if it's too early for for a test like that

Danny Ryan: i would say um when we're

given the the breaking changes that people are gonna have to work on over the next week or so that um i would target these more exceptional cases to be after we get that done uh just so that we don't have especially if we want to get other people using them um things are going to change and so builds are going to change and i'd rather have them target more production builds um yeah it's it's probably worth even on like some something of a continuous basis like building a chain with exceptional scenarios and and watching it run

Parithosh Jayanthi: sounds good um besides that i guess the

other testament question is the khinsuki successor um do we want to do this after eat denver during each denver what's the what's the plan

um i mean

Danny Ryan: we're at the as soon as possible part of the the phase where uh we need to get this release out immediately um and we need to get engineering done uh changes to the semantics as soon as possible and then start probably a couple of um sandy check builds before doing standing up a test net that we want to stand up for a long time my estimation is that

certainly we're not going to be doing that next week but if we are in a good place to thinking about doing it in two weeks that'd be great and if it's three weeks you know we don't want to go more than three weeks probably if possible but there's certainly kind of an unknown stretch in the middle between getting these updates done in clients um i would as soon as any clients are ready i think we want to do some like sanity check builds um and start just like standing up a test net that we drop real quick or don't promise to uh keep running for a long time

okay

Parithosh Jayanthi: that sounds good to me um and the last question i had was regarding paramount has everyone already

exited the validators is anything happening there

Terence Tsao: um for the prism site we're still running nodes there just for a testing purpose we want to uh see how performing our note is so we're still using it but i'm not aware if any other teams are using it

Danny Ryan: yeah it's officially whatever that may mean deprecated as in the client teams here are not promising that it has any stability and that it's good for end users but yeah some people may still be messing with it

Paul Hauner: sigma primer in the middle of exiting our validators by in the middle i mean probably maybe 10 through we've

submitted all the exits so we're just waiting for the queue so once they're all gone then we're shutting off our

servers um thank you nodes are all down nimbus is using a pyramid for the insectra

Mamy Ratsimbazafy: test where we created the fork malicious one of the of the chain and we are testing if people are using checkpoint sync on that what's happening

Parithosh Jayanthi: sounds good thank you

Danny Ryan: anything related to testing people want

to talk about i know there's various simulation efforts in progress i know there are continued hive efforts in progress and all of it will have to integrate upgraded updated builds in the next week or so

Marius van der Wijden: i started writing some test vectors for for the for the new changes i don't know if you already talked about the new spec changes i i joined a bit late but i have some test vectors so it should be easy for everyone to use that i'm also going to build a version of gas that can sync with the new with the new spec currently the the

nodes cannot with the new spec cannot sync but can just execute the blocks

Danny Ryan: great

Tim Beiko: one more thing just for people to be aware we talked about cornell's last week about basically what do we want to do about test nets after the merge because we have a bunch of uh execution layer test sets today and which ones we want to transition over i think roughly the the consensus was that we want to keep uh

one test net with a somewhat open validator set one test net with a somewhat closed validator set and we probably want to also transition robsten but eventually kind of sunset it um so on the on the execution side we'd probably transition gordy not transition ring could be transition robsten with the intention of deprecating over time and then transition sepolia which is the new test set uh with the intention of it being a replacement for robson over time um so no need to decide anything right now but i think it's worth it uh to align

obviously with the consensus air teams on that to make sure that every every test net that we're targeting to keep has a long term uh consensus their test that that's that's associated with it

Danny Ryan: so a few things the prouder evolution of gourley i think we're in reasonable shape and that can be a very stable test nut the other type of test that you said uh like no gating like no no uh very stable backbone from us and others

Tim Beiko: yes are they very i'm not actually sure how crater works i think is that most of

the validators are just run by like known entities but anyone

enjoying them yeah i write something like

Danny Ryan: something like 80 um right is us so it it it generally is stable unless there's some sort of

bugs right yeah and then yeah yeah i'm i'm not an expert here i if

Tim Beiko: there is value in a test net which has like a flakier validator set uh should probably do that i think but i can also see if that's just like way too unstable and provides absolutely no value

um so yeah

Micah Zoltu: michael so the the goal with having one of the test nets be validated by real people is that it allows the apps who want to

harden their applications against you know finality failures and things like that and give them a place to do that that is you know a long-term test down like they don't have to spin up their own and try to simulate their own finale failures because if we have regular people validating on test has been particular as we know they will just disappear randomly and then they'll have to leak out and then eventually we'll fight get finality again and then new people will join they'll leak out we'll get finality again and so it comes comes and goes and so it's a great place to harden your apps against

Danny Ryan: yeah understood i think we need to think about what like if we do that how many validators are run by a more professional group uh if if any

um and i guess what the requirements various teams or maybe the devops team would be here

Micah Zoltu: am i correct correct my understanding that if um we have an unreliable set of validators we'll still get blocks we'll block production just there'll be some missing ones here and there yeah and we just won't get finality until people leak out and eventually but we will always eventually get finality right

Danny Ryan: uh yes so that yeah that that's that all that premise is true um so i guess how do you how do you kick it off do you have because if you have any amount of professional or client team run validators quickly they'll

over the course of a few leaks and people coming and going they'll just become a very dominant portion of the set and probably then you have a stable test nut until somebody like dumps a ton of validators on

Micah Zoltu: um so you're saying we built things too well and we'll always trend towards stability no matter how hard we try

Danny Ryan: unless we just don't go on it at all

Micah Zoltu: even then we still might end up with stability because you know we just might stand up with some dude on the internet who just decides i want to own this test then right

and it will

Adrian Sutton: eventually it seems like it might be better off to just have us run a bunch of validators on

on the unstable test net but deliberately turn them off regularly and you know we can deliberately cause non-finality and we can deliberately cause chaos in that test net with anything else it's yeah it's actually really hard to know what people are going to do and if it's just that that is going offline you tend to get non-finality but not chaos and so it doesn't test anything because you get a very stable chain there's still a single chain and life just kind of goes on whereas the problems are when you get bug which is the only time you'll really get non-finality on mainland most likely and once you get that you start to get an explosion of forks like we saw on kinzuki and that's when you start to get trouble

Danny Ryan: yeah i guess we so there's a few things there uh every other week turning off half the validators is potentially valuable to those that want to test in a non-finalized environment um but if the issue often is when you have an explosion of forks we could also think about um how to induce such an explosion of forks rather than hoping for a bug that gives us an explosion of forks

Adrian Sutton: yeah just kind of a chaos monkey in there and some nodes that deliberately every every other we make new forks from 100 blocks back

Danny Ryan: yeah every other week turn off

40 of validators make 10 validators chaos monkeys and uh have the 50 run normally and then the next week turn everyone back on normally

Adrian Sutton: yeah i think that's the way you're going to get a an actual chaos test net i think if you just leave it to the public then there's a very good chance you just get a dead test net because everyone wanders off because it's not interesting anymore or you do get enough people running it stable and it kind of does centralize on them and stabilizes

Danny Ryan: yeah i think i agree

i mean

Jacek Sieka: the alternative here is actually to

launch little insecures every now and then it was kind of a fun process to get that launched because it like it uncovered a few unexpected effects of turning off say after validators now i was running it with you can for example not run attachment with five percent of divided errors that doesn't work um you can easily like if if we have the validators you know known to to a group of people that group can easily generate like spin-off um spin-offs that basically bleed out everybody else

so that's one way of doing it and and doing so also stresses forks because you can then generate forks at will and so on

Danny Ryan: right but when you insecure is going to test like a only a subset of what we might want to test right because if there is a continued public test net in it then those that are synced are going to continue to follow said public

testnet

Jacek Sieka: yeah for sure but if we like that could become a weekly process

like a new chaos right and then you have like a stable test net say whatever it is uh pratter now and then you generate like a chaos test every week which is broken in

in controlled ways from the broader snapshot right okay so uh i guess to naively

Danny Ryan: we certainly want to have a stable test

net that we point people to for testing standard things um and then i think we want to maybe get a little bit more clever about if we're going to create an unstable environment how to do so in a stable way

Tim Beiko: yeah that makes sense and i think and also in practice in the short run it's going to be two stable test nets so for example if we fork both uh robsten and simpolia we're gonna need two beacon chains for those with the expectation that um we shut one of them down in however many months after the merge so short term we need three with the goal to drop the two after a couple months

yeah and

Danny Ryan: so we can also um we essentially have the ability to make a

proof of authority test net by using the erc20 variant of the deposit contract where you just can gate it and you give these tokens to whoever we want to run validators so you have the prodder which you know we keep 80 very stable and people can join and we we make sure is is open and then if we want something that's more like gourley we can use this other gating mechanism as well it's just a tool in our pocket

Tim Beiko: sounds good i noted a couple of these these ideas we discussed in the issue on github so we can create a conversation there

Danny Ryan: for the result um sorry for the the

release that's coming there's one thing i think that i wanted mention i think that we have a we need to get the slight semantic changes into the optimistic sync spec does anybody want to take that in the next 24 hours take it doesn't do it

does that

Paul Hauner: update the optimistic sync api for the new execution api

Danny Ryan: yeah

yeah yeah uh yeah i can i can get on that um yeah thank you all right

Danny Ryan: um great so i think another thing we want

to talk about is next steps we talked about a little bit involving uh clearly kinsugi as it stands today will be deprecated due to the breaking changes in the engine api um and the intention is to stand up another um long-standing version of that maybe kimsugi v2 on the order of two three weeks the difficulty bomb is looming and there's plenty of things to do between now and then tim do you have any discussion points based off of some of the the planning you've been thinking about

sure um difficulty vlog

Tim Beiko: is hard to estimate but

roughly we can expect it to start going uh start to be noticeable on the network around june and for the network to be completely unusable around

mid-july um not complete wait completely well defined completely unusable 30 second box right and that means that we can

Danny Ryan: because it literally becomes completely unusable

right

Tim Beiko: probably two weeks later yeah exactly so it's like you go from 30 seconds to probably one minute block uh within two weeks so like and and the there is like an economic cost of like once the network is at half the capacity it's like half the economic activity can't happen fees doubles so or probably more than double so like it's a

pretty terrible spot to be in a 30-second box um so i i think you know that means we we probably want to um to if if we're not going to push back the difficulty bomb again we probably want the aim for the transition to happen on main net in june ideally like the you know first half of june at the latest um back of the envelope you know looking from that it's like what what do we need to do to to actually get there um i think roughly we would want to be in a spot where we're announcing client releases for for public test nets uh like existing public test nets around uh around mid-march uh which then gives us a couple months for those to like for people to upgrade their nodes for

those to fork for us to make sure that there's no issues and then uh to pick a ttd on main net and then put out the release with my net um so you know at roughly i think if we're in a spot for like early march so call it like a month from now because january is basically over client code is basically done uh that would be like ideal to hit this target there's probably some wiggle room we can have from there but there's not like months of wiggle room um so i think you know if we it like we were talking about in the chat earlier if we aim to have kenzugi v2 by denver which is like mid-february i think we're in a good spot um assuming

obviously this all assumes you know the latest changes to the spec are kind of the biggest changes we still have to implement we'll probably find some small bugs here and there but uh it assumes we don't find like a fundamental issue or something that requires like significant re-architecting of of how things are um well yeah so basically i feel the art is like getting consumed with uh mostly final spec in like the next couple weeks get it after that you know a couple weeks after that getting the final changes done and then starting to look at um at going on on existing desktops we would be in a really good spot

Danny Ryan: feedback reactions

any discussion points on that

Mikhail Kalinin: i have a comment on that um [Music] this is pretty tight schedule and uh um i i'm just you know i it's not about clients uh clients are not ready in april or this kind of thing but what worries me here if we take this shadow if if there is something appears not probably a fundamental issue but some issue that we will need to like test with more care on the testnet to have like two more weeks of testing it um it might be the case

that we will have to delay the merge for of course a reasonable thing and if we have this tied schedule we will we are risking to be between two fires so we have to release and we don't have the time to properly do all the steps right with respect to some unexpected stuff so that's that's what worries me with the with this

i think

Tim Beiko: yeah i think if we're if we aim for this we we have the wiggle room definitely for like one issue like that happening cause it's like a minor issue where like we need to do a fixing client pull out new releases uh test that for a couple weeks

maybe two but like if that happens three times i think we basically have to push the bomb again so that's roughly the like yeah we don't have like one logistical question on this

like

Vitalik Buterin: what time between the merge fork and the terminal block are people expecting to

have right i think yeah my

Tim Beiko: my thoughts on that was when we actually fork the test nuts we should probably leave a reasonable amount of time call it like say you know four weeks from the ef blog post but then when we fork mainnet we probably and the reason for that is it is kind of a significant upgrade right like everybody's running like one client now they need to run two so like give a couple of weeks for people to do that

ideally they've already done it on consugi but you know don't want to break out that but then when we actually do main net i would make the delay much shorter because of the difficulty of predicting the actual uh difficulty right and the fact that uh you would think hash rate would drop near the merge because people are going to try to sell their gpus already so right so i think we can we can hint when we put out the test net release we can say hey there's four weeks before the test nets um and by the way when the main net release is going to come out it's going to be on a much shorter cycle so keep an eye on this

um so

Vitalik Buterin: i think that's uh reasonable like i guess the other thing we could consider is like if there's like if there turn out to be lots of

problems then the merge fork itself is like an opportunity to shut down the difficulty bomb and that would buy us like a couple of weeks time

Tim Beiko: yeah that's a good that's a good idea uh and it's like during the same release we could push back the difficulty for like a month or something yeah

right and

Danny Ryan: um part of italic's question was also the the time between the beacon chain fork and the difficulty bomb so you know if you're call it maybe client releases then plus 10 days to that initial fork so that's really 10 days to get things configured and then plus a week to the difficulty bomb i mean sorry to the transition difficulty so something on

the order of slightly more than two weeks i'd say yeah full um i mean that's that's aggressive um but if we set the expectation uh maybe not

Tim Beiko: yeah we definitely need to set the expectation once we have the test that releases out um and even maybe before like once we

announce constitute um yeah but if

Danny Ryan: we're in difficulty bomb zone and the difficulty is also changing because mining makeup is changing it's uh trying to estimate beyond two weeks might become very difficult

right uh what would be okay

Tim Beiko: so if we decide to just push back difficulty bomb what is the procedure how much how much time do we need in

advance to like to announce this and set expectations what maybe

Danny Ryan: first week of may for a fork that's a month later

Tim Beiko: yeah i think that assumes we basically do what vitalik just said um i think there's two worlds right one is where we just need an extra couple weeks and we do this like combined difficulty bomb push back and merge fork

Danny Ryan: there's a world where like we need the extra two weeks doesn't actually buy you the extra two weeks you just gave people client releases

right right um

so you could do you're saying well

Vitalik Buterin: i think the idea is that it's okay that like if we end up giving people like we can wait all the way up until the block time goes to like 23 seconds or whatever to give people quiet releases

oh because

Danny Ryan: because that means when it reaches 30 seconds

Tim Beiko: right exactly because you can simultaneously diffuse and set the difficulty the thing for two weeks from there

yeah okay yeah

Danny Ryan: okay so i think that's like a little strategy yeah but we have to be like i guess pretty confident

Vitalik Buterin: yeah right i mean this is definitely already in a kind of contingency planning territory

yeah yeah but

Tim Beiko: that's like we need it an extra two week territory i think if it's like we need an extra two months

territory then it's basically you have a separate hard fork which just pushes back the difficulty bomb um yeah which we can we can implement pretty quickly it happened once in a matter of weeks when we literally forgot about the difficulty bomb um so it's not it's not like hard technically to do um yeah but i think and basically i think the right time to have this this conversation is if we do get to march and we feel like we're not close to being confident to have like a a fork for uh for public test nets then it's worth having the conversation around by you know why aren't we confident and you know do

we think that this is a week's themes or amongst things um and and we can kind of decide from there yeah yeah he's making me sorry go on

danny please

Adrian Sutton: it's making me really nervous that we've got essentially an arbitrary date our scope is fixed like any product manager walks into an engineering team and says hey guys i'd like to develop a fixed amount of software in a fixed amount of time what do you think it's not a conversation that goes well for them and that's kind of what we're doing to ourselves now

Danny Ryan: um i mean it certainly is because it's

the reality situation

Adrian Sutton: but yeah i mean i i like i get it but to me i i really i'd almost want to start from the assumption that we'll we'll push back the difficulty bomb because we want to get the merge right we want to make sure that we we've actually you know we are actually confident in it rather than we've got this date and we we're kind of going to er on the side of shipping i'd rather on the side of delaying and and be confident

Danny Ryan: i kind of take the counter to that and that we have to try to arrow on the side of shipping because we will take as much time as is allotted plus some amount of time no matter what that time is selected as

Adrian Sutton: yeah i can agree with that but when i look at the state of things you know i'm not sure we have a you know an entirely complete optimistic sync implementation at all right now to the point where you know you can just have the cl running and we're just starting sinking tests of things we're still working a lot of details there's still a lot changing i i know we've got kind of another month before we're talking about being done but man it feels like a lot is up in the air and there's an awful lot of engineering work before we can get into starting to test and starting to decide if we're confident that it really works i might be wrong in that i can be quite

pessimistic about these kind of things but um it really it really is worrying me that we're putting the difficulty bomb too much over our heads just yeah that's just my worry

Danny Ryan: no i i i certainly hear you i'm kind of in the position of uh i mean on the one hand we've been working on this for a long time um and a lot of the unknowns are uncovered and a lot of the core is in place but uh you're right i mean there are there's plenty of engineering to do and there is a lot of testing to do um and so

my personal is to attempt to get it done um but to be very willing to make reasonable decisions along the way um you know if we if we drop two weeks here two weeks there another week there then like certainly you know the timeline is too aggressive um but i'm tossing it out entirely at this point i think we'll slow things down more substantially than if we attempt to hit it

Adrian Sutton: i certainly don't mind having relatively aggressive timelines for consumer v2 and saying we want to have optimistic sync fully working in all clients and

you know this should be a reliable test that users can do all kinds of crazy stuff on with rethinking els behind our back and that kind of thing and have it all work um certainly have to keep pushing for that that means we can we can hit a merge before the difficulty bomb goes off then great um i think i think we just need to be really careful in setting expectations when we're talking dates like this that you know now now that we're starting to have these conversations people think the merge is slipping if we have to push back the difficulty bomb and you know beating the difficulty bomb is not something that i i'm prepared to commit to right now in terms of what i see in in the way

software's ready

yeah i

Danny Ryan: understood i mean to anyone listening uh it's more important to do the merge right than to do the merge by a particular date

right and

Tim Beiko: i agree with that and i just think we need to be mindful of like the bomb is there and we need to think about like i if if we do choose to delay it ideally we make that call kind of early and and explain why and i think that helps kind of set expectations better than if we're like mid-may and people you know know the bomb is in june and

then decide to delay it obviously if we find a major issue mid-may we would delay it but i think yeah just being aware that it's there and that you know there are a bunch of intermediate steps and and and if we see we're not actually tracking to something where it's realistic that they hit it then we can we can kind of make that call and have that conversation earlier and and i think that sets expectations better than than leaving it the last minute

it is

Adrian Sutton: definitely very helpful to have this timeline thank you for for pointing out we need to be ready by march to beat it basically that's that's key information

Danny Ryan: indeed i yeah i mean i i see the the difficulty bomb and the

stretch of things we have to do between now and then in that reality um it allows us to to create some milestones and checkpoints um that are on the more aggressive side and i think that we should at least utilize those uh as milestones and checkpoints to kind of check in along the way for the next few months

Adrian Sutton: yeah let's let's be really strict on those milestones though in terms of saying whether we're done or not because i don't know in the original plan for consugi we wanted optimistic sync done we we launched without it which i think was great but i think we need to acknowledge we

didn't hit that milestone either um yeah so when when we set these let's be really clear about what's in them and whether we've actually achieved it and level of confidence

Danny Ryan: tim on some of the things we've sketched out in terms of hitting those test nets and stuff maybe you and i can spend some time actually attaching some of the technical things along with what we want to see with them along with some of the the testing that we want to see as well so we can hold ourselves to that

Tim Beiko: yeah that's reasonable

Danny Ryan: yeah to fizz's comment uh frustration

stemming from poor singularly import communication i i agree i think it's also it's quite a difficult process to communicate about because there are so many different moving pieces and teams and uh the uh there's not a a dictate on high saying this is the timeline this is the ship date um and it's it's kind of more of a an amalgamation of many moving pieces so i apologize for the the poor communication i think we can certainly do better but it's also a product of the process itself okay um

anything else on next steps timelines difficulty bomb okay um anything else related to kenzugi the merge or cross elcl anything

yes so i wanted just to ask how the

Saulius Grigaitis: basically the question is does everyone agree that the polling approaches

for api is good but because for some calls and for some situations it seems that it would be great at least to have an option to to have a blocking call and to get the response immediately if it's available so is there any any ideas on that because the engine api it looks more like you know not a most of the costumes uh they do not return immediately that that that sometimes uh is very usable and sometimes uh

it's actually possible to return that result

Danny Ryan: i'm not sure exactly i understood the question um the polling so which which end point in particular we're talking about with respect to

poland so

Saulius Grigaitis: so basically if we if i add the payload if i insert a new payload and it's and it is quick to to actually validate it then i would like to get the response immediately that it's

it's valid uh basically i'm okay to wait uh like 200 milliseconds or whatever it takes to validate it

that is

Danny Ryan: that is generally the case if execution layer has the data available especially if it's adding on to what the execution layer sees as the canonical chain it will immediately return

oh

Saulius Grigaitis: okay so maybe i misunderstood so so basically but my my understanding was that as it's not clear whether the the call takes a lot of time or not then

uh then basically the general idea is that the response is is a non-blocking and let's say it returns sinking or or whatever and i mean maybe i just misunderstood the specification then so so you are saying that if if i insert the uh the payload and uh and this is uh and the payload is is something that the execution engine can uh immediately uh calculate it uh basically do the transition and say is it valid or not then it immediately returns that is it correct

the the answer that is it valuable

Mikhail Kalinin: let me give a comment on that there are two general cases there is the payload from the economical chain and if you submit a payload from the canonical chain and el doesn't have a parent state or doesn't have a parent block whatever it must return syncing if it does have the current state it must execute it and respond with valid or invalid if this payload is not from the canonical chain there are options that are and these options depends on the client implementation for instance like uh some clients maintain uh several states

so they can execute this payload immediately and respond with valid um if they have a state other clients doesn't does maintain only one state which is the canonical one the one of the most written block and they will just respond with accepted uh and once this payload is or its descendant is signified by the fortress updated as the head of the canonical chain then this then the client must run the execution and the current spec says that it's up to a client implementation to decide whether this call will be blo will be synchronous

and will uh span and the client will spend some time to execute a few blocks if it has these blocks and respond with valid or invalid status or the client decides to go to the network to pull more information or it's like a hundred block worth of execution and in this case client responds with the sinking and goes executing these payloads and verifying this payload in synchronous fashion so this are like these are the changes that are coming with this new release

Saulius Grigaitis: okay so so basically with if correctly implemented at the execution layer

then if execution layer client can do this quickly then it actually does that yes correct

Danny Ryan: right and quickly it's up to client implementation because it's really it's very difficult to specify it correctly so this kind of quality of service stuff so it's really yeah yeah that's but then it's it's okay yeah yeah say in particular if aragon is really fast in execution and there is there is a real work uh which which is like has the three blocks to execute

and their gunmate does it in like sub 100 milliseconds why not to do it like you know to execute them all and respond with valid so that's that's why there is a room uh in this pack left for this kind of

Saulius Grigaitis: um okay so if we okay and if if i get a response that it's uh it's thinking or or whatever basically not not calculating it immediately is there like some some guy that how often should we pause for the result

you're not the idea isn't that you pull

Danny Ryan: the idea is that you then i mean i suppose you could you could

just sit there and wait um but actually execution layer sinking processes often rely on continued information about the head and so the idea is um that you would then insert a descendant and a descendant and you might see syncing syncing syncing syncing until you insert one descendant where the syncing process is complete and you might see valid or invalid and that can validate or invalidate uh that branch that you were unsure of in the meantime this is um the kind of the interplay between the engine api and the optimistic sync spec the optimistic sync spec is in the consensus layer specs if you want to check it out so it's not intended that you

pull on the same [Music] call um if you get syncing

Mikhail Kalinin: yeah actually you may fall by using the box choice updated so you will get the status faster than the next payload uh received from the network but yeah it also depends on the implementation but this implementation is cl client implementation so you want to like define fine-tune this uh this stuff uh so you might probably want to

pull parts

Saulius Grigaitis: yeah but let's say i just get so so this is let's say it's it's a

[Music] station where i get a block via gossip and then i insert the payload of that block and then let's say execution layer says okay there is no immediate response and processing or doing whatever and then i need to do a test on on a safe head and i'm kinda i need i need to ask again the execution layer in order to check uh you know is is is the

is the new payload valid or or not or sure should i should i wait again for a while and ask again

Mikhail Kalinin: the default behavior will be not asking again but just waiting for the next block submit it and get uh get either still thinking if it's sinking or get valid if the thing is done

Saulius Grigaitis: yeah but then i miss miss the possibility to vote on uh on a on a new

safe head

Mikhail Kalinin: yes if you if if you're if you assume that this whole uh stuff will help to recover fast together this information that this thing has finished faster yeah then

it's it's difficult to say right now because we don't have we we haven't seen any clients implementing this new logic

Danny Ryan: to be clear like the the standard path is essentially you're just kind of inserting in lockstep like once el is synced and the consensus layer is synced you add to the head and it says valid or invalid very quickly um the notion of of el having to go into some sort of sync mode is uh at the very least in the normal case on some sort of reorg or potentially like a total reconfiguration of the underlying el software or some of these more exceptional scenarios not in the

case where you are kind of inserting just into the head so um but there are these potential reorg scenarios where if you more aggressively want to get the answer and the information polling could be maybe a reasonable strategy on the subslot interval to try to get updated information i would if i were working on this api that would be a potential optimization to consider um after i got the core in place i don't think it's critical for for the initial um because again like in some very except some exceptional scenarios you might for a slot have kind of a a blurry vision as to what is valid and maybe not a test but

not in a way that would greatly affect a validator in the normal case in my estimation

Saulius Grigaitis: okay so i'll just maybe say how how i would expect this to work so so let's say if if i'm reading validator and it receives a block that's in the first second [Music] slaughter then what i would do i would open the spinner call to the api that waits until the end of the first stick basically of the first four

seconds and i either get a response from the execution layer during that period that the payload is validated so i can update the save header and vote on that either i timeout then i test on the previous save head whatever it is or the another option is i insert insert the payload and then um if i do not get a response immediately uh it says sinking then then i need to pull uh often uh because i want to uh to

vote on a safe head as soon as possible on the news save that as soon as possible so then i have this not so nice or calling which is about every i don't know 100 millisecond or whatever until the end of the first ticket and uh and then i give up and vote on the older head so so this this looks like this from my perspective because at the uh the current proposal is there is no like very clear or this is not i mean optional for for execution layer clients to behave

slightly differently then there is a bit of of uncertainty what is actually uh happening on the execution layer so so that's my idea maybe it makes sense

Danny Ryan: i mean there's certainly uncertainty because there's any number of reasons why the execution layer can't immediately respond to something

Saulius Grigaitis: um yeah but but i but i i understand that they cannot respond it quickly but then i'm left with with the station where i even don't know should i pull it and the hope that the head will update until the deadline of uh

when i need a test or i just should give up because this is at least from from what i hear this is slightly unclear

um it

Danny Ryan: it is unclear um as to how quickly the execution layer might be able to fix itself in an issue because in the two extremes you might be doing a one block reorg um or it might be doing a full sync from genesis and that is not the granularity of the difference between those two is not specified in the call again polling doesn't do any damage so in if you're trying to highly optimize these scenarios

for attestation inclusion i mean it's a viable strategy but um i don't think it's going to buy you much in almost

all most cases okay all right be clear let's end it one more

Saulius Grigaitis: if i can take some time so so if i understand correctly even the latest changes they do not address the attacks that keeps let's say bouncing attack which which makes you [Music] to have a two running or at least two running the

forks in in parallel and it seems this problem is still not sold right as as it's possible to kill the execution layer by by just keep reorganizing it which takes a lot of time or is there a solution for that

Danny Ryan: if a fork choice balance can be done in perpetuity at a deep depth i think that it could short circuit some execution layer clients um depending on the depth and depending on the frequency

um yeah if you understood correctly that the depth is like for gas it's one one two eight

or so so so i'm talking if

Danny Ryan: and a mining cartel could do the same thing to get today as well

Marius van der Wijden: okay so so this there's a there's a small difference like we have 128 reorg depth that we can do super easily so this like we we maintain 128 blocks in uh reorg like like the state for the last 128 blocks in memory and we do maintain the state for the last roughly 90 000 blocks on disk so we have those tri nodes on the disk but if we were to reorg

so we can handle reorg up to 90 000 blocks but that takes a longer time these 128 blocks

Saulius Grigaitis: are instantly reorg let's say 200 blocks on the main net uh what is the real time for that

Marius van der Wijden: i'm not sure but it's like it's a couple of seconds so it's not

okay oh okay okay

Saulius Grigaitis: again it's gonna depend on the execution layer but i i claim that uh such such an attack can be done by any consensus mechanism if there is a uh some sort of cartel or a certain threshold met like you could

do that with proofwork mining um at a 200 block depth but you need to have sufficient you know crypto economic weight to conduct such an

attack okay okay

Danny Ryan: um any other merge related items 20 minutes left okay we don't need to do a deep round of

client updates but if anyone has uh some items they want to share please take this time to do so

paul from lighthouse

Paul Hauner: i don't have much to talk about in terms of um client updates but lots to what's going on but not much to talk about uh but we did have a question from michael he's interested about the status of proposal boost um he's particularly interested in who is running it on prada and if we're waiting on anyone to finish implementation please

Terence Tsao: yeah so um prison here so we have finished the implementation i believe it was merged as of last night we just need to cut a read these so so yeah so just let us know when you

like to see it on private test net

Danny Ryan: yeah i think it's probably good to move in that direction right paul

Paul Hauner: yeah i think all of our nodes are running i'm not sure exactly but mike has been running it on some nodes uh i think teku's been running it as well i think i think he is keen to see it running um on more nodes so that we can see the actual effect of it um so yeah i think he would be keen to see prism running it on prada whenever they're ready

Terence Tsao: got it yeah we we will uh prioritize that

and i can confirm that tekku

Adrian Sutton: has been running it on all our proper nodes for a

while now

loadstar has merged but not run in the notes yet

Danny Ryan: great um other any client updates or any discussion points in general for today okay

we're putting the final touches attempting to put the final touches on um this release there is a couple things standing in the the authentication that needs to be merged and those optimistic sync slight uh return call status uh updates that need to happen uh but that will land very soon then we can uh work on kenzugi youtube okay thank you talk to you all soon take care

thanks everyone thank you [Music]

[Music] [Music] [Music] [Music]

[Applause] [Music] so

[Music] [Music] [Music] you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.