All Core Devs - Consensus #56

Transcript

[Music] [Music] [Music] [Music]

[Music] [Music] [Music] so

[Music] [Applause] [Music] so

[Music] [Music] [Music] so [Music]

[Music] [Music] okay stream should be transferred over welcome everyone um the

Danny Ryan: agenda is there and we will go ahead and get started we're gonna start with client updates then we're talking about this

um video upgrade uh let's see i have a document somewhere here's a document there's some stuff the top top three things i think we are generally ago then there's a couple of fork choice stuff issues that um specs are being updated to resolve these issues right now these issues must be resolved they're a little bit less for debate but the second of the issue um i would like some engineering input on

complexity of dealing with that fix in the context of uh proto-production prototype implementations beyond that there are a couple of nice to haves we've been playing around with specs haven't found we're looking for the right kind of compromise to see if we can get the stuff cleanly out and that we'd like to make a decision on those next two by two weeks from now but in the meantime would release begin to release test vectors and other things for the features that we had decided on um anyway let's do client updates and then we can talk about anything in there that we'd like to and then

go from there we do have a i guess a research workshop that xiaowei's been organizing um be a mix of some presentations on educational stuff on uh merge and sharding and hopefully a lot of like dynamic discussion engagement this will likely be streamed it will certainly be put on youtube for public consumption at some point cool so let's go ahead and do client updates and we can start with nimbus today

and someone from nimbus ready to do an

update if not i can go to somebody else uh yeah just wait a second mommy's having connection issues he was gonna uh uh that's difficult um let's let's do uh prison and we'll get back to the miss hey guys uh yeah our update for for the last few hours we've been working on uh on the lining to eip3076 the slashing interchange format we're complying with the lighthouse spec tests we're working on the e2 apis implementation just making progress implementing the different endpoints

and improving proposal efficiency and packing at the station and orders of order of magnitude improvement and working on an internal design dock for how we're working towards the first network up upgrade uh scoping out the necessary work and planning we are revamping slashers from the ground up and uh improved and totally revamped local slashing protection aside from that just working on improvement stability uh and just uh trying to harden uh the phase zero uh code base curious on the slashing protections what um what's the kind of fundamentally different in the approach there uh yeah we just used a smarter database schema previously we

had a fairly naive approach that ended up keeping the min and max fans a lot of them in memory and it ended up being just really inefficient especially when you had like thousands of validators on the single um on a single instance like even when you had a thousand validators per beacon it would become unreasonable so we just leveraged the database had a smarter schema indexing and yeah let us down down a better path all right

Danny Ryan: cool thank you and um bobby did you get your mic sorted here let's just we'll do nimbus last

um and maybe it'll be sort of it then let's do lighthouse

hey everyone um so recently we've released a new version 1.1.0 which uh had a lot more sophisticated uh validator monitoring features which has kind of been well received on top of that we've been looking at um some more detailed analysis of attestation performance across all of our nodes and we've found that like late propagation of blocks is kind of causing some of our attestation performance to degrade so we've been optimizing uh the cost of propagation for for block propagation um and we have them fairly flat at about three to fifteen milliseconds which is a significant improvement

um so hopefully that should land on actually some of the improvements to the attrition performance the other thing that's been degrading them is we've been receiving blocks early to the slot so we're adding some caches to uh keep those blocks kind of in memory and then process them when they come uh we've also kind of been looking at start looking into the f1 and f2 merge api so hopefully we should start uh getting an implementation for that in lighthouse in the near future so over the next few weeks we probably should be focusing on uh getting artist station performance to be perfect or as close to perfect as we can get cheers cool thank you age um let's start

seems like cayman is not here so i wasn't expecting but i'll do some update so basically we have a new thing hey you sound a little bit far from the mic uh is there anything you can do to adjust that ah yeah maybe how about now great thank you great thank you yeah so we basically have a new sync and it's supposed to be a lot faster and it does a lot of things in parallel and we are testing it out and we are trying to keep to our two week release cycle so hopefully we'll have a stable client we are also

running a validator on a test net and he's having some hiccups here and there but generally seems fine and we also started doing a lot of like lifeline prototyping we just implemented the the spec and we are doing some changes to the database to allow us to keep like forks from different uh blocks from different forks yeah that's it gotcha thank you marin and taku yeah hey so um

the next uh release was made it's 21.1.1 uh here we added support for block station and we want to exit event streams for the standard uh api um currently the european pr is almost uh ready and we are we are currently working on uh refactoring to be able to support uh upcoming and future upgrades forks um this is probably the most challenging part now uh also we are working on enable enabling gossip scoring

[Music] continuing improvement of sync process and we are also up uh updating to the latest blc library so that's it from us got it thank you and nimbus is anybody ready i saw you on mute mommy but i don't think we can hear you yeah no audio

well perhaps i can give a partial update i mean i will have bring some updates from the top of my head probably i'll miss something uh the most important thing we did this week is that we've integrated batch verification of bos signatures and this brings some significant performance advantages we will also introducing more multi-threading in this area i will be trying to gather a large number at the stations before verifying them and then using both the algorithm by clarification and multithreading

to make some gains uh we've also solved an important issue which resulted in very long processings of blocks with deposits which rendered the client a bit unresponsive for short periods of time which somehow sometimes resulted in misations there was and now we also solved an issue with our 81 monitoring which manifests when the web3 provider replies with incomplete number of reported deposits which sometimes resulted in our client getting

into an invalid state that required a restart to be fixed [Music] and we are also experimenting uh with we did some quite a lot of work on pr management in lip p2p but this is a bit ongoing

Danny Ryan: got it thank you okay um moving on so i expect this conversation probably to

be ongoing over many weeks uh but that document was shared um based on your team's review is there anything that people would like to discuss today what we can do so we can do a more asynchronous engagement is move most of the structure of that document into more of like a checklist

format uh onto the spec repo for asynchronous discussion

and

Vitalik Buterin: i guess while we're here do people have either more general or more specific thoughts of any kind on the idea of doing uh well you know and the purple was always doing this kind of um hard fork at uh roughly the time that we've been suggesting

um yeah i mean

Danny Ryan: broadly timeline being this initial upgrade mid year

summer um focusing and then production test nets of one of the major upgrades charting or merge by end of year

Jacek Sieka: um i guess those are the two big things we expect maybe maybe the better question is like if we want to do this hard work when do we start upgrading the test net like what would the timeline what what kind of how much time do we need to test this before we were comfortable releasing it

a maintenance yeah

Danny Ryan: good question um i would presume a minimum of one month lead time on like

transitioning our major test net uh but to see test nets uh local test nets and other testing certainly a month prior to that

right like i think part of the reason like well the reason why we've been just asking and pushing for people's opinions on this is because we really want to get to hard agreement on the contents of the hard fork asap um like if we can get total consensus either on all but one of the features or even all of them within and again two weeks that would be amazing um and

like one monthly time between the main tested and the main definitely seems kind of roughly standard for uh especially given that things on uh on eth1 usually have like either roughly that amount of time or so or slightly less the but then like smaller test um test nets should i should probably just start happening like i'm as soon as it's uh it gets developed by each client

Terence Tsao: um when do we need to come to consensus on the bonus um section is would you say that two weeks is a fine time frame

Vitalik Buterin: good question um so part of the reason why

i called it the bonus section in that document is because like i know that those items are kind of a bit deeper than the other items and so it might take longer uh so if like in two weeks we get consult like uh against that just on everything except for the bonus section that's then that's fine um especially since some like developments like if that happens then then what that means is that people that client teams should just start developing building um and even testing everything outside the bonus section and then now you can fill like add in the bonus section one and basically do it last um so it is it is okay if

coming to a final decision on the on the bonnet section takes a bit longer than the other parts

i would

Danny Ryan: say in terms of having a concrete direction for those items um i'd like to have in two weeks time either we think this is a reasonable path on specs and would like input and review or you know we don't really have a reasonable path in specs and want to drop them so like at least in two weeks time even if we're not

making a decision we're in a place where we can like at least evaluate yeah and evaluate and uh like

ideally com if we if we can uh then choose a path like either don't choose choose don't do it or do option one or two well

Vitalik Buterin: for each individual item either don't do it or do it as uh as written or do it some other way um but like basically the the more we can decide on the better uh but there's obviously the possibility that like there would be some deta some small detail that we would just decide which um should be changed after it gets implemented a couple of times and we just learned that it makes more sense that way and there's and that's unavoidable anyway um one more just kind of quick warning on

the um on the bonus section by the way for those who um have been following that document closely i did um come up with a new version of the uh empty epoch uh processing improvements that's that got literally turned into a pr over the um over the last two days uh so this is it takes a different approach it basically does not try to do any of the quotient uh or based stuff it just uh basically makes uh amortizes penalties only processes than once every 64 ebox instead of every epoch and it does and it does have its own trade-offs um

so just like you know but when you're reading through things watch out for that aside from the bonus none of i don't think any of the other sections have really been significantly touched for quite a few weeks now

okay anything else we want to discuss here today and just uh on the on the age on the hard fork one documents topic and

we are expecting uh to have uh people we are expecting to come as close as we can to making uh to making decisions next time around so hope we can all be prepared for the discussion in two weeks and like before two weeks as well obviously the asynchronous channels exist and i'm happy to answer any questions on those or uh on any of those channels great

Danny Ryan: okay moving on um research updates

i believe it is on february 2nd yes on tuesday um we are having a research workshop as i mentioned um show away has been organizing that and i believe should have gotten in touch with the teams here and many other teams that will be available on the internet for public consumption i don't think we have much to discuss here um shall we do you have anything that you want to add here uh yeah just i think i will send the um uh another announcement to the

attendees uh by this week and uh we will share the youtube link uh later the release week or monday i think

yep okay uh knowing that we have workshop coming up on tuesday is anything else people want to share today about research updates um so dan grad has been doing you know he's around but he i'm he's been doing the primary work um work on this

i've been helping a bit uh a implementation of virgo trees though that's probably realistically more relevant for eth1 developers than these two developers um i can't think of anything on the e3 side for myself if vertical trees are imminently viable would you don could suggest migration to vertical trees instead of the binary merkle tree right yeah basically like

for storing stage it's would be much more efficient at least from the um from the witness size point of view to use the trees because you always get much fewer levels each levels need slightly more effort to improve but we've got sheer levels [Music] yes mostly irrelevant statusness at the moment but the catholic commitments we're using um are obviously relevant for you and i'll be giving you an introduction for the workshop on tuesday

and just for everyone like if you are if you can or if you need refreshing one stuff like finite fields and basic elliptic curve stuff like basically pairings yeah do you have

Danny Ryan: any particular resources you'd want to share if people wanted to get caught up on the basics

um yes sure i could do that thank you there is uh dan grad's own

blog post on godzilla communicates which i think is linked to in my most recent blog post a couple of times if you just like go to vitalik.ca click the top link and then scroll down wait at the bottom of that you'll find that that's one of the things [Music] got it about qatar commitments so people have shown interest in implementing this now and maybe it's a good idea to start some kind of group or like share our findings as we implement this um i know mommy from nimbus is working on their implementation and then i have my own duncan can

probably help or is interested and then i know of one other outsider who's looking into implementing this and then anyone else here in the call or maybe outside of the car if you're interested please reach out to me on discord great okay other research updates yeah quick update uh from the merge side

we have started to work on the specs there is like an initial draft and after some rounds of review it will turn into the full request against we great thank you uh we have been integrating the last uh updates on rumor uh on the cro on the crawler and um we managed to solve the problem uh with the with ports that we were having previously um

so i think now we get a more accurate vision of the of the network i just posted a figure on on the discord um and yeah so we are moving forward on on that aspect that's it cool

Danny Ryan: thanks leo other updates okay great um and networking items does anybody have any comments questions thoughts issues

um

i'm looking at that blocks by range request that we still need to resolve and i'll try to resolve that today uh is there anything else you want to talk about okay general spec discussion uh i could mention one thing

so we were playing around with some optimizations in this pack and um there are these functions that uh that are executed for for the block right process block and everything under it um so we noticed in the spec tests that [Music] if you run the test as the spec function is written it works but then the given pre-states for the tests are actually invalid um if you work with the preconditions that are established in the

in the order of operations in process block so for example the proposer index it's always going to be in in the header which means that instead of calling you know get become proposer index which is kind of slow even if you cache it then you could potentially just use the value in the state and that will be nice and fast um so i'm just kind of curious if there's any appetite for fixing this stuff for phase zero or do we just leave it as this

Danny Ryan: um i mean if you want to point me to a few of them i can see if they are like the discrete ones that you point out i can see if they're easy to fix um and we'll be happy to um we're talking about like a disparity

between get prop block proposer from state versus getting the index from the header

yeah

Jacek Sieka: exactly it's like it's as if the header wasn't updated as if you know um right i think the function is called process block better or something like this hadn't been run uh during the test generation

nice because like oh

Danny Ryan: because it's actually in the header that's sitting in the state rather than in the block

yeah exactly so so like the test is kind of valid but that's all but it gives a state that could never like it could never happen in in in

reality right it's a little bit of similar issue to fuzzing like like the the state transit like the logic can go into the state can have certain values that will never happen due to other logic in the state transition function like what do we do about those we're tolerant against right now but

Danny Ryan: um is this because you want to have stricter validations on data or is this because there are certain optimization paths that you can't take because the way testing might have forced you to use certain uh sources of data

and things uh well it's the latter really but it's also a statement on like how

valid is a test that doesn't have a valid incoming state like you could ask yourself that question perhaps

Danny Ryan: right i guess if we're testing a function we're testing a function yeah you can make an argument that it doesn't matter but i see i see the the value there i look i mean if you want to as you find these uh write them down i'm happy to see if i can fix the test easily um you know there's a lot of different tests and types of tests in the way that they're kind of generated and some of them might be easier to fix than others

yeah i mean because like even the

optimization it's kind of minor ish it's a dumb little look up you can say but you know look up there's a lookup one yes and you're processing lots of locks it kind of adds up so again i'm happy to clean them up but i i don't have like the i don't have a a single fix for the entire test suite um to just make it all coherent so i'd have to just know which ones to look at

Jacek Sieka: i'll open up the first one that i found

because i can't really progress or you know whatever we'll see how it goes

Danny Ryan: okay anything else today spec

or otherwise great um we will have a long fun hangout on tuesday so i'll talk to you all then and probably talk to you on the internet otherwise take care bye

guys all right so everyone bye guys cheers [Music] [Music]

[Music] [Music] [Music] [Applause]

[Music] you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.