All Core Devs - Consensus #37

Transcript

[Music] people on YouTube let me know if you can

hear us okay here is the agenda pretty normal let's see testing or lease updates

Danny Ryan: we found a small issue in a couple of tests these actually manifest as like it's gonna be fine on your in because the test should fail but it fails for the wrong reasons and that has opened up a little bit of a can of worms discussing whether we should handle this m.d.

attestation case is kind of an exceptional signature case there's an issue and we're soliciting feedback from client of the miners' on the east to spectra though this we could introduce what would be a breaking change suspect state transition when we bump to the IETF standard if people want to do that if we but we could also handle this exceptional case as it currently is in the spec so please take a look at that issue and let us know what your thoughts are as well thanks on that there are a couple of clarifications and tightened

gossip conditions and the dev spec for networking and I do expect none of this is breaking with respect to be 0:11 one I think sometime in the next week if we accumulate a couple more of these minor networking changes non-breaking networking changes were released a b 0 11 - if you're interested go take a look there's couple PRS that Paul helped us with through something cool that's what's going on my end any other updates on testing

releases small cats so testing tuning has been

improving for others to look at their status issues were and see about optimizing that and there is no - speck is optimized and running reasonably fast at 40 shots per seconds on my toes does not without mid-august are I've heard and aridity within gun also has been efforts and no no fights and together if the rumor tooling the network during earlier it's doing 49 slots per second built by us and 217 slots per second without pillars and this is still all

the thoughts good question for the cast purse decompression of the keys or other velocity musicians and on the student is in use for network testing and we have Lux Lancome I think you can get the introduction and share program some network disk yeah

Lakshman Sankar: hi guys so I'm working on basically creating a network testing suite using rumor the idea is that just like simple like feature and like RPC tests can exist in CI as well as a couple like multi-client tests to check

like that propagation works still the basic infrastructure exists I'll show you the repo and the chat but still pulling together things together I'll probably show up in people's dis cords asking questions about setting up clients

but uh yeah more on that soon

Danny Ryan: other testing updates I did see in the chat that Alex has reopened the fortress s that he been working on and updated

them to 10:1 and but I need to dig into the issue that he opened so take a look there okay any other testing updates great

furnace fish to

Danny Ryan: clients as you may have saw in the chat we're interested in just getting a simple list what the last things you need to get in place to be able to stand up a single client these are 11 tests net and this if you can just get me a list of those things but again tomorrow would be really helpful to help with planning in the next couple

of weeks in terms of like targeting tests sensible to find things in addition to that today if you just have an intuition have like on win you might be able to have these v-0 eleven tests nuts up just let us know but normally it will just do a regular client updates

we can start with prismatic hey guys here yeah so we're v-0 point eleven or one in a deaf closer what the real deposit flow so just pending pending any any other critical bugs like will likely announce of Genesis time and do the restart with the next few days so nothing blocking in that direction so from that since last

update yeah so we have better slashing detection all the way since Genesis we now have proposer slashing detection given the proposer indexing blocks we fully revamped and optimize your initial sink we integrated beacon fuzz from Sigma Prime and added first targets as well for a Peter peace dive we've been finding a lot of really easy low-hanging fruit through fuzz testing so that's been really great and aside from that we also looking for a security audit for prism so we post an RFP a last Friday thanks to you know support from the single crime guys and kind of a feedback from them so yeah that's pretty much it we're just getting ready to launch things in the hopefully within two or three days

where everything will be up great let's start hey so

Cayman Nava: we're still working through the 0:11 changes we just merged our 0.10 branch and a master a few days ago and we are about to emerge in our disk c5 integration so that'll unlock a lot of the all the en are based issues I would expect that we'll probably have a bare-bones something that we can know that we can spin up and start syncing things to probably by the next

next meeting so you

know within a few weeks great Thanksgiving lighthouse everyone yeah so

Adrian Manning: from our end we've updated everything to 11.1 as far as I know everything's sort of there so it finished off all the main net required protocols that we need so we're kind of ready for that we're just doing some internal testing so we're still going to run an internal private test net try and I know any other bugs with that and then we'll also be doing you know a kind of a long style large public mostly client test net for 11.1 we have pretty much all the main

protocols for the spec that we need for main net for other kind of internal things that we've been working on is like kind of a more sophisticated peer management reputation system we start upgrading to rust stable features which is kind of a thing we've been working on for a while and we're looking at testing some of the snappy compression stuff that we've done in gossip sub and the RPC so the RPC stuff we're going to try and test with rumor but I'd be curious to have a look at gossip sub as well

as friendship from great thanks age Tech Oh so the big news dress is where we were

able to sink to lighthouses test net and catch up to you and track the gene hood which is a pretty big milestone for us we've made some general stability and improvements in the last couple of weeks which were crucial and allowing us to catch up in terms of performance we've implemented some optimizations are remarkable implementation we're now able to import about seven to eight blocks a second on average that's with main that config 32k validators we also finished implementing a new rocks TV backed database which also gave us the local performance boost we've done some work to improve attestation aggregation

particularly fix some issues with block creation which we're causing us to failure to include a lot of applications we're just now starting to look at v11 but we should be ready for a V 11 testing it in a couple weeks and that's it for me

Danny Ryan: thanks Meredith Congrats on the sync

exciting Nimbus hi

Mamy Ratsimbazafy: so in terms of course backup dates we all know at the zero point eleven point one we also checked that we are compatible with this yellow generous interrupt state so the state if we don't

put strap from f1 and deposits and we have the proton rate based for choice that is almost ready it's tested it's not integrated into or block pool and attestation for services on benchmarks fronts we added blsm benchmark suits and we will provide mobile benchmark on phones and a raspberry pi soon on fuzzing so we had in a code base a workaround to skip macro proves so that we could further the consensus parts because we had an issue where and this has been fixed and now we don't need to have this skip marker walk

around on the networking front we worked a lot on stability hunted memory leaking and p2p we are still working on some edge cases in block thinking in terms of features finished and large Italy is almost done snappy support is life and on the discovery front so we can discover and be discovered by lighthouse and we have been working on refinements in terms of test net we are currently trying to connect to lighthouse but we found a case where we are in consensus with this

ally but not with lighthouse so this is under investigation

and as a general honor code means we are revamping oh okay thank you never mind you understand and now working on the synchronization and session management mekinese is a 16 single everyone

Alex Stokes: mainly working on node stability of the client and then also

the feel of an updates nothing too exciting on the test net front but definitely should be ready to join in the next couple weeks

Thanks and we saw some time do you want to go this empty yeah well so not much in particular were the we're still working on the models both the the model safety for showing essentially be the theorem bound on what is what is selection and the setting of dynamic validation sets and the K model the

giving an abstraction abstraction of the state transition function of the beacon chain and and linking these two so yeah so we are the work is ongoing and we hope to be able to share this work with everyone within the coming few weeks okay I think every or anyone wants to speak about test nets I know we're kind of in

this interim phase where people are getting up to this test on spec but any updates before share yeah sure I can give some comments on what I can do in the last two weeks I last time I said I was diving to lighthouse setting up a lighthouse based testing it and in the meantime I was able to make Tikku circle eyes to it right now I'm figuring out if I can also at Tikal validators in the meantime I break the chestnuts was due to some network fragmentation I

discovered lost first I lost finality years and I was ability to sear new blocks so I'm in the process of resetting the entire chestnut I'm trying it's kind of it's not kind of long-term go is more like trying to find out what clients are capable of and main roadblock right now for multi-client tests as I said the clients as to working towards different versions of the spec so I was there for I was focusing on clients that still implements a wood version Oh ten spec in a more stable way and therefore I

focused on the lighthouse and Kiku first but I'm looking forward for stable releases of the version 11 one spec in all clients so we can eventually brought this up to more clients yeah yeah and I gave this gives us a name I put my config on github it's called sleazy I think I linked it somewhere someone wants to join this affords just hit me up great thanks every many questions very cool we'll move on to research updates you want to get us started

I can I can chime in here great let's start off cool Wolfram quo alright cool so one may give an update on some phase 2 stuff and so you guys should be hearing some more announcements or releases soon but through Stanford blockchain ECC Paris there was a lot of interesting discussions and developments and so kind of just go you know back for a moment and then what's what's forward from quote so one of the things that we had been working on christening was just general Yee logic execution environments and one of the some of the things that

we're looking at was how do we make the state provider network works work how do we you know how do we change the state dynamics so we were looking at SSA versus DSA to make things a little bit more a little bit more effective and so there's a lot of interesting work that we did around that and also keeping like the core protocol layer light which kind of decouples some of the account logic from from some of the logic and so that kind of gives some the the back period um so one of the things we're trying to figure out how we can support getting to phase 1.5 and a practical system as soon as possible and kind of avoid waiting for another indeterminate waiting period and so you

know we have this full new model ready and so what we've looked at is just trying to be a little bit more practical and have a phased approach to getting a ease and moving that forward and kind of starting off with one of the first chunks and so what we're looking at is initially starting off with a case study and introducing account abstraction into each one and some of the eath 1x efforts that gets us a big chunk of the way towards e's and some of the models around the ease and so we are we were looking at that and then also how to migrate eath one into an SSA format as well from DSA one of the things that

also like the e azam team has been looking at is this concept of ether when X 64 and this is where you have kind of each one and we continue to upgrade and upgrade the functionality eath one running on the 64 shards are running on a determinate number of shards and so we've been supporting some of that research giving working on some data framework so we can access some data and see see how functional some of that that is and so one of the questions would be you know in this phase approach do we have eath 1x 64 or do we create a new execution model from the ground up if it's a little bit too difficult to make some of these upgrades that we want to get to eath 2 to applied eath 1 and so

that that is in progress that that decision would would need to be need to be made so yeah that's what we're at so some of this some of this work around the counter distraction come will answer a lot of these questions and and yeah so just trying to get us to most functional eath two-phase 1.5 as soon as possible otherwise we've also been working on the eve two books so that's been moving along and that should be coming soon as well and trying to get feedback on that that's gonna be going through the Molech down here soon we'll also have that up on it Congrats cool cool thanks cool to that end

Danny Ryan: I posted a write-up on how like the

relationship between eighth to an eighth one client and such a merger or at least with the single chain merger they extended this to support if you're interested in that ongoing work and getting involved in prototyping respecting some of that stuff take a look at that

user research post it is shared cool other research updates a show xrx so we're working on an eighth one e to merge kind of

Joseph Delong: requirements and constraints document and that should be coming shortly but was in progress right

now updated ma throat which is like this this essentially live p2p in a wrapper so that you can kind of like control it the latest spec version and that's part of our our work to build this network monitor called PR KL which I'm told as a finish curse word and so to that end we're also updating our stimulating air further sv5 and as we mentioned before our work choice tests

cool other research updates

Vitalik Buterin: we want to talk about

hashes and polynomial commitments at all and I guess the the the main updates there has basically been that what we've been pushing forward on both the front of figuring out what the benefits would be of our completely replacing Merkel trees with polynomial commitments and we've also worked out that with some of the more recent and a ploink are gonna work up table type techniques and there's me to basically verify hash functions inside of us inside of us in our course start in a very low number of constraints and so

both of those both of those paths are potentially played by um quite viable alternatives to the Merkel trees and potentially even in head of the in the fairly short term like it might even make sense to and every three architect phase one with some of these ideas and my I'm in mind in a couple of months or whenever and have work on that on the part the those kind of other parts of phase one starts there really and need to kick off like the benefits basically are that you get much smaller witnesses you get a raise your codes for free

we're also considering mechanisms where you can have a self verifying from custody so you don't need to have IP reveals origin or challenge games so that's all been moving forward

and they have

Felix Lange: started working on creating simulation as well although the goal there is mostly like to figure out the

algorithm for the topics so this is kind of something for a little bit later and we are pretty much done now getting the grants proposal together for this work and well we submitted I think at some point in soon and and then they can start working on the ground but yeah even right now I mean they've basically been researching a little bit like what could be ways to do the topic index in a sort of like simple to understand way and it's yeah they had some good suggestions already and other than that you've merged you go implementation into the

master branch and I'll just keep working on it

yeah that's exciting thanks Felix

Justin Drake: one update for me is that it's something I've floated a few months ago but basically supranational which is the the main team working on the the vdf project you know the accent help people and so they they're very much into like performance computing and basically they they proactively started implementing a be left three to one library which is performance oriented and kind of as a as

a key key our goal is to be friendly to for more verification and it looks like we will have basically another three-way collaboration to basically develop this this library between supranational different foundation and and protocol labs and so in terms of the performance for what they've updated for what they've implemented so far they there faster than hiromi on every single operation you know likes more performance gains maybe you know and 20

percent faster but kind of the more exciting thing is that there's been kind of new ideas in terms of batch operations so it seems that the Hiromi is mostly interested in kind of performance in in the when you when you do operations one by one but if you do operations in a batch which is better than 482 and then you can get even more more more performance so kind of in in the absolute best case I think potentially this library could be two times faster

then

Mamy Ratsimbazafy: to add to the performance parts I've been discussing with them on a

batch API it's actually once you have the primitives that are fast it's basically used in each update and finish scheme but that's what I proposed to them that is similar to like sha-256 hashing and with those primitives you can easily add any kind of batching on top like batching interleaving checking proof of possessions maybe dealing with public keys and messages that come as a pair or maybe as different arrays and this is very flexible scheme as long as

you and you get the final exponentiation and delight Nilla loop that you keep for the very hand to get all the performance it's I can share my repo my implementation I suppose they will go for something like this or maybe they have new

optimizations on top

Justin Drake: yeah I mean at this point we're trying to incorporate all these ideas and it turns out there's there's a lot of ideas for tim is a shion's here you know because they've talked to various implemented teams including lifehouse and your team and I think what we'll do is we'll you know we'll write down in the contract

you know something that seems whereas as a first attempt but you know having worked with them on the PDF project they're extremely flexible and they'll basic you know they'll basically do what's best and best for us if kind of the requirements change over time thank you I guess and another have a quick update is on the on the vdf project so the the lis hero team now have delivered both the academic paper or it's available publicly on on ePrint and the implementation which is publicly

available on github under the Apache License and so we're basically going to start hopefully soon review security reviews of everything they've built and you know in terms of performance you know they have basically met the performance goals you know for for over a thousand participants in in an MPC which is you know an order of magnitude larger than anything that has been done before in the context of these

embassies like just any other questions for Justin the that the project I think

timelines and is still being sorted out but the the library would likely be usable within the year right Justin and the form verification ongoing audits might extend past that yeah that definitely I mean they've

Justin Drake: actually been working on it for for several months and they already hired someone forth basically full-time to work on this even even before the contract closing so yeah they're very very proactive and yeah I definitely expect the code to be usable this year probably not you know at Genesis but it will be a good hopefully drop-in replacement for

here roomie

points

Danny Ryan: great thank you other research updates

okay next up is networking happy to discuss any items or issues you run into I will gauge this mature after this call and whether there's an appetite for doing a networking dedicated call next week if not we'll certainly do one the following week it just kind of depends on how much it's bubbled up on your stack and things that we want to talk

about but today or their networking things people don't dress great

Danny Ryan: independently working on some api's but at the same time also attempting to conform to the youth - oh api's Rico I think the outlier there is that prismatic has an independent repo written in protobufs that has gotten some traction with some integrations I think others have also been playing with lighthouse API to date some from Bureau suggested to port the protobufs as the canonical spec or at least as a starting

point and generates HTTP api's based upon it there's an ongoing conversation a pretty good conversation going on in this issue about this and it we've also hash this out a few times before it seems like people would rather write in HTTP and migrate to produce rather than vice-versa if the goal is to have a robust HD yeah another thing that I would like to address is the use of hex strings for bytes I saw in some discussion on the Hispanic Rico that that was one of the sticking

points for not wanting to utilize or begin to collaborate on the e2 api's repo so a few things one does the utilization of opiate open API HTTP API is within this repo with CI that checks builds against migrating the protobufs so that we can have good specs in both does that meet our standards and - am I correct in understanding that that was one of the main sticking points are there other sticking points for not bringing prismatic fully into the fold and

collaborating in the single Rico so first question are there issues with using open api's and as someone suggested adding in well one doing due diligence on what it looks like to actually build proto's from open api's and two if that is reasonable adding that into CI to make sure that we're not you know that we can actually have this kind of dual conformance thoughts on that

Adrian Manning: we just had a quick look that you could

convert an open API interpreter bus but didn't actually use it

that's about the extent that I know about it I mean well one of the

Danny Ryan: one of the issues I think on both is that we have types that aren't well represented necessarily on in JSON or photos and another option and I think that one that I have not had positive feedback on but I will throw it out there is to just define everything in s Z and define a standard function to migrate them to either or but that doesn't necessarily get us all the way there and is a certainly a nonconformity

and strange decision

Mamy Ratsimbazafy: maybe this is something important enough to have a call if not if we don't do the networking call next week for example you can do a call on API next week

Danny Ryan: I'm happy to do that that would allow us to also laid out our arguments a little bit more and hash it out

person a person

Preston Van Loon: from phonetic here you've been following a thread I haven't caught up on it this morning yet but we haven't tried to use open API to protobuf I with that said I don't

immediately see any issue with that just about every field in JSON this a string so just so it needs some way to distinguish between this JSON string is unsigned a 64-bit integer and this is a kind of byte array on to your point about texturing versus base64 I would say it's not a complete blocker like it's not possible to for us use extreme it's just that we would have to write a little bit of conversion tooling

and that would just take some time which just kind of been the hurdle that we haven't wondered across yet because if we don't need to commit that time than then we don't want to so all things are possible here it's just a

trade-off fun right so do you know yeah we we exposed to JSON endpoint and your bytes are in basic support yeah they're just that's the default canvassing yeah

Raul Jordan: Danny adding on to that as well open API recommend using basics before to represent binary data strings over over over api's so we've been just sticking to kind of like

the recommended standards under documentation

Danny Ryan: yeah understood and they want that I as I said in the comment I don't really care that much about the bytes format you know and if we can get conformance across clients and change the bytes format that's fine I think it it was a decision made to have conformed into these one but we're not even using the same structure in front of no structure of the API as easily so we can discuss that a little bit more and address it I do think that we have an opportunity to unify the efforts here

and have a you know a better experience for the people that are trying to interact these clients you know I think that almost I think that everyone here has that goal so let's have the concerted effort to take this conversation to an issue and as mommy suggested reconvene on Wednesday to hash out any remaining items does that seem like a good idea

I can create an issue to start that conversation

Danny Ryan: thank you

and [Music]

yeah we'll lead with that there's there's a multi-faceted portion of the conversation I think one is the proper agreed upon structure the agreed upon you know what will be representing the repo how it would be additive how what CI would do the other is there are the api's exist any to api's there are api's that exists in the prismatic fear in the api's repo and there are a few different api's probably that are hanging out and like lighthouse and other clients that users have begun to use a little bit so the second part of the conversation is to have a structured conversation about what we're pulling from these api's to

to bring into a common place I know there's some good stuff in the COS matically code they've had a lot of users touch their stuff and I think there's some good stuff sitting in some other he goes to and there's they're probably doing pretty similar things we just need to kind of conform and agree I think I would just like to add quickly one of our concerns from the in fear aside the reason that we're approaching this with a bit of urgency is there is good work in the f2 api's repo and specifying who make validators API in open API there's nothing in there yes specifying the broader beacon chain API and so because prism already has that you know we thought maybe that could be

a good starting point but ultimately we're happy to support any effort to you know unify the api's and agree

Danny Ryan: agreed so I think the second part of the conversation is expanding these API user api's and probably using a lot of the prismatic stuff as a starting point

what if you did a one-off conversion from the prismatic two forever fragrants where would that land us like would we gain useful insights from that Oh Justin there was talking available yeah

Danny Ryan: certainly valuable you know sometimes tooling like that ends up with unreadable garbage and

sometimes it's nice we can try it out

Jacek Sieka: that could be an action item before the meeting next week

Preston Van Loon: hey just one quick thing on that we have already generated swagger documentation API labs net so you can take a look at how that looks it's not too bad

in my mind from from loading like descriptions comments restraints and all those stuff is just as basic stuff as possible so you're saying that the auto-generated

stuff does miss some of the coordinates everyone yeah basically we separated all the types and everything so it's easier to maintain in the e2 api's and when we generate you just get one blob with all those stuff incorporated so it's hard to like separate types and reuse types and everything

Preston Van Loon: yeah and I think that it doesn't generates any validation so I think you guys have like this is a field of this length validation within open API spec the thing that we're using does not do that

so yeah but the end point should be pretty easy to copy and just

use the existing types instead of like out the generic is one we're gonna

Danny Ryan: open up an issue to hash out a proper unified format we're gonna open up an issue to begin discussing the existing beacon api's that are in the with Matic repo and any that might lie and other clients to find a good set that works well and we will haven't have an issue to discuss what migrating from

photo to open api's looks like and also what generating proto's from open api's looks like and whether there's some sort of tooling things that we want to integrate into CI and very likely on Wednesday we will convene for what is hopefully a relatively short call 30 minutes or so where we can try to hash out the things we've been discussing for the past week at that point thank you Mike this is something that I think has been at the back of our minds and some worries but we haven't had the emphasis is kind of bring it all together

okay moving forward spec discussion any

Danny Ryan: anything pop up in your work on the spec recently I know that's just a very tiny portion your work these days okay a reminder the bug bounty program is out there find some bugs people with the internet find some bugs I know there's been some people digging in be in touch okay of discussion closing remarks anything else people want to just talk about today

nothing at all great

Pooja Ranjan: there is one

request from the cat hurricane this is pooja from the Canada team so there is this survey shared by its an in that is available in the comment section of agenda to collect some basic information about liking frustration fear and suggestions about the current AIP process we would appreciate the developers taking out a couple of minutes to open up a survey and this will help us the EIP improvement group to prioritize and discuss the issues and mitigation plan to improve their current process I understand some of the developers in this call are also participating in the etherium one

directs call and also taking care of a theorem one dot X so it would be super helpful for us if you guys can take out a couple of minutes and provide the and fill up this survey thank you

Danny Ryan: yeah absolutely I just dropped a link in the chat okay great if you can please send me a list of the items left to be able to have a fully conformant be 0:11 single client test net and if you have

any estimate on when you think you're gonna be able to stand up that single client test net I'll be sure not cuz mentor your long-term tests not just being able to stand up a few nodes and chestnut fully conformance to be zero any useful we're trying to keep things moving on live and let's client assent from all right thanks everyone be in touch will plan on having this meeting in two weeks we will have a api's discussion on Wednesday and very likely the following week we will do a networking call but again we'll we'll take that one in stride and we'll address that of our week thanks everyone

right thank you [Music]

[Music] [Music] [Music]

[Music] [Music]

[Music] [Music] [Music] [Music]

[Music]

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.