All Core Devs - Consensus #29

Transcript

[Music] [Music] [Music]

[Music] [Music]

[Music] [Music] you

[Music] okay the stream should be transferred over people on YouTube please let's have here is the agenda for today's call issue 108 testing an update releases we have a few things on the pipeline

similar to the two weeks ago one is the removal of signing earth which proto on the last call said he was going to execute and we were gonna get consensus by this call and weather change great I have had some conversations with a handful of you that are overwhelmingly all those were overwhelmingly positive and that it is much cleaner and it cleans up some issues around caching and cleans up some issues around not having to provide zero signatures zero doubt signatures and consensus which had some typing issues and in general positive

but I will not speak for everyone are there is there anyone that is wants to speak out against this as it is a substance of change looking for winter right now so I'm sure that you the plan will be to probably merge it in three hours so speak speak now on that issue here on that issue here if you have initiative

in general I'm I've it's very nice it's very clean makes things a lot better so I'm I'm acutely that's issue 1491 sorry four plus four fourteen ninety one okay another item in the queue for this next and what we will call very near frozen release is the fixed support choice stuff I gave that there was an issue that arose and some of the academic work around formalizing effigy and ghost there's a case you can check out the

issue but we have a fix for it but we have it's actually pork choice stuff is increasingly difficult to test and kind of specify scenarios so we will probably Murchison soon but we're working on a better way to more generically specify complex for choice scenarios this would also be in conjunction with some of the intended for choice tests release had a couple of conversations I'm a little behind on these conversations on what that test format might look like there's it's kind of between unit testing some of these functions that are in the spec

versus more just integration tests throwing a bunch of blocks and add two stations at a client and getting a result in head so that's a priority that we're working on we'll probably get this released without additional work choice s very soon with the removed sending group but poor traces are still suffering one thing that could be done I'm just thinking for multi-threading testing it's very hard also because we have to like test interleaving of four or five phrases in different order and there are ways to do that so maybe we

can reuse the same I can put some blog posts on how it's done we can use the same technique because sometimes you will receive an a decision just before the photo is just after so it's the same scenario interleaving of various inputs yeah down to check out this you have a particularly format which is specifying the size of tests no I don't have a specific format we should decide on abstracting for service or not since

I've become a little easier to probably create many often depending on when there's less state and in like say the process should be irrelevant like right for example you know do you specify this as throwing a bunch of SEC objects in a llamó test file or you specify it as some language or structure that says yes it's a decision so I think easiest would be to have like lots of SSE object and

then test values interleaving and they also used because it can be an exponential explosion of of states they use something called state our strength for addiction and there is a paper on that too prove that what was tested was exhaustive enough the other thing cannot happen and didn't need to be tested so yeah it's better if I I link the blog post to the issue directly can we have the humans in so there's this pull request here not or

we're investigating a testing for that but where it became increasingly painful to write tests the way that we were writing alright and we can take the restless to a conversation either in an issue on the repo or and discord so again those two pr's need to be merged before we release a major release very soon I know some initial test nets will be on the Uni to which I think it's totally fine from my perspective just

make sure that we specify when you're specifying your tests at which which it is there are some the integration of the modified bill aspect with respect to the standard there's a couple of pr's out those are also a priori to get merged and to get out to be built and tested I think the intention is to likely do a release right after New Year's that includes the BLS modifications such that test nets during December will have the old BLS but maybe during December people begin to work on the integration the new BLS and then come to your test nets will

have the new BLS implementations the any any tests net that is long running and gives us signals to move to main net would have to have the correct signature scheme so I don't have I haven't taken a look at these I know some people on my team have so Karl or Justin are these looking to be merged relatively soon are there any unknowns or unexpected things that we have to get through the changes are not major it's basically a then you then you hashed occur version

as mentioned previously my new PR $14.99 that's standing there that basically tries to separate the eath to specs from the BLS specs so you if you are handed a sort of black box you're less library that implements the ITF the ITF spec then these are sort of the wrapper functions you need to make them compatible with with the PI's that the each suspects so it should be classic clean and nothing expected already needed there I'm in the process of changing PI ECC so we can get all our test vectors updated for this and then

hopefully in the next day or two one of those should be should be really cool for merchants as then he says hopefully get that out late this year or yeah that kind of thing yeah so I think we might put it into into doubt but at the same time cut test vectors off of it so that as you'll begin to work on these BLS implementations we can you can generate the vectors you can test my selectors we will keep you informed on s that the bestest till next week so okay say so the big the big thing is figuring

out these tests to test for choice test vectors so I expect lots of conversation I'm not investigators well anything else on testing and releases any questions I don't think we have some from the fishing team in account what they do know is that they're making progress and they reached out like India respect listen like fitra are integrated and I know they're called English maybe mommy can

paper soon enough or maybe too early I don't know sorry there what was the last thing on the thing they're working with you guys yeah I'm fuzzing and net might catch everything but Nimbus as a PR for become fast if that's what you and I think it's pending Mary Mary plans are giving us up they probably every other call and it's also doing some blog posts so but did you know that [Music] cool let's let's start to find updates I

have a Nimbus so we're gonna use a cross Peck we finished sir last SSE bugs last week so now you can use n CLI again to compare Adams against yours regarding optimizations we removed the last vestige of our quadratic behaviour now we reduced OCI time for over 50 minutes to less than 20 minutes and the last bottlenecks that we have are sha-256 and brss but it's not really blocking for having the state

transitions under 6 second on test net we are debugging syncing it works but we have some edge cases and we met a lot of polish on the instructions so and it has been tested from first start and from restarting so it should work much better than the previous week on tests and CI we are in the process of setting up our Jenkins so the goal is to be testing members on control hardware especially on Draenor on our phones and we are no auto tracking on tests especially state transitions to make

sure we don't have any regression also we don't have anymore git LFS live workers or the EF on the phasing product we now have something similar to NCL I called NFS fuzzing endpoint and we have PR pending on CPE become phase repo with shuffling as a proof of concept and for networking we are now ready to integrate people so it does all the building blocks are ready so it can't aggression can be done in the coming weeks to remove the p2p demon um so I guess next

purchase is this v5 thanks Trinity everyone yeah generally a lot of work to get up to spec 0 9 - so it's a lot of fork choice updates and some refactoring on our end we have an open PR that I believe is pretty give us better sort of currency handling overall and then a bunch of various bug fixes stability work as we propose we talked about that great thank you we talked a little bit about this on and

that we can call yesterday but if anyone as you do the aggregation integration if you have feedback comments questions I'd like to hear it just because I haven't really had a conversation people that have yet so I want to see how it's going let's start Hey so we swapped out our pls implementation in the past few weeks with a Wasson build of Hermes BLS and great great results things like 40 X roughly speed-up across the board well yeah we implemented the the BLS vips to

333 through to 335 just as some utility libraries not immediately used but will probably be useful soon we're still updating to 0.9 point that are left like for choice getting the chunked responses I think that's actually 0.8 and the getting aggregation going and our disk p5 is still work in progress so we're still just churning through things right now great thank you there's the awesome

Artemis yeah so we're up to date with zero point I'm going to including naive aggregation implementation in chat with gem about that request responses done we've been testing against prism on lighthouse and I think there might be a few little corner cases we like house but our Adrian will reach out that team to discuss our main focus for this sprint remains jawed being able to join public test nets so it's a lot of work on syncing importing deposit receipts and and that bread-and-butter staff and also finishing off the disk v5

integration I think that's it great Thank You Barry and I will reach out to Prismatic hey yeah hey guys our single clan tester has been running smoothie although we saw a few instants of finality regression and thanks to Danny for helping us with a debugger on that that was super helpful and we all will have a pig in Chains plural Ronnie and thanks to Peter from b-fly are making that happen so with that there was lots of faces and hardening in our RPC server so we can serve people that

want to build that became chains blower or like any cool stuff with it we're primary focus on this 0.92 relaunch so and then remaining ward of aggregation protocol and then the equal one data majority voting and I'm Raul spend all time optimizing SSC by creating a functional cash for the cost for a costume state SSC with that we sell about like 50 times performance improvement during doing syncing and yeah that's about it great thank you yeah I dropped the links of the pockets for it's very exciting

White House Hey yeah we've been pushing for a public test net we want to get it out this week so that's looking like towards the end of the week we look like we're gonna be on track for that but was still kind of squashing bugs and doing a number of different optimizations a lot of optimizations over the last two weeks to being hopefully kind of like database focused we've implemented a schema where we don't duplicate any state and we now kind of have these checkpoints so that our database is pretty small when we when we had an original in our original test that was kind of growing way too far so it's been a real time it's

chronic sorting out our DB we've made some pretty decent gains with as I said decoding we were checking VLS keys we take that out and we can actually with a huge performance gain across the board where were in in order to get our testing it out we have had to kind of we've rebuilt our sinking algorithm as everyone else seems to be doing so that we can support you know multiple peers over a longest longer-lasting test now with a lot of we found that there's a bit of lag and some issues when you're trying to pull out a large amount of blocks from a single node we've tied up with our logging to make it a bit more user friendly for when people come and join this thing our validator onboarding Doc's are all

written it's in a PR we're waiting to merge that we've got metrics and monitoring for the tests and all up and running so that kind of set that's that's all good and just as a PSA we pulled our testing it down today because one of our major boot nodes or some of the nodes booted the up boot or the other nodes often ban them we were being fairly strict with nodes on the network that if anyone gives us anything incorrect or invalid we kicked them forever so we've made up here though that would just need to get most down that I is a little bit more lenient so we read only kick you for thirty seconds or something now yeah that's pretty much the main things over the last two weeks but hopefully should see public test sent from us in either in a few days

great thanks age what was the optimization with respect to the LS and SSE when we were reading state from the database which which happens all the time we were checking the BLS case say we don't go do that checking as in making sure they were verifying them okay yeah so we got that out and so like 99.9% games in in a lot of the reading rate in mine it's also a bottleneck for us in Nimbus and but I didn't make want to make the change because it was too

complex but with the new unsigned or signed blog header it would be much easier to do that that's what nice and the premise being once I've saved these keys to my database I've already verified them so don't do it again the idea is I'm not sure how it's implemented for my house but the idea is only check them just before use so lazily checking them instead of when you load the database for example on reboot doing all the BLS stuff and loading everything in memory and then using them because that makes a great latency drink when starting thank you and I know

Harmony's still working on some slightly and if in networking Artemis to harmony give an update for us yeah we have updated two hours back implementation to 0.9 and point to not including aggregation stuff as this update was done as a part of our work on the focus tests we have like and alex is working on integration tests for the pulp choice and we are almost finished with them and ready to push work on that community dispute and for choice yeah hope it will

happen next week also we have finished this graph I've nursed it first PR and now looking into this crew v simulation stuff and made some progress on gossip stuff simulation and the update was shared during the network call yesterday so it's probably call all the updates that we have great thanks betrayal and parity yes so for us so

updating the subject the Casper consists of the engineering is subject it was thought that there would be a vulnerability in how that consider the injury interact with rent I'm a quick module later I figure out it's just a config issue but India we need to probe for the wording somewhere senior shouldn't configure Chen this way so yeah that's that's for the second part but and we are also we're still and spike 0.9.0 so we are trying to join the currents public prismatic test net disarmed updating the networking module

but currently I'm still facing an issue in the pr i--'s verification which is slightly weird but I'm saying what might be the issue so we pass over there for nigh on zero spec types including the BI signature verification but we seems cannot verify the signature on the prismatic test net so yeah so there's our current progress is the prismatic Testament at oh nine zero turns yet yes I'd I'd feel front a knife on two-year-old

I said checked I think that's correct it's on interesting cool good luck with that thanks way and I noticed Tomas that you all had an update the other day that never mines been doing some due diligence on you do an update for us oh I didn't expect to be I called out but yeah so we started being from implementing the thing so we would be kind of this back but hopefully should have the spec implementation because you're annoying to in a month or so so we have like all the basics this is that being recognized ation BLS is working

fine so you passing the bill this is that all the container casts we have all the containers so we'll continue now at the fork choice and and all the become change specification we also merged with we the team with a guy from Australia called swine yeah so Santa may be a request for invitation here he's been working on the bacon chain implementation already so we have some code that passes some of that we can chain tests and we just merge these two code bases since we sure had some conversations with the guys from security team about discovery v5 so we'll be looking at this as well and there is some team that can work with us

on the p2p implementation so starting but so I think we have some steady pace for now let's deciding thank you okay on to research updates it's Alec you know sorcerer research like some stuff lady is probably going it's just increasing the efficiency of start proving by a lot otherwise and I've been

doing more kind of Apple more kind of a per application level and of course contract fighting really related stuff this this week and I have a blog post that should be coming it should be coming out student on that it's those are those are probably the the main things from my side and also it's talking to and a few researchers about how faced you'll work and they're kind of cross charm de jure that across our transactions and faced you know and whether or whether crush our transactions should be something that's

been joined in the protocol or whether it's something that should be done on basically done with receipts not the application layer kind of the same the same way as it is now and there's this kind of spectrum between whether it should be a yeah like basically what whether you should be a fad protocol or whether it should just be a like I did I feel availability and constantly familiar with a couple of other things on top and there's this and if challenge of figuring out like exactly how much it should include um yeah so right up coming up soon on that do that and I did have a write-up on one way to do the

appearance get cross charting on you know the write-up coming soon on how to I mean to accomplish the things that I was trying to achieve without needing to do that on the phase one side this is somewhere between research inspects were figuring out a plan today on how to best test it and kind of move it forward to get it integrated likely when we release the BLS stuff on around new year will really say at least what's beginning to look like is stable version of that that phase one spec Justin I'm going really

really deep on the snark stuff and actually I I recently did zero knowledge study club presentation on the polynomial commitment schemes which should be very accessible so I'm posting the link in the chat and I've only I only went through I think roughly one-third of my slides so there will be more our presentations in the coming weeks I guess where I talk about polynomial commitment schemes and there's like this fundamental central thing and snarks and once you understand the role they play and this bigger framework like everything becomes so much simpler if

you want to actually understand how stocks work and yeah now that I understand stocks better you know as the tag was saying there's kind of some some new ideas that came out you know from multiple people and and yeah potentially like really or pretty significant break for snark world so hopefully a write-up will be available very soon there's also been you know a pretty significant breakthrough in MPC the MPC so my DD the MPC has successfully has now

successfully invested with 10,000 participants so you know that's kind of that's awesome openly on bias you know couple orders of magnitude and it shows that yeah so the new hero team has done an amazing job there I know we only need a thousand people and I think that's sufficient decentralization it's good to know that it can scale awesome thank you proto did you I know you've been looking into crush our transactions you have anything to share on that or otherwise

are also on the as I said a this is communication 3p I also do spectrum because he's talking about from a very proactive solution solution we really think of it as a data layer indeed and I wish rabbit about very budget and like just thinking about every resource thing as a commitment that we probably want to trade off stomata or I like mine except after thanks let's see

Musab from rent I'm the only updates sure we we just kicked off the the new the next phase of the project which is about building abstractions of the the big chain model that we developed over the last few months and of course the objective of all of this is to to show how safety and lightness can be proved on the change specification so so yeah so we just started this week and I am personally going through the details of

the gasper description and and then that documents great thanks and the area also prevent that verification is doing the last bit of work on the deposit contract and working on a write-up for public review to be released within the month I know many of you out there potentially listening or anxious to see that deployed and this is getting that out for public review is is that last requisite stuff okay

other research updates I'll we quilt help chaperone the first base to it you're gonna be call this week and that was a lot of fun we had some interesting conversations and we're trying to target the next one for a mid January so hopefully we'll see a lot of people from this call there you can see some of those conversations regarding to phase two will is aggregated some questions regarding stateless protocols and put it on each research oh that link here and then I'm working on some tools to improve the process of building and testing execution

environments which has been one of the big pain point for both of us explore in that area and our new hires are still working on the simulation they've been on board it's the simulation it's progressing pretty well great thank you they say you can call other research updates okay next I had networking and then realized that we had to call it up by yesterday I dropped two very good links from then and mommy there are notes from the call that you can if you were not a

part of that call you can check out in general it was a survey of what is going on all the various problems at hand the maybe more interesting component was trying to figure out the problems that aren't getting enough resources currently in terms of research and so we are likely going to do a follow-up follow-up call in about two weeks and I'll post that soon are there any just general updates that people want to share about networking here cool mikail added an agenda item about

discussion of ways to reduce persisted state database size I know that this is kind of independently being explored by a number of teams I know specifically Adrienne lighthouse y'all seems been working on it do you want to give us a critically synopsis on what y'all are doing sorry miss that mikail is interested in sharing and discussing techniques for reducing persistent database size and I know that

you all have y'all been used y'all introduced a hot and cold scheme and curious if you would give us a synopsis on that yeah so we there's two parts throughout well I guess three bucks point is the the hot cold as you said so hot is pretty much anything after finalized things that move so once things are finalized that kind of goes in into the cold database so in in the cold database which are which are things that shouldn't be changing or moved we can we can kind of streamline where they're actually stored on the disk I guess the the main gains that we had is doing an a Nimbus style

approach which we store state snapshots instead of every state and then if you if some if we need to read the state for any reason in the cold database at some stage you rebuild a state or replace that or you buy the state from the blocks that we that we have in the diary so you keep the blocks and checkpoint states we've also added a essentially we don't duplicate any of the states so if there's a state duplication it kind of just read references it but I think the main gains which I think are the easiest wants to do is to do like a staged snapshot in there in the freezer DB and are you doing these like evenly spaced or exponentially every two and for every

eight how are you choosing what's not fair yeah it's configurable at the moment Michael was playing around exactly when to do it so it's obviously a trade-off between if you need to pull data and you've got massively long snapshots then it takes time to actually build the particular state you need to pull I think there was there were some some metrics I can't actually remember the distance between the states but we're measuring the time it would take to fill 32 gig on a hard drive and we worked there for like a decent gap it was about 70 years which was pretty good for well considering we were doing it in six days before we started ago yeah and I have a question and in the hot storage you have all

states for all box imported since finalized one right yeah I ablation because it could be huge even bit not big number while the daters I mean hundreds of megabytes for to represent an epoch and yeah it it it requires some reduction as well from my opinion so Michael from outside has been focusing specifically on on this particular problem so maybe we can have a chat now to this ok cool problem do

you have any suggested techniques just quickly mmm I was thinking about incremental storage but I didn't think much about it so as I'm storing just the dips between states yes yes right yeah but it requires like it designed for it and I didn't think much about the design has anybody gone down that path yet well the tree said I did write up about a few days ago comes recipe close we only really started changes to the Merkle

tree of the state and this is something you choose for the of our two states right and you have lots of parking it's a very short amount of history rest for the part instead I think approach from lighthouse is really do I just said listen outside just make it fly off world right can you link to your discussion of the triste stuff and yeah see yeah I just wanted to mention that the problem is somewhat linked to which states we want to keep available for syncing later on

if we ever want to do state sync across the network any protocol so to speak it's probably useful to times agree a little bit on which states they sort of keep available to others and whether that should be like lost finalized or or some particular week subjectivity parent state or or whatever that ties in a little bit to which states really worth keeping in mind when this when when designing if within a with if within a week subjectivity period and someone can state sync any state that's not a great it's not a great design right is what

you're saying yeah exactly right so I don't know what I'd like two states per year that are blessed or like you don't I really don't want every mod be something exactly okay that actually does bring up a good point and that we do we should very likely specify states we have at least you know a week yeah but even then you can still do block sync you have to like show up with a state route get a proof about a block from that state route and then state sync to

that block and and then block synced to us block synced to that block and then show that it's the state route that you cared about so there are there is a technique in which we don't eat it but it would be come increasingly useful uh-huh well everybody agrees on Genesis states are basically the weak subjectivity period is what it does is it defines in you your Genesis state every now and then easy way to think about it yeah thanks for the update actually Antoine is going to look into states in design in the next month I

guess ok yeah thanks guys for we are updates were there any techniques that people are using or investigating for working with database and state that we're not much okay cool the girls you investigate if you can even bullet point some of the techniques that you've been investigating in people I've been doing they do valuable yeah well do that yeah thanks cool

general spec discussion questions comments concerns thoughts um when are we gonna include the photos you're for phase one into these back proto and I are how come on I'll call right after this to make a plan for phase 1 and phase 1 testing over the next few weeks I think will likely after that call either merge or make last changes on

that but then it needs a bunch of iterative work and testing you the plan will is to release when we release the update pls read the beginning of the new year will probably have this version whatever version of phase 1 is released so that people can take a look at it yeah so that's the general plan but we're gonna discuss a little bit more after yeah I guess I would feel like doing it at least this I gotta pro mean like at least nestled in the kind of the scaffold or phase one is importance to get out there just so people can start building the thing and testing the thing yeah yeah I think there is the fraud

proofs part as well but the fraud groups part is like like if we assume notes are honest and it literally doesn't matter uh so I guess the foot look we're not notes are honest but the point is that it's like something that can be right it's like an add-on to the basic structure to start building things yeah well yeah generally agreed I'm hoping that the team or two can do some R&D on it another thing to think about is given that we're not doing a explicit fee market and face-blind like basically a figure out like what mechanism will you

will use the suggest that people used to pay for getting data included and if is one blocks or vary and option would be it's like so as I see it the meet the primary users their phase one are gonna be like cat pictures and optimistic roll-up yeah you might as well like use optimistic roll-up itself as basically a mechanistic aware to mechanism to provide ease to these and to these validators but that's something that we

might want to think through all of this requiring exposing that state route it turns I don't think we need to make anything more to make a phase one optimistic or friendly because well oh yeah the thing that all right sorry the thing that we do need is we need some way for the youth one chain to learn about the contents of the youth through chain right so basically this stuff we're gonna do anyway to build a bridge oh we also by the way we also need a bridge research team yeah like ideas are there but I would want like by a few

people humming along in the background to and you keep iterating and doing and bring your closer to concreteness yeah I agree I mean come with faith stable phaser appliance we can do all of research and experimentation yeah and the other approach would be it to like implemented youth to clients inside of youth one trollolol do it and do it and optimistic roll up for efficiency yeah sounds like a house made

of cards yeah except it's like the same card that's kind of sitting on top of itself so it's actually secure thanks italic other spec stuff okay next item I think most of you have touched on this and that we're still in the hardening optimizing databases etc

phase to have stable single client test nuts and experimenting you're joining each other substance that's the label and as I see it other comments on test notes what I do want to see is taking some of these single client test sets and scaling them up to very large now they're numbers maybe just using some DevOps scripts and deploying to cloud because I think that's we haven't seen we haven't pushed the limit on those so

if you have the scripts to do so talk to me and we should play okay last item opening discussion closing remarks did I say opening discussion open discussion cool I I know everyone's three heads down and we're gonna ask off so let's keep that

up I'm not let's see two weeks from today is the night so maybe I'll take a look at the calendar it's very close to a time in which people take off but we also a lot of for you so you'll find there a balance and either meet that week or let's go no thank you talk to us soon right [Music]

[Music] [Music] [Music]

[Music] [Music]

[Music] [Music] [Music]

[Music] [Music]

[Music] [Music] you

[Music] [Music]

[Music] [Music]

[Music] [Music]

[Music] [Music]

[Music] [Music] [Music]

[Music] [Music]

[Music] [Music]