All Core Devs - Consensus #46

Transcript

[Music] so [Music] [Music] so

[Music] [Music] [Applause] [Music]

so [Music] [Music] [Music] [Music]

[Music] [Music] okay the stream is transferring over here's the agenda i'll share it with those on youtube let

me know if you can hear me people of the internet on youtube um okay what do we have today uh number one i put natasha um all of you were very intimately aware of um the incident since friday i don't think we need to belabor exactly what happened uh prism has a pretty good has a really good write-up um on what sparked the incident and then that triggered many many more incidents um the really good part of this is that sometimes it might even be hard to

simulate the stresses that we created the forking and craziness so i think all clients are in better shape than they were last week so that's good i know it was also a little painful for us and for the stakers involved a lot of requests to update nodes have happened and i'm uh at this point i'm a little bit i'm sure the uh client operators are a little burned out on getting getting the all call to bump their notes so hopefully we can see a little bit more stability moving forward um in on our to-do list has been to do some client stress tests um on

cloud instances distributed across the world um this is something that i hope to see in the next couple of weeks some people that you have planning to help us um but natasha beat us to it um that said hopefully we can still find some stuff in these external stresses i'm inclined to keep madasha moving forward but we could also certainly debate that uh i i think it's a good thing to have a stressful test now um but there's maybe a balancing trick um [Music] okay that's there's a lot to talk about if we want

to talk about madasha do people have particular points that they want to raise okay um people that are watching this we've been in incredibly active communication so there's not a lot of information exchanged to happen at this point but um i do plan on writing uh a blog post i'm sure ben is going to continue to keep us updated with his posts and clients as well um if you're validating or listening to this thank you

and uh early on in the set and very likely early on at mainnet you're gonna have to keep your ear to the ground you're gonna have to watch what's going on and be pretty active so thank you for updating your notes and getting those releases out um we do have a minor release coming up we're going to officially the solidity contract has been integrated with the repo and that's going to be released i believe there's also two additional gossip uh validations that were added that took two uh very cheap um validations from the fork choice that

were elevated to these gossip conditions to just prevent a couple of uh kind of invalid messages from being able to bounce around the network in terms of testing there is still some work being done on network unit testing i think that there's a few clients integrated into this and we're working on adding some more tests which is nice hopefully we can catch some stuff some of the something that i'm keen to get into there is some blocks by range requests with complicated databases i think we saw some issues on test snap last week

and beyond that testing can be used on phase one um fork choice testing still does not exist in official format although it has been continued to be updated in the specs test repo i know a couple of you are working on using that other testing and release updates yep just the quick fuzzing update that's okay yeah hey everyone um since i gave an update on this call uh what has been happening uh we ran a community fuzzing initiative uh

got a lot of people from the community involved which is good to see so we basically got them to run our lockers that contain fuzzing targets for most uh implementations and um four people from the community identified bugs which is good to see so the latest round of bugs um let's see we've got three bugs on lodestar um including one um memory exhaustion vulnerability that was found on the passing um we found one overflow in the proposer slashing processing function on nimbus and for tekku there was a crash that identified the lack of checks on bodybuilder indices in the attaching

processing function basically triggering an unhandled uh index outbound exception uh we've also made uh great progress on the structural differential fuzzing side of things so we've revamped the differential part of beacon fuzz that we call beacon plus v2 in rust and we managed to identify a consensus-related bug affecting the batch verification of at the stations on prism so basically that particular code path didn't check for empty attestation indices and erase that with the prismatic clubs folks and they were super super quick at producing a fix so well done to everyone um yeah so we're continuing working on

on the structural differential for fuzzing made some really great progress there which is good to see next steps for us will be integrating teku to beacon fuzz v2 so we currently have prism lighthouse and nimbus running um looking at integrating take you hopefully next week we're going to deploy the state transition fuzzers on a dedicated cloud infrastructure hopefully next week or the following week and we'll also start attacking the networking side of things starting from next week with lighthouse and yeah that's pretty much it great um in terms of zero to being complete how would you say those uh structural like mutators are

the stuff that i want to work on after uh integrating teku and deploying the fuzzers is building our own custom mutators so the way it works at the moment is we're basically leveraging this trait in rust called the arbitrary trait that produces smart inputs instructs the fuzzing engines to basically pass a roll bites into proper structs the types that we use in these two so that's great that's been a huge help in getting proper coverage as you can imagine instead of fuzzing fuzzing robots so your next step for us will be to work on custom mutators so basically we're riding our own fuzzing engine so itching uh lib fuzzer honkfuls afl

and the fossils that we've been playing with and basically riding our own um so this this will happen in parallel as in we'll keep running these buzzers i'm sure they keep finding bugs but at the same time i think there's some room um there to write our own so to answer your question um i'd say it's kind of hard to say like i can't i can't see an actual completion um sort of threshold uh it's more of an ongoing process there's a bunch of other targets that i want to dig in as well um but for the structural differential part after integrating will be will be really good yeah cool thank you any questions for ready

uh you have a blog post maybe you can link it in the chat yes thank you mommy do that right now and i know there's a lot of you um in the process of getting security reviews uh some of your auditors may want to run their own fuzzing tests um i guess feel free to you know post them link them to us and more than happy to point them to the work that we've done so that there's no application of effort great thank you other testing updates great okay

let's move on to client updates start with lighthouse uh hey everyone so i won't go over all the madea stuff and the chaos that kind of ensued but mainly the the main improvements we've done to the client since then um well that a lot of the the chaos is going to help us fix and identify so i guess some of the main pictures we've improved is uh sinking from uh having uh such a long finalized slot in the past uh so we've increased like the stability um some of the algorithms in how we do it and and a bit of the efficiency so it's

a little bit more robust um we shifted one at one of the reason one of the cause one of the main kind of issues we had in our client is that if we had lots of blocks or attestations causing a lot of processing we had some processing happening on our core executor which kind of slowed and stopped the entire um the entire process uh and lighthouse itself uh sometimes deadlocked so it shifted a lot of this into like a queuing system and moved it off the core executor so it makes it significantly more stable and run a lot smoother and especially in terms of the networking where we can handle all the messages in a timely fashion and send them all out um we've improved handling forks whilst uh whilst actually running on uh

at the head especially madaya so there's in the day there's a lot of different clients running at different heads and previously we used to try and uh download them from their head all the way back to the previous chains up to a specific rate and continue doing this which caused some issues so we've found a way to uh case the failed ones and prevent that from happening in the future uh we've also in our adventures added support to be able to import prism key stores so people can transfer a little bit easier between clients specifically prism for that one we had some various interop improvements so we saw some issues during this where we had some interrupt issues between us and some other clients

and we've been addressing those uh one of the other things we've kind of been focused on is looking at the attestation inclusion rate uh so i think across all clients it's it's not a hundred percent and ideally you wanna get there it's a little bit difficult to figure out what the actual cause is um but we're looking into that and i'll probably be having a few conversations with everyone about that as we go forwards uh and finally we finished uh implementing gossip sub 1.1 and we're now looking at doing uh kind of simulations to check our implementation and also come up with a set of scoring parameters tailored to the ethereum two network so ideally uh after we come up and design a few of those and stimulate them we can publish them so that everyone else can

can use these uh the network should become a little bit stable if we all use the same scoring parameters they're more effective that way um yeah that's that's about it from us great thank you age um hey everyone um so for update today i'm going to be covering the main issues we ran into on the dash over the past week i can post a link to this mega issue we've been using to track everything if people are interested in more detail so i'll post that in a bit the first big issue we ran into is related to state

regeneration so we had not been persisting any hot states we just keep an in-memory cash and when states drop out we regenerate them as needed by replaying blocks on top of the latest state that we do have in cash the problem was that we didn't have any controls in place to limit this regeneration so we were getting a ton of requests for states which was causing excessive cpu and memory usage in some cases we were regenerating the same state in parallel across multiple threads which doesn't make any sense so to fix this we added a state regeneration queue that limits the number of states being regenerated at any one time

and deduplicates requests for states um relatedly we found a few spots where we should have been checking our cash before regenerating states and we just weren't um so those were easy to fix a little bit embarrassing that they were there um our startup processing was taking way too long to run at startup we were walking through the block tree reprocessing states and blocks to fill up our caches um but it was just taking way too long on madasha it it turns out that skipping this uh skipping this um regeneration at startup works out

okay but we would still sometimes need to regenerate um a state and replay a bunch of blocks which took time so we added some logic to persist hot states periodically this allows us to pull recent states from disks to avoid reprocessing long chains so we're no longer doing this expensive processing at startup and block replay should be capped to a reasonable number of blocks um we found a race condition and our fork choice processing that could corrupt before choice data the race condition occurred when uh we calculated the chain head at the same time as processing and attestation if the ordering events were just right

so that's now fixed um we updated our sync processing um a big fix there was related to how we decide what blocks to download so our sync algorithm has been pretty simple we basically pick appear download blocks from the latest finalized block up to chain head um the problem is that if like the pier drops away or returns an error the next pier that we sync to we start from finalized again so we end up wasting a lot of time not actually downloading new blocks um so we added a step in the sync process where we try and find a common ancestor from which to start our request and so that makes

think a lot more efficient and then finally yesterday when uh the test net did finalize we ran into a deadlock in our block import logic uh which would only happen if the new block that was finalized had dropped out of our cache so that was a good one to find in this fix now um and yeah i think that's all the major updates i have great thank you um and we don't usually ask each other questions after clients but if anyone has questions for age or meredith i think everyone's learning a lot over the past week and solving all the same problems so

any questions um not a question but we did have the same problem like for example the stat regeneration in nimbus and we had to fix it two weeks ago so and i suppose some of the updates we have uh would be useful also for uh others and maybe we can have a small um um hackmd where we share tips and tricks that we found for improving everyone's stability and speed yeah absolutely i think increasing the knowledge share especially after this

week would be excellent if you want to get something started you can share okay uh prism hey guys uh yeah the last five days i think had more improvements than probably the last six months i think uh you know but i think the biggest thing that we that helped us really sink to the chain head and and help save a lot of validators that were running on prism was our assumptions regarding kind of uh peers and and basically determining the best branches from the peers so we had some flawed assumptions around kind of a like fork branches

uh and network partitions for example so refactoring caches a bit to uh change the way they structure uh the cache keys uh helped a lot overall like the peer scoring improvements that we had were saving us significantly basically assumptions about like you know are you requesting from good peers or if the if most of the network is junk are you just getting junk data so that really helped us get get over a lot of the hurdles that we had and yeah so yesterday similar to what teku mentioned that once finality was reached we did have an issue where we had an order of operations in which uh you know we were saving all the all the states uh upon finality but you know that happened uh before we were saving the finalized route so uh you know that that should

have been done uh concurrently so that led to people having to restart their notes given there was a context timeout in that operation that was so expensive um overall yes seems like participation is is you know is feedering on the edge but on the upside now a lot more eager validators people just excited to keep participating um overall the community's been extremely supportive i think that's something that you know we want to highlight everyone just really understands you know like the implications of this uh realizes just how how big of an event it was but also like you know how how amazing it was that uh the teams were able to coordinate and basically reach finality once again um so that's that's something that's been the highlight i guess of

of the last few days um yeah i mean there's a lot of other specific uh you know details about the code that you know we won't get into but overall we're feeling a lot more confident just in the code over the last five days and we did uh i guess i guess like over the last few months so uh does your bug explain that after finality uh where we had uh 67 validators we had only 20 participants yes that's right so basically there was a you know people were regenerating the states uh and that and basically that routine uh ended because it hit a timeout and uh basically we never saved the finalize root the disk so i was leading notes to basically

be in the position where they were stuck and validators couldn't really do much it was fixed by a restart in which the final issue was indeed saved and we deployed a fix to that as well so yeah that's what happened as well a lot of peers were dropped from the network at that time so that's you know that created a lot of chaos but we noticed that it recovered fairly well i think by by the end of the day it was already up to 60 so that was essentially a longer due to the long stretch of non-finality the cleanup operation at finality took much longer than expected and had applications because of that yeah i had to learn unexpected and we should have been uh do saving the finalized route first should have been done concurrently any other questions

one more thing to cover uh so we've been in touch with the cloudflare team so if you guys are aware the you know the rough time bug was kind of what catalyzed this whole situation so essentially you know there was a there was one we're using six times different six different time servers uh for the rough time and one of them was reporting i think a 24 hour offset and uh since uh since the rough time was taking the median of the responses uh basically it was reporting a offset of four hours because we had six servers um and basically uh yeah that that essentially caused all the craziness and the mass slashings and everything you know you know the the team is aware of that and cloudflare is working on changing it to be a median or be more

robust to these sorts of failures uh but yeah since then basically we're not we're no longer uh adjusting the the user's uh system time offset we're just using system time and basically giving them an fyi that their their clock might be off so that's kind of the direction that we're going just still giving them feedback but not adjusting it in their interest gotcha so yeah it was using a mean right and now they're considering changing their protocol using media docker that's right yeah i mean just maybe a quick comment on this if anyone's writing any guides i think like the good behavior for a validator whenever they query any time sources is not to do any automatic adjustment if

there's a big difference to the local time i mean it would be nice if you could automatically adjust sort of clock skews um so if you have a really small one then i think that it's not a good attack vector because like nobody can really like i mean it's very hard to attack but if you only adjust like say a second per day or so it would be detected way before it becomes a problem um but it would so be enough to like stop

because like real-time clocks and computers aren't really that good typically so you don't do any adjustments that's probably also a problem yeah for sure but like for a great tight definition of small yeah my definition of small is probably like sub seconds cool um so yeah rough time is actually stands to be an interesting protocol uh and maybe as a backup or maybe as something that could be

used as secondary information especially because it is you know it's a it's a different protocol from ntp so it is a natural fallback and it is also encrypted so it might not have some of the issues from ntp but my also and my understanding of basic ntp configurations this is based off of conversations with the digit is that you generally are using like a nearby ntp server so even if somebody's attacking that it's hard to attack the entire global network if everyone else is using local tv but there's clearly a can of timing worms that can be opened so something to think about more other

questions for raul or any other comments on time okay thank you all and nimbus hi so like everyone update on my dasha uh we have uh three issues currently uh number one is uh losing peers number two is seeking performance and number three is a very high memory usage so regarding losing peers the issue we have is that when we start nimbus

we subscribe to gossip but when we think we cannot verify and so propagate the latest attestations and blocks and so we are ejected by other clients and we lose connectivity and this also impacts thinking because when we subscribe we add all those blocks that are thousands of epochs in the future uh to our quarantine system and uh this uh create extra processing uh during the sync so we have a pr uh in progress uh the first part has been merged and the second one so that we don't uh subscribe to gossip too early and

we hope that it will significantly uh improve both the stability and the syncing speed of nimbus and then for the high memory usage we had a report of people reaching 12 gigabytes of memory used in nimbus this was due to two aggressive optimizations that we did for epoch cache that caches validators public keys and we had many of them running at the same time so those have been consolidated so that we don't have duplicates and the second thing is uh regarding uh

fork choice and the votes caching so we use a proto array like everyone and we used to use pruning every 256 epochs but in period of non-finality this grows and grows maybe in the epoch 2 5.0 we have hundreds of people without finality and the cash is not pruned so we are changing this uh using uh another optimization that proto lambda proposed two months ago uh by keeping an offset in the data

structures so that you can prune at every finalized epoch without having to wait for too long for the pruning to be worth it so besides falls uh we have also we are also keeping maintenance of the multinet script the one that were used in the old interrupt event and we used that to create small test nets uh for now lighthouse and nimbus to debug specific gossip issues and we are looking to add prism in the future and besides that we are setting up a fallback so that if

we have a critical numbers bug we can change all our validators to other clients so that we do a part of keeping up to the 66.6 of validators great thank you and at least the nodes i've been monitoring nimbus is currently following ahead and reasonably attesting um yes so i think zary can comment on that i just looked at the stats and my note has made the last 10 at the stations in a row

so it looks probably better now congratulations very good um questions for the nimbus team before we move on uh yeah just uh just a point we we don't kick you if you are subscribing to any gossip sub topics uh while you're thinking uh we do the same thing so that might not be the the cause of losing fears our investigation we actually found that usually it is us who disconnect from the other peers uh for relatively serious reasons like we consider the timeouts certain operations take but with this usually us who violates

the time out by spending significant time processing something so we induce the timeouts and then we finalize the pr for misbehaving yeah okay cool thanks other questions for nimbus okay um trinity hi uh the past couple weeks we've been working on rounding out some things with node operation uh so we've added the ability to restore the fork choice context from the

database and we've also made some simplifications to the notes configuration uh at the moment we're working on fixing some things in the network layer so for example getting noise to work with other lit b2p implementations uh we're working on sync performance and we are also in the process of splitting out the f2 components of trinity into a separate recall and that's basically it got it thank you um let's start hey so last few days we put out a load start update medium kind of just discussing um our madash madasha

progress on what happened at genesis um and generally speaking we were not able to sing to the head during this um madosa finality incident i i think just our our sinking algorithms are just uh and the code itself is just um needs we need to have a deeper look at it and so we would love to look at any hackmd that anyone put together but we'll be looking at what some other clients do for sinking just um and probably be looking at a larger refactor that we have had our gossip sub version 1.1 on ice basically ready to go but needing

to be integrated into the js the p2p ecosystem and this week it looks like maybe the end of this week maybe tomorrow or maybe beginning of next week we'll be able to make a cutter release uh in sync with the a new js the p2p release a lot of interrupt tests a lot of just test more tests in general so things are all looking good on that right now uh we fixed uh a bug in disk e5 um that was probably the source of some of our appearing issues where we were sending back nodes at the wrong distance

and moving forward one thing that we're looking at doing is integrating blst uh we're swapping that out for um in our bls library ideally we would have something to switch between uh native code and a wasm-built prc implementation i don't think they have the pc implementation yet but that's they said that that's the direction that they're thinking of going so that's something that we'll be looking at when they put that out and i think that's it press thank you any questions for cayman

okay and another mind you have an update i'll say about resourcing so we finally moved one more senior death full time for for the ethiopia so um i hope to progress much faster so lukasz fukash is now on ethereum too and will stay there for the next few months i guess full time and so far he was working mostly on the uh hum1 deposits and so on so it's just about catching up but uh definitely will be much faster pace now as previously we couldn't really allocate that much time to it too got it thank you okay before we move on to research

updates i wanted to say has been working on a document on kind of the the must-haves and the nice-to-haves um between now and shipping mainnet she shared that i think in internally with my client most client teams seeking feedback um and we'll probably put that on github to just have it in a more public fashion um aphry has a number of points here after would you like to address any of those today oh hey danny i can

quickly read them out but most of them are as i said or organizational things i don't know if this is so it's more about how are we moving forward with uh with ace two from here and with the learnings from um also learnings from medusa and all the other test nets so one thing i notice and i have with me dasha being our main test net right now is that clients should consider working towards some kind of stable track or better release track

for uh the public test nets or upcoming maintenance and have different branches for other feature integration or optimizations um i i don't have any recipe here for the best practice and there are probably many different uh ways to approach this but uh what i what i believe is really important now that we are moving towards a maintenance potential maintenance launch that all client teams need to come up with strategies how to stabilize their code base and how to prevent uh breaking things that used to be working before otherwise we would be stuck in an endless loop of

optimizing and breaking things um what else did i say just open it sorry um so um i mean we always we always had this in mind that in case something goes wrong with with the mainnet that we could just uh basically restart it using the same deposit contract but different again this time and i was thinking a stack step backwards that we just don't call it main at launch but may not candidates

so we we we do like uh a better release uh of of a network launch that can be maintained and that will use real estate but in case something goes terribly wrong you can just say oops and uh everyone who is uh making early uh maintenance deposits should be aware that spending time on maintaining validators could be at their own risk of losing money on infrastructure and infrastructure and time maintaining it but that they will not lose the deposits in case something goes wrong so we can just reuse this deposit so i was just playing around with how to

frame this um what else was it oh yeah and then i thought about what happens if we are beyond what this might be like going too far for this call what what happens if we are like half a year into mainnet and then something happens where a lot of validators get slashed due to a bug in spec or in a client i have no answer for this but i just was thinking about that that we need maybe some stretch strategies to deal with this and last but not least uh i was uh actually surprised uh when um when we learned medusa that that we didn't have

the prison launchpad and uh i would i would encourage all client teams or anyone who is developing the launch pad just deploy it and i know it's an additional operational burden but for the sake of decentralization and in case uh one fails or uh there's one is better than the others or other users it wants to make deposits have certain client or tooling preferences uh why not have have multiple launch pads instead of one official it's that's that's uh i think that's what i wanted to comment today

thank you um and some of that's probably a longer conversation outside of this call but are there any immediate comments or discussion points forever i mean i think like i um yeah we like we we don't really want to encourage people to think that when anything goes wrong and there's a must slashing we'll just turn it back because that just leads to complacency that um like for example with respect to distribution of validators like more clients um so i mean i think it would actually be

better to like raise the stake somewhat and say like well if if there is like a massive bug in one client and lots of people get slashed due to like all of them using the same client then well they shouldn't have done that yeah it's tough balance there um i yeah i mean we do always have the network restart and early phases uh in our back pocket especially because there's no early phase because there's no transfers um the public framing around that is certainly debatable

carl uh do you want to give us a quick update on the status of launch pad and where that's going um yeah so first of all we've i think we've tracked down that bug that caused some people to have double deposits that's very good we are slightly stuck waiting for available dev time but that looks like that's being fixed as we speak so that means we should have movement on that very soon so excited to get moving on that again um yeah so that's that that's all very good in terms of your idea of uh launch pad decentralization are free uh definitely

keen for that um certainly in the long run i think that's a smart move but i think in the shorter run um it may be better just to have clients not have to focus on that and we can maybe settle on some standards for for what needs to be included or not and work from there and rather try focus on having the tuting that people want supported in the launch pad um if it is tuning that they feel like they're missing i think that's probably an easier way of reaching what we need to reach yeah well and short term more than just clients maintaining their own launch pad i expect other entities to probably do so for

example like my ether wallet um or my crypto there's two of them um might very well support flows and things like that in the future um but yeah i don't want to put the that extra requirement on them today agree on that i think i'm thinking the same thoughts in that metrocard this was the one problem there is if yeah watch out for scammer launch pads absolutely watch out for scammer deposit contracts as well um i think maybe the main issue in

promoting that at this point would be i think prison has a pretty sophisticated launch pad whereas others do not so we might see some additional client asymmetries if we go that path just said i just want to make a quick point that there was starting a discussion regarding uh migrating keys to keep clients and how that's very important but also how it's important to migrate the uh the slashable slashing prevention information between clients so i don't know if that's something you want to cover uh just wanted to yeah let's at least throw it out there

being able to migrate clients in the event of emergency is a very probably critical um and being able to do so safely is certainly critical as well um interestingly enough in most scenarios if you turn a client a off and you waited for epochs and you turned client it client b on you wouldn't you wouldn't be uh doing anything slashable but funny enough the incident that happened on friday that had that time skew that property did not hold because client a went into the future client b essentially was in the past so it might be though um there michael sproul put a document out for kind of a minimal interchange format between validator clients

and has done some dev work to put the basis of that in my house if you haven't seen that document and you're working on validated client stuff for your client take a look i'm also encouraging that type of standard to be elevated into like the eip erc zone um to make it a little bit more official like some of the um wallet and key store standards that we've had i think that's a good place for this kind of stuff um i'll share the document after i stop talking um but being able to standardize that and also um putting some effort into documenting how to go from client a to client b is definitely a priority i'm going to

see if we at the ef can spend a little bit of time once some of the interchange formats have been worked on um practicing going from client a to client b and documenting our experience and maybe putting out some standard docs for people to use um if we can agree on some stuff these docs won't be kind of outdated but hopefully we can agree um other thoughts on bc interchange and things um one thing that's tangentially related

um in prison at the moment to what extent is uh or are wallet stores being used isn't jim's jim's proposal v2 um sure so so our our structure is basically we utilize the derivation path where you know we keep the it's account index withdrawal key validating key we don't allow for i guess multiple wallets included in the derivation path um this is the design that we decided to go with this is that kind of what you're wondering uh yeah i'm just trying to so so under the hood you guys are still using

uh or are using wallet stores is that correct then with the um what's the field called next account uh yes yes so yeah we we keep track of the next account that's right um yeah we use that we use that uh we keep that in a json file and we use that to basically configure the you know keep track of the next next account that should be created okay um the reason i ask is over time these standards are sort of drifted together and now they are very close but just used a little bit different decryption methods um because now uh due to unicode support in key stores

um but asides from that and next account i believe those are the only two differences between the standards so i think it's almost at the point where we should just try to collapse them into one sounds good yeah i'll get in touch with you on telegram sound scratch cool reminder i have a hard stop in eight minutes but i can make one of you the host and you can go on um well let's try to move a little bit quickly uh any pressing the interesting research updates i know that's a tough qualifier that's

something really interesting okay i'll keep the fun going in discord and on each research um networking so definitely important agent proto met with some of the protocol labs guys to talk about tuning all these gossip sub v11 params and some of that is going to be maybe some like wise defaults but uh ideally we can get some of these clients actually running in simulation framework and tune these a little bit more continuously yeah yeah i've got i've got one's at one

point to race please um so in the design of these uh scoring parameters uh something we need to discuss comes up which is uh clients subscribing to topics while they're syncing i'm not sure how many people do that specifically on the subnets as well because if we're subscribed to topics while we're syncing we can't verify the messages and so you act as like a sync or a censoring node and you stop propagating messages which which isn't which kind of degrades the network so it would be ideal if we don't subscribe while we're thinking if we don't really need to and if we do need to we need to modify the scoring parameters to allow it otherwise uh nodes that subscribe while they're syncing are going to get

uh bad scores and they're they're mesh in gossip subject it's not gonna it's not gonna work so i guess i'm kind of curious which which clients are subscribing during syncing and to which topics and is it necessary yeah for prism uh we do subscribe to topics during thinking but uh we usually ignore them so uh like from the other ps perspective it would look like we are censoring client um we could uh probably change it to subscribing after but it kind of complicates the logic because like

you have like situations when you need to resync and you again go to unsubscribe and subscribe so it makes that part a bit complicated yeah yes the same issue with us you guys don't don't subscribe to subnets right until you are updating yeah until we are fully saying we want to subscribe to the subnets yeah i can take this offline yeah i just want to talk i'm sorry okay uh the reason i bring it up is just that uh it affects the decision that we we have in the designing of the scoring parameters so i can take this offline and we can deal with that well the other question here is like how do

you decide you're fully synced because that's not something that you can decide actually you can know you have all branches from all of your different reported peers yeah that's about as far as you can get right yeah you can know that something recent on your branch was finalized but that doesn't help you in scenarios like over the weekend yeah so that's another yeah i mean so these things definitely are play like the difference between sinking and not syncing initial sync versus like a chasing some branch that you found

it's definitely blurry and then the other thing is when you're close to to the end like and you might start producing blocks say on a slightly different branch and those blocks might not propagate as well you might be censored and that's a little bit of a death spiral as well if we're too aggressive here yeah yeah i mean missing like if you're if you're close to head and you miss you know uh like three or four slots that's fine like it's not going to be that strict yeah but then we have something like madaya where three or four slots seemed

reasonable but then perhaps in order to recover from an event like madaya automatically uh it might be really hard when when policies are too strict yeah definitely something to keep in mind as we investigate those policies but probably not something we solved today i have a hard stop at three minutes other networking items cool um on the east r d

let's see let's okay um as we discuss some of the v101 stuff um i think some of it might make it into issues but try to use the networking channel on the ethernet discord just so it doesn't get lost in private chats i think a lot of people are going to have a lot to say on some of this stuff um general spec discussion um hello another discussion but a quick update about the deposit contract so thanks to all the solidity of deposit

country contributor the second release of the solidity deposit country has been released and being ported back to the spec repo and the only difference is the metadata of the contract so the buy code content is remain at the same so the first release has already been used for madasha testnet already but if you want to initiate a local testnet or a mini test net please use the latest by code that i am posting

on the chat yup that's the update um and i think it will be the final version before phase zero launch so um if you would like to try it with um for your local test now it would be nice to do just a sanity check for it yeah thank you yeah again the functional bytecode is exactly the same the metadata has been changed a little bit and the formal verification has been rerun on this slightly modified metadata in the

same place okay i gotta run um anything else you all can carry this call on if so okay thank you for being timely um lots of lots of active conversations around madasha please keep the information sharing high everyone's solving a lot of the same problems everyone's hitting slightly different issues that we're all going to eventually hit so reach out talk to each other share information thank you everyone and congratulations on stabilizing talk to you soon

thank you thanks very much thank you [Music] so so

[Music] thank you [Music] [Music] [Music] [Applause]

[Music] you