PeerDAS Breakout #10

Transcript

e e with e

hello hello everyone seems to be

working yep okay great um yes I away again today um so um um yeah I think we are ready to start um most people on the call now um yeah I can quickly start with um updates from Lighthouse um in the past two weeks we haven't really made a lot of progress in P we um the lighthouse node has fored from um um the main um the main for uh the main chain and we've been looking into um the issues and it seems to be related to um how we um do block lookups and uh it seems to struggle a little bit when there are multiple Forks going on

so um we've been looking into it but we haven't haven't made too much Pro progress um yeah hopefully we can get it to sync again with the network um but yeah I think I think I think leaving the devet running is good for now um because we try to try to see if we can sync um our fixes um to the to the rest of the network um other than that we've also been working on the The Gap blobs V1 stuff so less less related to P but um yeah that's um currently where our focus is hopefully get the the the get Lely one stuff out first and then and then we we'll get back to um focusing on P defet um yeah that's pretty much all from Lighthouse uh but we'll pass it to

maybe prism hey hey everyone uh yes during the last two weeks uh yes we work on several stuff first one is um when running a data col by data col by run request against PRM full n sometimes this n does not return the colums greatly it was a bug which was notied by the te team uh there is no issue for super node only from for PR full nod and so we work on the fix on this bug um also we work on optimization during the sync um before for example if

I don't know and prism nde needed um colon five for Block 10 and colum seven for Block let's say 15 so the presis N was requesting colum five and seven for uh blocks 10 11 12 13 145 which is like a lot of waste of resources from the perer we use and for the overall Network bandwidth and we work on an Al algorithm to really Quest by range only columns we really need also uh currently PRM is only able to think about data colums uh from Pierce which

custody a superet of our own cust colons basically it means that prism prism node is only able to F colons for super nodes which is an issue and we work on this now PR is able to think for every n and also we have a sometimes an issue during the sync uh when where at the end of the syn especially if there is a lot of different Forks um prism node could be stuck on the bad fork and so could be unable to to um to to finish the synchronization and so we work on this uh that's pretty it and yes about

the devet it seems that that all the nine um all the prism n are currently up running and and proposing blocks and that's all for me uh next one I don't know if te is here or any any other client hey I can go for so um so I I don't really know the reason where the folks happened and maybe uh but uh I know that once the FK happened and some of the ladar nodes

fell out of syn uh because they restarted or whatever uh they couldn't catch up with the fork that was uh up to the up to the slot head and uh so I debugged and figured out the reason for that was that loar was for example taking blocks from one fork uh which was not syn up till head and then it was trying to get the data from the other Fork because the first Fork itself might not have the data and probably that was why it was stuck in the first place and uh so uh even with the lar restart and drain saying lar was not

catching up uh and that was particularly the reason for that uh although I think we have in our uh in our um logic of R sync that basically we don't group The peers we separate keep the peers separate for different folks but I need to know what is going on because uh because it should not have happen that uh it fetches blocks from uh from one fork and in the next round basically when the pier shoing happens then it picks up the Pier from the other Fork so I need to deback why that H why that is happening uh but I think that is the

reason why none of the restarted lar nodes were able to sort of sync up even after the data wipe and yeah so basically this was a particular reason for it so saying that um low star low Star notes are able to sink from scratch to head so in for example uh so LSA note should have been able to screen from from because there was one fork I think in which there was also one lar node one Lighthouse node or I don't remember which other no so there were two three nodes which was still

running unfinalized and still basically those were the notes which were to the head of the slot and I tried to make Lota basically think of that uh but again this problem was come occurring where it was not able to uh sort of final sort of fetch a batch properly download a batch properly because that is what it was doing uh and I did verify that uh the nodes there was there were few super super nodes in the sync fork and I did verify that they had the data uh that was needed to sync uh so that is what I figured out was the reason that uh it was picking blocks from the

unsynced fork and then it was and obviously it didn't have the data so it was then trying to get the data from the sync for PE which obviously did not have it because it might because it was a rain sync and obviously they shouldn't be serving it anyway so all that was correct Behavior Uh but yes so the the bug in the lar is that it should have the when it goes for reshuffling and to get data for example if didn't whatever data it didn't get in the previous round it should not select the Pier from the other fork and uh that is something I need to debug because as such uh by default it should not be

selecting PS from the other fault I see um yeah thanks ginda um who else want to go um maybe sius yes hello so H from fellowship program was working on on pidas he he did the Reconstruction because we didn't have uh that before however seems there is still some some bugs that uh prevents us even uh with syncing the the pet 3 uh Network so uh I mean from from

Genesis so uh yeah I think we will keep uh debugging uh so we not not very ready uh at the moment that's all thank you Solus um have we got any other client teams on the call nimus teu anyone else no doesn't seem like we have them um yeah um then guess we move to um Dev updates um do we have anything from

devops not really I mean network is broken I'm happy to roll out any new images or you know uh deal the database and try to sync I don't think we have any update from our side though cool um yeah before uh we move on is there anything else that anyone wants to talk about regarding the de net there's a proposal of probably relaunching with only super nodes I'm not sure if that's something that we want to

consider does any anyone have any um opinions um we've kind of mention ours on the on the Discord chat not sure if anyone else wants to have any thoughts about um running a super nut only Network I mean I would basically like to wait a week so that I can sort of figure out the bug uh otherwise I'm fine yeah the relaunch wouldn't be like on this week if they do relaunch it it would probably be next week or even the week after it would be very good to get

this CH back to finalizing so before do that I think as I said on the on the Discord so there is one third of super n in total on the network and currently there is one third of s superod as well on the network so like it seems that being a super n not super n does not really have to think currently on the devet so I'm not sure that running a full super network will help I mean I I think one thing about

that is that to begin with the fact that the networks breaks into pieces I mean it might not be the case this time but it can always happen also because of the fact that there are full nodes like especially if we it it's like a relatively small devet and it's 2/3 full nodes which I think on on you know in the actual validator set that would never really happen um it seems like just making things like un unnecessarily hard to me like yeah it's you know it doesn't seem like in this case super nodes are doing any better but on the other end it's you can't really know that the reason that the more complicated situation happen has nothing to do with the fact that there's a bunch

of full nodes um at least that's my yeah I mean I I'm not I'm not involved in the buing the deet so definitely not going to impose this opinion uh but um yeah it seems unlikely to me that that it wouldn't be easier to to just get a a Dev Network in with with super nodes like that should really look a lot more like 4844 basically um and like yeah it feels like it should be uh it should be like a good first step like once you have that working then you know that most other problems should be because of a partial availability um as opposed to always wondering like is this because of partial availability or is this just some kind of uh you know random bug that

um is doesn't really have anything to do with the with Pier dust actually I think that's a good point and possibly we could do that for the next St not to reduce the scope reduce the doesn't goope at yeah um yeah I agree that um like using full notes kind of make everything a bit harder um but I also think that we should probably wait um uh for a bit to

debug the current definite issues and see what they are and then maybe when we do the next launch we could consider having more Val weight in the in the supernotes um and if we need to test full noes we can always like run full noes locally for testing um it feels like it's it's probably good to to give that a try because because having uh not only Network on us small PS like small no set it's a bit difficult to really figure out what's happening um it's hard to find PE in the right columns um if we don't rely on the super nodes and especially when we have 4 it get even worse

um any more thoughts on this topic it seems like we we just need like a bit more time to debug the current defet now and see what what we want to do with the next dead net but it seems like a like good suggestion to consider for the next dead net cool um then uh maybe we can move on to next topic um is there um yeah there's

nothing on the agenda um anyone like um so just wants to check if anyone wants to have any discussion regarding spec there is a discussion topic last week about uh the Reconstruction the speed of reconstruction and whether uh clients should reconstruct uh only finalized EPO or everything when they are forward sying from Genesis I think this came from the fact that prison was uh thinking I don't know

one slot per every second or something like that due to the kcg uh commitment completion overhead yeah unless I'm mistaken it was not about for construction it was about uh ver verifying the proof thej proof during the during yeah yeah yeah so good points so yeah um so ctive prism has two let's say two issues uh first one is yeah regardless the block is finalized or not uh during the sync we do run the ver all the all the verification including Kaz verification which take really a lot of time and we and the majority of time

spent uh during the the thing is is actually spent into the verification also because we don't um we don't verify KJ Pro by batch but second sh which is a yes you know we we lose a lot of time uh in this so it's a bug we have to to fix but also yes the question is do we really have to to verify this K proofs um um for finalized block because they are already finalized and so normally they are already um safe so the ideas was to one thinking was to not verifying proof for final

block and verifying proof only for um block which are not yet finalized I don't know U what other clients do uh I not for example lious do you veryify proof for fin block or or not uh yes we do at the moment um but I haven't really noticed any any issues with the verification time I I have looked at the ver verification time myself the single verification time looks okay but I haven't looked at like

what's how long it takes um like on average for a block so it maybe something that we can look at next um yeah but curiously here if any other clients have any issues with verification times during sync um just curious when you were verifying proof during s how how are you calling the verifi bat function like how many like how many proofs do you verify per Co per verification function yeah yeah yeah it's one prism we we we do not batch the verification we we do it uh basically block block um block by block which take a long time

while we should be able to uh batch uh m um sorry we do verification column by column yeah uh and we should be able to batch uh verification to to to and it should really improve our our verification time so it's it's one issue we have in prism okay that that makes sense I'm wondering okay so you're doing column by column you can upgrade to doing need uh verifying the entire Block in one call that would be the next step I'm not sure if there is a step after that where you

verify multiple blocks in one goal you know what I mean like maybe maybe you can pass multiples blocks at the same time in the function yeah I have to see I have to check yeah but um I think back to the question whether we really need to verify um yeah I don't I'm not sure but I feel like it would be best if we verify and not rely on the majority um even though it's finalized like we that's what we do with blocks so I feel like we should also verify the the um

the kg if you're someone that doesn't have the whole data I think you definitely should verify because otherwise you you can't know that the like if you just download a single column you have to know that it actually matches the the commitments so I think it's like really like a must to to verify even if it is finalized um if you have the whole data I think in principle you could not verify the proofs and just kind of check the um like you know you take a whole blob and you could just compute the commitment and check that it matches the original one or something like that and maybe it's faster than than verifying all the proofs like you could just kind of forget about the proofs and just

check equivalence uh or like correspondence with the with the commitments um and then you could only maybe check the proofs if you actually if someone requests like if someone wants the column from you you need to send it to them you could check the proof uh just before doing that but I don't know this might just be like more complex than than just checking the proof and like I'm not sure if it's that much faster to to to do it this way but that would uh require you to backfill all the blobs no um do clients not uh eventually get blobs for the whole period like the whole two

weeks or whatever period even if uh like even for parts that are finalized um I think we do um I think I thought that's the requirement to sync back to 18 days but you could start uh well dating even without that no oh yeah yeah that's right but but we have to serve the data uh if we can so so so so so so could you not do the verification based

on request as in um like when we request column from PI do we not do verification was that the question Yeah so basically you would only do the verification if someone request that column from you instead of verifying while you thinking on pris we only verify when we receive the column if we run a colum by R request or colon by root request before storing the the colon in the store we we run the verification once it done we never run the verification again yes the same for light

house is it a big deal then that it 25 seconds or like a what was it 25 seconds per I think yeah not actually yeah actually I think if I run a verification by batch and s in the verification linearly like I I do it today I think it will really solve the issue um it will take I don't know I have to do the MK but something like two seconds is of 25 and so I think we can consider there is no no problem problem anymore and so yes we can continue to verify for finalized block it w be an issue especially if as you said in in the

Discord a lot of people will probably think from the checkpoint Sy and so so the Sy won't won't last long hey Manu um when you say 2 second um how many block is that sorry um for the two second time that you mentioned like how many blocks is um how many no I just I just said two seconds like that it's not I I don't know um I just know we have um I just know for let's say something like 64 blocks we

had I had 25 second of and and so I hope that for 64 blocks we have we can have two seconds of verification but yeah really don't don't take this figure for scientific publication I don't know okay I see uh um yeah so maybe it's worth for us to test out the optimization and see like how fast we can get the verification verification to when we do bashing um yeah we currently I don't know if we do p bashing um on rench sync I don't think we do at the moment um so that's something we can optimize

for actually Justin just did some napkin Mass uh so he said yeah he said so yeah for 1,000 192 colons so uh he got 4 verifying colon by colon he got 40 seconds on his computer and after bch it takes only three three seconds between three and six between three and 4 seconds so it's quite a huge a Time 10 Improvement

using batching nice cool um then I guess we um that seems like a good next step to to try to like implement the batching the optimization on other clients as well um I guess next topic we could talk about metrics Kya yeah hello everyone um um I would like to proceed with this um PR uh I will share in the chat uh so I've splitted uh the big PR into small parts so the first part is more about the D

process verification reconstruction and um I feel like we are more or less agree on this part um do you have any I would like to to share the screen and uh make a short demo of how it works uh uh here is the link for the dashboard if the clients have um the access to the dashboard can you see my screen yep so uh currently the spirit us specs dashboard is um here in my folder but I

hope maybe we will move it into consensus clients later so uh what we have now we have um we have filters uh where we can use super nodes or full nodes or both and we can choose um the client as well um and even the node uh separately um for for now uh we have uh for example how uh how can we use this dashboard this is Lighthouse and we can see for um this group of metrics uh that have some issues uh I mean you don't

only use uh logs uh I think it's it's not easy to use logs because uh logs contain a lot of messages uh but here you can see um strictly to the point uh what issue you have for example for Lighthouse now um we count um the requests before gossip verification here and here we can see that uh there are some issues with verification and we see the failures as well and you can easily compare um your metrics with other clients for example Lighthouse and Teo if

we do here oh no sorry not but few so these are for supernotes um and you can filter them and you can oh here here today Teo has issues for uh for this metrix um let's check full notes oh no as well so for example you can uh compare your values with the values of other clients and sometimes it

can be helpful for example grandine which is in the building mode actively building mode um it uses uh so um I would like to hear from you uh what do you think of this group of metrics uh they are about reconstruction computation inclusion proof um if you have um any opinions or we can maybe merge already and [Music] Implement um yeah I think lion mentions um a suggestion last time to um to look at the metrics and cons and consider like which ones are necessary to be part

of the spec and which one are just uh nice to have in individual clients um cuz um yeah I think the more standard metrics we have um it will be harder for clients to to to change things so it's it's a bit more rigid so um I kind of want to figure out whether we need all the metrics are all of them going to be useful for every client or and is it is it worth comparing these metrics between clients if it is then then then sure we should include them the more uh but just trying to figure out whether these make sense for all the clients for example like the processing request total I don't know if that's um something that the all the client track things this way

um because I I think I think this is a bit closer to how we do it in Lighthouse I guess maybe other clients do it similarly as well um at least in I implemented it for te yeah we have the same now uh but probably for full noes it depends on uh how many um site car is your uh custody and uh another note if you implement um the metric with these names you will see them on the dashboard at

once and uh another one I would like to sorry I just skip skip from the topic uh the that currently curtois also has this dashboard uh so if you just uh want to check and run it locally uh yeah you can find it it is named beas metrix so you will find it easier you launch grafana cores I guess maybe now this has broken

down into individual PRS maybe we could try to look in at least the first PR um this one you have on the screen and maybe we can maybe all the client teams can have a look and see if it's if everything's okay because I think K has been um raising talking about this PR few times I think last time we had this call was quite big and we decided to move out the P2P and the gossip stuff I think now this has broken down to just p as I think it's a bit more easier um to maybe just discuss on these items that's that's race um see if if we want to add any more or even just modify any of these um I think maybe we could look into um

targeting getting this um merge in the next one or two weeks if people agree at least I encourage you to try maybe at least locally um and if you have uh like I think prism team has this one metric that should be just renamed um yeah and you will see it on the dashboard and please feel free to add the your issues here in the comments if you have issues about this

metrics and uh yes Jim said we have another PR which is more uh we have a hot discussion about lip2p matrics and gossip sub matrics so I think we will just uh move it to the next call yeah I would focus personally on the uh P death specific metric because that doesn't require any renamings from table UHC names and all of these should be new metric names anyway for every single uh client and maybe if you have if you use um any specific metrics that can be

helpful for other clients um they also feel free to share in the comments uh what would be nice to have as well because I just uh I use it dashboard uh often and I can see um the issues when they appear so um for me it looks like another helpful SC for testing um for Lighthouse we also have um a few more uh metrics that could be

interesting maybe I'll yeah I maybe I come into this afterwards um because um when we're doing the the Gap lops experiment we we're using a few more metrics and for example the computation second one the sorry the beacon data column cyle computation seconds that that metric we also have labels for it um I don't know if if this is something all the clients want to implement because then we can kind of look at the numbers based on the The Blob count as well um so that's all thing um I will have um yeah I'll see if I can add a comment on the pr afterwards yeah thank you and regarding to the seconds um I've experimented with

the milliseconds and seconds and it seems like um seconds are enough um let me quickly show you so these are seconds and these are milliseconds milliseconds um seems too detailed um I don't know if you agree or not um so we can also discuss it but yeah yeah yeah sorry this might go into too much details on the metrics but um I wanted to ask whether the the bucket is

also part of the spec because I think that would if we're going to use the same metric the bucket will be quite important to also be part of the spec um I think we can um we can proceed with it I don't know maybe later or we should do it now because for example I'm currently implementing histogram for Teo and uh um Deco doesn't even have an opportunity to change buckets at the moment so this is what I'm trying to solve um yeah but I think it should be helpful as well what uh what how I see it uh we can uh later add buckets as well and

probably the average um values um so everyone can see um nice um is there anything else or otherwise um I think the client teams should review this PR and um add feedback in the next week thanksa um thank you I will share the

second PR as well in the chat thank you nice thank you any other spe discussions one quick question has anyone started working on re your death branch on top of VRA it's probably still a bit farfetched but just uh a temperature check

um yeah uh pris is in the developed branch and uh we rebase regularly let's say every two weeks we base a p branch on top of develop branch which include pict changes I'm just checking if we get alha eight for if we do a real launch with super nodes only so that would be basically require uh I think a rebas for every client but if it's too much work then I

think we should spend extra Cycles on um does anyone know like what's the state of the Petra devet like how stable they are and um whether they're um they're launching the next one soon yeah so devet 3 is very stable it's 100% participation right now um devet 4 is targeted to be launched by the end of this week hopefully so there's still client implementations that are needs to be done but as of uh this pack is kind of Frozen as of now but I think we might

have at least one more devet before we can go on to for public but that's going to be a a small change like I don't think we're going to have anything meor added to picture anymore we could of course wait till uh Factory is absolutely stable and is rolled out before we do the Reas I think the longer we wait the harder it's going to be possibly great thanks um any

other topics to discuss any anything else anyone okay yeah um yeah I would like to make it clear Barnabas um this metric specs um are they most for uh more for um developers or for uh users because uh I find it super helpful for a current testing process um and I'm not sure like um if it's needed for users uh for nodes operators I think it's more more

developers at least right now I don't see any users getting too much valuable information from kg commitment computation time for example um yeah I agree and that's why I want to um point this out uh probably we shouldn't uh take a a lot of time for for these agreements or namings like to help the process uh of testing like to make it more effective um yeah so I would encourage to try to try these metrics um thank you I mean it's important to agree but

um yeah uh just to keeping it keep it in mind that um it's not for final use as for example bacon specs it's for our inner testing process yeah yeah yeah I agree I think definitely more useful to have these metrics early on than later on great thanks thanks

SCA um yeah I think last week um uh sorry in the last breakout call um St and Josh um sent out a a a form to fill out um for the P contributors um I don't know if that if that's been finalized or but if you haven't through out then just just a reminder that um the form I think the form was shared on Discord earlier um if you haven't F out great um anything else anyone if there's nothing else then I

guess we can finish our call earlier today yeah there's also Daylight Saving here um in Australia I don't know about elsewhere but I know that Europe is also having it in two weeks um so I think the time might suck a little bit for some people um maybe we can revisit this in two weeks time if anyone has any thoughts um feel free to share in the Discord channel oh thanks for the link fresco cool um yeah thanks everyone um we'll

keep chatting in on Discord and we'll talk to you guys in two weeks thank you thank you bye have a nice day bye bye bye bye bye bye e