PeerDAS Breakout #16

Transcript

e e e

H hello helloo how's everybody doing good good all guys hey hello hi hi hello nice maybe just wait for one or two more

minutes nice look got a good turnout today um yeah I'll be helping more doing the call today um hello everyone okay um yeah welome to the P breakout call um well okay put a bigger group today okay think it's to recording as well cool um yeah I think we can get started um maybe we just start with some client updates first um is Manu here today yeah I am hi he uh yeah so during the last two

weeks uh I worked a little bit on valid dat custody it's not yet implemented but it is in progress uh also I run uh almost I run all my uh test scenarios against uh teu lious ners and uh Grand and prison of course uh it's mainly okay there is some a few test Cas which are not okay uh but it should not prevent us to in my opinion to run next devet also uh I had some bugs like sometimes prism used during syn to to

request blobs by range for uh columns for blocks without any blob um and yeah I fix this kind of of bugs uh yeah that's pretty it nice thanks man um yeah I can give an update for lith house um we've also been doing pretty much the same um been investigating uh B custody um and also we've we've done a bit of interupt testing with most clients and we've found a few small bus um on the lighthouse side um the bug that we found are mostly not p as

related but it when it happens um um I went up the sync but um when it happens it's a lot more easier for us to walk away because um because I'm running on a small devet and um the um the custody like because of the custody requirements it's a lot easier if you lose one TP then then it's hard to sync when there's only like five nodes so I think um yeah we've raised issues for those um but I think we are pretty much um ready for um death net and we're probably just going to spend um one or two days doing the test scenarios that um man came up with yeah um that's pretty much it from

Lighthouse side and and we're also going to spend some more time about custody this week yeah um that's all um is take here today demri yeah uh yeah uh we have updated te to the latest specs including the couple of subnets uh and we are ready for theet uh we have haded great acceptance test with two ERS thinking uh to avoid regression in the future and we have added reconstruction of data columns from y get engine blobs it is turned on uh by default

in super node and I I think I have a feeling that uh it it would be good to include this in Spec uh so all clients uh will reconstruct it but I want to revisit it on three on three questions uh that's all nice thanks dietry um yeah I think I think um adding the the get your BL the construction will be uh pretty useful um so we have we got Nimbus is um I don't see

nus um maybe lar Matt you want to go y uh we're on the latest p and got the decoupling done going through the test scenarios and we're going to look at uh validator custody this uh this Sprint so but we're ready for devet as well awesome thanks thanks Matt that's great great to hear um what else we got have we got Grandy yeah um so um for over the two week um we uh Po from grandon Team he

have found um the issue uh on the um peer that tried to SN from Grandin and get immediately disconnect from Grandin um due to the issue on the final route checking that um The Grandin only checking only that memory on not uh because the finalize EO hav um persist in the storic so that issue has been fed and and another up that is I been move the uh dat colum topic into the call topic description also have been prev the from

subscription sorry I think it's a little background from sias yes I might have to I might have to mute actually can't mute so let's do do you mind yeah thanks cool um have I missed any clients I think um okay so so gring speaks um a syn bug um you guys also appear already um so for the scenario this um I have been um Deb backing on of all those that failed for the Reconstruction errors we found a back on over um data availability Checker uh that's why that

um some of uh full Note have been F out because of the checkings when we hand when handle the blocks so this uh issue has been fed and also we have been refit colum syncing and another issue uh on the um when the Cent when the peer Tred to sing in the finalize uh Network this issue also have been fed that du to the PE disconnection issues and another issue that we found not really related to the P but uh regarding to the friend when we appear tried to

when gring try to sing from the unfinal networks so this issue have been report so if we uh some f from granding Team be um because this is not related to the P right I see yeah I'm already okay great great so like they mentioned that the syn issues are not related to are mostly not related to P yes um for the thing issue is not P okay nice um thank you an um don't have num I guess um maybe we can talk about um

devet um does anyone is does anyone here want to talk about um a devet launch date like maybe yeah in my opinion we should launch DEET as soon as possible um and if everybody's not yet ready um I think we should launch and and for people not ready we should for cents who are which are not ready we should um create the validators and they will uh Miss some slots uh until they join the network but it should be okay it should not prevent the finalization of the network and I don't

really see reason why not to launch devet right now maybe it will fail but at least we we will learn uh something yeah I think it would be great to start um well given that most clients already um have started doing drop testing um yeah um Parry mentioned that um want to decide on Deet parameters um anyone have any thoughts on this should we go with what we had before yeah I'm just going that is it

worth doing one with super nodes only first seems like a good idea do we do we want to like maybe just put the validators on the super node and still run some notes does that make sense actually that would be good so we could see what's happening in the logs uh on the full node rather than super nodes cool um any other any other opinions can we start with super n Only and if we see that everything is going right going

well we can just uh remove the super nut flags from some of the neser you repeat that I realized my audio was piping in the wrong place sorry uh could you repeat what you said I didn't feel yeah yeah yeah um so there is this idea to start a new devet with super nodes only which could be a good idea my question is would it be possible to so to start the devet with super node only and if after a few days we see that everything is going all right we just

remove the super no from basically the subscribe to all subnet flags for some nodes we restart the nodes and so we switch from Super nod only devet to a mix of super nod and full nod devet yeah that would be possible um do we want to have the devet to be the same size SL number of nodes as pias devet 3 or do we want more nodes or smaller Network or what do we want that I I don't remember what it was for for the net Tre it was three super node and six full

nodes if I'm not sure I think think had something like uh probably 15 10 to 15 notes sorry I'm par 10 to 15 noes and and how many super noes were there uh I can look up the SP sheeet I'm not I don't have that number at heart right now ah no it's fine it's okay um yeah I think I think it'll be great um what uh Manu um and Fresco suggested maybe we start with supern network um that has very attached to it we can run some knowled with no datas and then if it

goes well then we can probably switch some v data be no to not if that's not too hard yeah that should be easy let's do that and everyone's okay with me using the branches we have in the spec sheet right yeah te is okay I'm okay as well your Li have is okay as well yeah perfect I think that's everything we need then yes of regarding to the image um U I would like to review the image from

the same branch that been pinned on the switch so same same Branch for grandine as well yeah yeah yeah yeah so I miss uh lar um Matt is it the same image as well uh yeah pure d uh so I I talked with about it okay great thanks and I think uh man mentioned that want to launch ASAP um yeah I agree as well um anyone want to wait a bit more or we go to launch ASAP it looks like we can launch after this

call thanks Perry um yeah so cool so we've got dev4 coming is it um yeah I guess it's a bit too early to talk about Dev 5 anyone want to like talk about what to include I guess we can proba move it to the next SC I guess there's a lot of testing required I guess in 5 we can include validator custody yeah nice um yeah I think that would be great and also I was thinking um I think sounds like T's already

implemented um distributed block building I don't know how many clients have implemented it um I think we've talked about maybe including that in the spec and I think it would be great to maybe include that in DEET 5 as well yeah ver custy was optional for de n yeah I don't I don't know if anyone's implemented it um but it'll be nice to include it in De live for all the clients um I was thinking for death 5 do we want

to maintain the same block block counts or do you want to try something a bit more a bit more aggressive if we are good in the net for it would be good to try something something more nice thanks Demitri anything else for DEET how do we generally plan on testing how well distributed block building works because it feels like such small

Deets wouldn't really be helpful um like if if it's you know I know 20 nodes or something um we can't even fill the the subnets with regular nodes like everyone is just going to be connected to Super nodes and and that'll be like the main way that you get data so it's like a very different kind of um yeah I mean in general just like a complete different network um than than main net or test net um so how yeah like at which point will we start trying to figure out how well it works in practice and how we go about doing that yeah I agree with uh Fresco I think

I think um we probably need to like draft up a few scenarios that we want to test um yeah yeah I think we probably want to like just make sure we have enough scenarios maybe um if anyone has time maybe we can like work on something to maybe like list a few scenario like what what Manu did and have MD document to um to figure out this scenario one test against um yeah like and we also need probably want to think about like the number of nodes as well to make it useful um yeah I feel like this this this this video actually require a bit of analysis as well um I'm not sure if if people look at um like a blog post at

Lighthouse um I would I we've done before um we we've we've spun a network of 100 noes and trying to like do some measurement of the the the the effectiveness of um distrib BL building um but that was kind of basic so I was hoping if we had more time I think it'd be great to maybe do like a cross client effort trying to trying to test out like how effective the distributed block building is so Paris link the defit five document

maybe we can also include that in there as well um yeah yeah sorry Franchesco I'm not sure if I answer your question um does anyone else have any thoughts on um testing distri bling an idea would be to to limit the the peer count in clients I don't know if it's possible to do in all the clients at least in prism you can specify the maximum number of peers you want so it will help to test the distributed block building in U like in a real Network where not every

clients not every pi are connected to every other paer of the network we also have the ability to add some packet drop and rate limit and latency filters on the network itself like on the um on the Nick so if that helps with pushing the network towards more block building tests or HED case tests then we can also enable it so par I missed that um um enable what yeah so so we can do rate limits packet drops and latency increases um on each host and as fine grained as we want

um so you could say the default pias dnet 4 is every node has gigabit internet everywhere and is basically peer to everything um but then we could limit the peer count as well as drop packets as well as limit um internet bandwidth to like 10 MVPs or whatever we want to see um and then see how the network performs if that helps but I think from our side it would be really nice if we can have a list of things you guys want to see I think how we achieve it we can figure it out quite easily we just need someone to come up with the list yeah sure I can probably start with

some some um some testing um scenarios and and add it to the devet 5 documents and it would be great if people can review and maybe add a few more if you have anything in mind yeah um I was thinking maybe pack may I don't know I think it might be useful to test scars where like columns doesn't reach the pi and um and and and like know could find other ways to get

the the columns in some other ways yeah so I think we could try to artificially disrupt some some subnets I don't know what would be the the right way for to do that just select the subnet and and some nodes which can easily support that are not pushing that out Simply and U you know don't not every client has to support that just one and and when it happens that's not pushing it out we know which which one it is we know uh that that will happen and then we see what's what's happening as an the after mass of

that okay I got it um yeah I think prison and Li house have ways to to dis disabling publishing some some columns yeah it would even be good to just not publish anything just publish the block only and then just see if things can work uh without actually the proposer really doing anything other than than pushing out the block yeah um yeah I think we can include that as well yep if there's a few um things in the

chat I'm just going through just to make sure I'm not missing anything um malicious Pier yeah we've got prison that's malicious yeah all right cool um yeah maybe um yeah I think I think there there's a lot of scenarios that we can add um yeah PR that the focus thing uh yeah I think we can probably um discuss these scenarios as think can and add it to the document um i' um yeah I've linked the PS dev5 Link in the in the the no as well um any other um any other topic to discuss for deify it sounds like we're getting a bit

a lot closer um to doing some some like uh some some EDC testing the next few DEET which is good um cool um if there's no no more defnet updates then we can move on to spec discussions any anyone wants to raise anything about spec I think I think there's something in the agenda a second yeah I did mention a few things on the agenda um so so so one of the the question I want to ask is whether it

makes sense to add um some more some more um description to valat custody um because um I think we probably need to uh figure out like how the back filling of columns work um or I think I think at the moment I we can assume that if we add more balance to the validator then we increase this the custody group count and and the no the no then supposed to serve those columns um but it's not explicitly mentioned the spe that we should back fill to um to the entire da window um does it make sense to add a spe item for that just to be explicit

or yeah I I think I agree explicit is better in this case yeah um yeah I haven't actually looked too much into into into the back feeling um so if I mean um we can probably look into that unless anyone else is also looking at V the custody I think I think man's been looking at R custody have you have you fig have you noticed anything else that we we should add to the spec

um um yeah um actually you motioned it maybe two weeks ago um should we update the node record and the metadata only when we back only when we finish back filling or as soon as uh some new validators are added to the validator client so issue if if we update if we increase the the cgc in the enr as soon as um new validators are um managed by the beacon node is if we don't have had any enough time to to back fit the blobs the data columns then

some peers will may try to to to to to request by rout or by range some data columns we don't have yet and some peers May disconnect or at least decrease don't score us for that given that this version Val just fully relies on kind of uh honest behavior of nodes and doesn't have any sort of accountable component I think it would be better to just only update these things once you have backfilled everything so that yeah you avoid what what you said given that there isn't any like possible punishment that can come from that like it can never be that someone um says oh you attested to this thing that wasn't

available even though you're supposed to have back filled it or or something like that so it's like safe to do and I think it seems like the most straightforward Behavior yeah sounds good sounds good um yeah seems like a simple simple solution and I've seen some thumbs up so maybe that's something we can go with um yeah would um yeah will someone be able to um um create a spec PR um I think I think there might also be some some some clarification that we wanted to add including um uh actually I I think Franchesco

solution might have cover like the questions that we talked about before um if if anyone can work us back um P that will be great cool um yeah another question I had was um with distributor block building um I kind of wanted to see um what everyone thinks like what's the minimum form of distributed block building we need for

um shipping p in in fulu um because at the moment I think I think I think clients have implemented it maybe slightly differently um do we want to have a have a spec item for describing the distrib block do that goes into sorry hello fres I just saw your message um what should go into that spec here um yeah I guess not updating after until after

backing and explicit requirement to backfill yeah I think I think I think that that was all right um I don't know if there's anything else missing um man is that cover what you what you mentioned yeah exactly is that yeah awesome thanks thanks right then then um we can move on to the um distributed block building um topic um yeah I was asking if we want to have this in the spec or do we just go like do we just Implement

our own solution does anyone have have any thoughts on what sort of dist block building we should have for for Fu um I think I think there's a separate breakout C that's that's been happening I'm not sure if there's any progress there some something that could go in the specs is the uh publishing Behavior like something like U proposer May publish just a blog or or something like that um if all the blobs are just coming from the M Pool um or or I mean anyway it's some maybe some kind of specification of like when it is when we consider it safe

to to do that like I don't know maybe you're supposed to only include blobs that uh you saw a few seconds before to be safe or I don't know something along these lines um then I guess there's the reconstruct and repr propagate part which I guess is already in the spec and it's sort of maybe not oh I well okay yeah that's already in the spec but not necessarily there there's nothing in the specs about U pulling blobs from the mle right not specifically for p sup I know I think I think I think there's a PTP clarification um PR um that um um I think en code did but that was for blobs

I don't think there's anything specific for for columns and and and having to do reconstruction for on the blobs and Publishing The Columns I don't think there's anything yet in the spec um yeah I think there's also one there's also one um optimization that we've done in Lighthouse and um like we call it um like gradual publication when we do when we do um fing block e so we don't

publish all the conss that we reconstruct um because that would like take out a lot a lot upload bandwidth so we gradually publish them so if we receed them over cive then we don't publish them so that will kind of reduce the the required um outbound bandwidth for noce that Implement distri block building everyone ended up like publishing a small subset of random columns instead instead of like publishing everything over 128 columns um yeah I think that's probably also worth uh I don't know I don't know if that's worth including this but I think that's just a bandwidth problem any any thoughts on

this I mean to me just doesn't seem like something that there's very clearly a lot of value in having um like a unique spec strategy like it's I mean this what you said seems pretty reasonable but I could see that there's many like little variants that clients could Implement yeah yeah yeah I think that makes sense it doesn't really impact interrupt or anything um yeah so um yeah I can I mean I can open an issue on in the consens spec rep on on this see if anyone's able to um a for the publishing Behavior a

bit I think the the next item I wanted to ask um in the agenda was um the death perimeters that we want for production um we kind of talked about this in the defet five spec um it feels like we still need some research and Analysis I don't know if it's if it's a bit premature but um I was wondering if if client teams think having under column subnet is is okay um with the current setup cuz I think with Lighthouse we've tried doing some simulation with um a big Network and AOS so there's no supern noes and um feel like sometimes we struggle to find enough peers and we might have to increase block

counts um but that this is something very liouse specific but I was wondering if people think um that if we need to do any more further analysis on things like uh something that count and in any any other Des parameter update or are we okay with the current spe parameters you should maybe maybe let me change the question a little bit um I think initially we start with 64 subnets um we change it to 128 was it just

because we wanted to make it a bit simpler and um and also a bit faster to to do to do one colum subet no the the reason was because we kind of needed once we basically relied on the subet sampling part for uh for a testing um like there not being like peer sampling kind of in the critical path or right now at all um you kind of need the number of samples that you get to be high enough and then if we had not enough subnets uh this would mean that we almost there we get like very little out of out of um sharding essentially like with 64 we only get that you you

basically end up downloading like uh one quarter of the the whole data U so it's it's really yeah basically just like an an efficiency thing like from from like the perspective like how much data you you end up downloading um out of the total sorry I mean like yeah like combination of efficiency and Security in that sense that security like forces the the fact that we have to download at least eight and this is a number that's independent of essentially independent of the total number of subnets um and then yeah the given that you have this eight floor then the the number of subnets kind of um determines how efficient the whole

thing is feel Sim it's very hard to get a good sense of whether or not there's going to be problem here without uh something much bigger than this St yeah I think it'll be interesting to see um how um how plant perform on a f only definite with like sufficient number of notes um because that that'll really test out whether some count is too like to too much for the current GE

account um for each client um I think for L house we have 100 peers um like Target peers um so sometimes we would like I would like notice that there are some subnet that that has no peers um so we 28 something account we might have to probably increase our P count and and for us that will probably consume more resources so just message what's um what's RS

expansion askon extension of the in the quote extension factor which is two okay got than 128 as the factor of two and that gives the kind of the Baseline or cord late if you want yeah I think I think my only

concern is um changing K account on on clients uh may have an impact on the network structure so I think that's something that we would have to really test it out so with one to 28 subnets then we have to increase um the the target per count in Lighthouse y there are clients operating with larger P count already so that's gives some peace of mind but that's for the the D structural issues but individual implementations each might have difficulties with l p counts

all right thanks um any more spec items to discuss um not so much of a spec item but uh going back to um blog publication and like or sorry column publication all this stuff uh most wondering is this strategy that's currently implemented like the sort of um normal one I guess that we do for a publication where uh in each column subnet you're just going to send the uh the message like the column to to all meshers um or something else so I was wondering is there any reason why we wouldn't want to um for example do something like first send um one column

to like make sure that you send one column first as opposed to like sending everything in in parallel um like especially given that we have um that that we do have the read Solomon like uh expansion so it is even okay if like some column or even like a bunch of the columns end up making it uh through the network like slower than others does anyone have any thoughts on this so I can just say what what we do in in in simulations and what's pays out is is really to to make sure one copy of

everything is sent out first and then then a second copy and then a third copy and you are not prioritizing in any way one column over the other uh so which kills per yeah yeah I I I didn't mean prioritizing like a column over another column but like we each column subnet to yeah exactly what you said like to like first send to like one perer um and make sure that like at least one copy goes out like each subnet yeah that's that's what we do in the in the simulations and in the AL in the in the P2P namely p2b based implementation and that that helps uh because you don't have the that kind of artificially induced tale of the

distribution where you are leaving something to the end and and it's it's it depends on how much bandwidths bonak you have but if you do have problems pushing out then you are shooting yourself in the food by basically delaying part of the uh so you have some things which are Super populated super diffused in the in the network and other things which which you still didn't send out so they cannot be just by definition uh diffused in the system and that's that's a problem but uh yeah and and this is one way of mitigating it yeah I guess that's that that will be really really helpful but um I think the

current implementation actually um yeah I don't I don't I don't think we have control over like sending things sequentially to mesh Piers um yeah yeah anyone please correct me if I'm wrong I I don't know if any lip2 impation supports that at the moment there is a stuggled sending in uh in in the name implementation which is kind of doing something similar uh but that's as far as I remember it's time based so you are so the thing is if if you want to do this you have to keep state of you have to introduce another queue or something in in some way one way or another uh because I suppose everyone is

just taking a a message uh basically send queuing it to to be sent to to every one of the pills uh mesh pills and then then going to the next message and that's that means that uh that you don't that the ordering will be wrong so you would have have to Quee them somewhere uh and and uh you need an extra implementation and that has some uh complexity I've started to add this to the name implementation for example but I also didn't finish that because of the complexities and the and the queuing and and and all the things I had to introduce

so um yeah thanks Shabba um so it seems like this is not like the default I mean this this feature is not supported by default in the p2b spec um yeah sorry go ahead it's absolutely not in the spec uh the spec is not saying anything I think the spec spec is not saying anything about the immediateness of sending out uh it's just simply not part of the spec when you send out okay thanks um does any client have implemented yeah it seems like this is

this will be a lipy I mean I mean we we we need some changes at the lead PP layer to to allow for this um it seems pretty useful um yeah think I guess we can probably yes go ahead I'm just saying I can write it up for the P2P I already had some writeups on this uh in the FAS write up for example so I can I can try to specify it uh in a way that it's uh that it's more more implementable you're awesome thanks yeah I think this will be super

useful um maybe we can test out like how the current impation um how will the current implementation work but in the future this is definitely something that we improve um although I'm not sure if we if we if we um if we just can't go to production without this I think maybe we should do some testing to see if we if we need it as far fulu yeah thanks sh and Franchesco is that is that are you happy with this yeah yeah I mean what yeah Whata was was mentioning is exactly what I what I

meant than right um I think on on the agenda there's also um Perry also brought up that um wether want to move the core to e o' in the future to align other breakout rooms um anyone have any thoughts on this for the dummy question but what exactly is is it's a 300 p.m. using is yeah exactly uh it's just that most of the breakout rooms happen at that time and I think there are at least a

few people um in the US time zone but uh who who would want to join but yeah it's up to I guess the implemental to decide what's the best time is that okay for Lighthouse it's or or it's too late um yeah I mean for me I I definitely can't join but um you've got people in 19 team that can join so that's no problem is that three hours after this one started or four hours after this one started it's basically the same time ACD

happens as in the time y for it's an hour yeah that's fine for us also uh should we just take a poll and see if uh see who would want it change to begin with um I mean this time works for me as well but I'm mainly just conveying the information from people who aren't usually on this call maybe we can create a poll on P testing and take it from that if sounds good um yeah you also mentioned about

the frequency par do you want to talk that yeah um it feels like we're starting to hit the stage where um we're having more realistic Dev Nets and we want to try and ship on Main net as soon as possible post Petra so I wanted to bring up maybe changing the frequency of the call from once every two weeks to once a week um that would also align with most other breakout rooms or testing sessions um but the implications also that clients are getting closer to like shipping it and figuring out edge cases and so on we could also reduce the duration if we want if we increase the frequency but just that we don't have to wait for example two weeks to decide on uh DEET 5 specs but instead we could do

it async plus within a week okay so I guess we take it based on the results of the poll and then we can change the frequency to weekly thanks yeah um per do you have um access to change the um the calendar the the Pro Cal uh I don't think I do but uh if you tagged him I think he should be able

to change it okay cool nice thanks so um per um you start p on Discord and then we can decide a think whether to change this the next one right um actually I have one more open question um I was wondering um yeah I don't know if we have um Care on the call I was wondering um what are the states of the kg libraries um because I kind of want to understand um where we want to go live with um is is ckcg and

is like both ckcg and Russ kcg going to be um audited and production ready um and is a longer term to maintain both libraries I don't know if anyone can answer the question on the call maybe George um yeah I think I don't I cannot answer it very well I know that ckcg and rust kcg will are ready I I think go kcg also exists and I think it's ready as well I saw Kev doing some commits last week and I think all three of them will get audited like we we've talked to the Auditors about auditing all three of them at this point uh but but I I don't know if someone if

a client uses go kg at the moment I'm actually not sure yeah I can talk a little bit about about it right now Prism use CG and our plan is to switch from CG to go uh we made extensive Benchmark comparing C and go and right now uh the performance of goaz is not as good does a performance of s energy especially for verification uh I'm talk regularly with Kev uh about this and he still has some ideas to

improve go Kaz performances and my question is is there any client planning to use c as production or not I guess all the I guess ground lighter will switch to rust and I don't know for other clients we're currently using ckcg but we're switching over to rust one eventually I think in Te we use CG and this is better than rasg rasg is not scaled that well as C but if it's updated we could switch

but but currently we like C more nice looks like we um looks like we have at least one client using like preferring one Library um yeah for loar sorry for Lighthouse again um um we are using C foret and we're using Russ k for p and yeah at the moment we don't have any plan to switch I think we I think that's one of the reason why I asked this question because one if you go really want to support both librar when we go to go light with PS um it sounds like ckg is used um and preferred by teu

and prism prefers go kg lar is going to try Russ kg so yeah we've got like we've got at least one client with using every Library yeah thanks George um yeah it's good to know that they they all production ready any other open questions any any question anyone want to talk about hey yeah hello everyone uh i' like to

talk about metrics uh uh I would like to quickly share my screen and show you uh I'm currently experimenting with the intervals and the default interval is 15 seconds and it kind of breaks uh the intervals of the slots uh it looks like this and uh I used the scrape interval 12 seconds and adjusted the query and uh it looks like more accurate R um so probably uh this is the largest scale 1 hour um and it's easier to monitor um from my side um and the question is um uh this is the whole

picture for for three clients uh locally uh the question is um can we apply this scrape interval for devet uh or or not or it will break any I think we can even reload uh Prometheus on the go not shutting down the whole DEET uh bar what do you think sorry could you repeat the last B yeah I think if we start deut with the 20 second scrape interval and it works bad uh so probably we can just reload um Prometheus not shut cutting down the

whole yeah that should be okay so it's it's kind of uh up to you uh if we can I would like to try this scrape inter off for the next devet uh and yeah let's another question I'm trying to investigate if we can align um the beginning of the slot uh to the Prometheus uh so uh the idea is to to compare the Prometheus with the um slot start because I think it's kind of it starts um somewhere every F

every fifth slot starts at uh 0 um so probably we can kind of use some delay maybe or anything else or reload of promas at the very beginning when we launch the devet for example and try to catch the beginning of the slot um um yeah I don't know I think we've tried doing that in the past it just gets very tricky um the easier approach is to just scrape more often for example 10 seconds instead of 15 um and then you're definitely getting the value at every slot but the problem with doing that is you're increasing the number of

um scrapes and then you're using up more disk space and at least in the past it wasn't really worth it um we can do it for pias but yeah I think our default is at 30 seconds of scrape or something and it doesn't meaningfully change cuz it's very rare that someone looks at metrics on a 15minute time scale um realistically on a devnet you're looking at like 6 hours worth of metrics and a 30 second variation or like a 15sec variation is just too small for the human eye to proceive anything yeah I don't think we we need to scrape it more often it would be like overwhelming um I think 12 seconds is the minimum we can scrape it's somewhere

close to 15 seconds which is default now um okay so yeah probably we'll discuss it uh in DMS uh so uh another question is custody so this is uh what I have for custody now I just count uh The Columns when they are written in the database uh this is the prism uh example and uh I would like to know if if it makes sense to count it in that that way or or not what we would like to see um for custody measurement

so we just see the columns how many columns we right into the database uh does it make any sense or it's not clear if it was a block with blobs or not if it so does it Miss uh downloading and writing or it just a block without blocks [Music] uh sorry um I think it it's just kind of frighten not

downloading I I mean you have gaps here this zero with zero data columns and it's not clear was it a block without blobs at all and so so that it should not be any data columns in it or or it was an error uh yeah this is uh something uh that should be investigated I I would like to um discuss it with Manu because it's it's not zero it's eight and the yellow is a full node so for super node uh it works good like it's stable 128 and for full Noe yeah I have I also have questions for this but uh just to

abstract from from this um from this graph uh what do we want to count for custody what do we want to monitor for custody uh is it how many columns uh because um we write columns in into the database um and is it enough or not to measure and to observe because uh if if we use some more complicated calculations for how many columns we store um we need some events to you know to monitor how many columns we store um so

it could be an issue for the clients my question is why do we want this sorry goad go ahead um I was going to say that we um the time is up so I I was wondering if we want to discuss this um async um but yeah maybe um man you want to finish your question first if it's short yeah no my question is about the necessity of this metric do we really need this metric or not I mean what are we going to do with this metric yeah my question is the same do we

actually need it or or not um so probably it's just useless um or I can try at least monitor it locally or some clients and see how it goes yeah I think it might make sense for people to look at um the the spec PR again um just to get a bit of text and maybe we can continue to discuss this async there and to decide whether we want to um include this metric but I I feel like there might be a bit of lack of context here so it might be worth that to do this asnc just to make sure everyone um understand before we remove

this does that make sense yeah sure I will yeah sure I will share the link to the spanks uh in the testing Channel you're Channel thank you yeah thanks thanks K yeah yeah sorry guys we yeah we a bit over time so um yeah so we're going to wrap up here and we will continue the discussion they thinkc and um I guess we we'll PLL on the um on the next breakout time on Discord so yeah that's all um thanks everyone for joining today have a good one bye thank you R bye bye hi guys everyone