PeerDAS Breakout #11

Transcript

e e e

hello hello hey the the daylight saving transition cause some damage to

people's calendar always hey guys yeah e

end this today not sure if we can do all the C updates yeah if anyone wants to give a c updates first gy hey hello um from the lighthouse side um we don't have a whole lot to update this week um we've been working on other um priorities preparing for our release so we haven't made much progress this week uh we've got some refactoring happening on um on the syncing bid but we haven't make uh we Haven missed it significant progress um hopefully we can

make some Pro thumb on progress in the next next week but yeah we're still working on sink at the moment um but not much elsewhere that's all from our side no um and then yeah hey uh yeah from side uh M uh has been reasing uh uh to Petra but uh he didn't manage to complete it before his vcation and said it and it's another week probably to complete this

uh from my side I'm doing timization factories uh uh removed uh recently removed that soly think like uh for sampling like retrieving um uh R samples from database and validating them again so and I just yesterday saw that uh po PR yeah that makes sense uh that's it compite thank you um um anyone

else um hi um I'm from the nimbers team um there were a couple of issues that we tracked down during pias devet 3 uh one of them was um there were times when the Nimbus BN would Boot and the CSC param inside metadata wouldn't get populated um this was an issue actually raised both by Prism and Barnabas um during the devet 3 run this was one um the other issue that we had during the devet 3 run was there were often Sometimes When We Were broadcasting reconstructed columns

but um there was a proof and commitment length mismatch uh and possibly possibly some client must have uh picked it up and still accepted via gossip or something um I think it caused a little bit of breakage in the network but uh in spite of that I still found load star teu and prism to be fine so that that issue is fixed as well um so this primarily happened during Reconstruction and it was it was a very like Niche um reproduction it didn't

happen all the time but it happened sometimes so uh coming to that I actually tracked down another issue Nimbus was having so Nimbus actually maintains something called quarantine so it basically is like an in-memory cache which keeps the side cars uh against blocks that have not been processed yet now since Nimbus was reconstructing directly from the database like reading and writing into the database there were often times uh let's say when uh we received block proposal and we didn't receive all the columns against that block maybe due to some

reason uh in those times even if our database was populated with those reconstructed columns our quarantine cache kind of kept on crying that uh some of these columns are missing and uh turns out it like kept crying for a few epochs and it kept root requesting other peers and probably it is a possible reason for us to get Downs scored so I tackled that issue uh by basically enabling reconstruction right from Gossip so right now if a super node uh let's say receives enough number of columns

directly from Gossip validation it keeps trying to reconstruct uh rather than a later reading from the database and then writing back uh this way we don't have any inconsistency between the quarantine cache and the final database like all the columns persisted in the database so that was the other issue um apart from that I have been actually um doing some experimentation along uh column syncing for the super node uh which basically involves uh attempting to reconstruct U columns against a block index um let's say say if you just received 50% an up via a range request

from another client you don't need to rely anymore so you try to reconstruct and uh populate it in the database populate the rest of the columns in the database and thereby proceed with syncing um I think I would want to keep this feature specific to Dev Nets because there are occurrences when we are receiving bad columns uh but in practice I don't think uh we would trigger this reconstruction while sinking that often and lastly uh we are so Nimbus doesn't really have a proper way to handle soft Forks so what we are trying to do is since there has been some talks

in and around uh activating pias at Fu we are currently integrating the fulu fork in Nimbus unstable and we are planning to activate pias at fulu right now in order to in order to somewhat decouple some of the Electra code and the pias code uh like like keep it separate for now right now until and unless there has been uh some finalization uh from EF or like all of us so currently our design States like so we'll be using the same Branch for both both pectra and pias and uh we would really like to

trigger pias at fulu um if so doesn't happen if let's say if the curtosis config still says that uh let's trigger it at some EIP 759 for fork our plan is to pick up that like parse that number from the config and replace it with fulu so we'll be faking fulu uh because there is no other foolproof way in which uh we can basically have a soft for going on in pias so that's pretty much uh updates from Nimbus over the last two week last two weeks yeah uh that's pretty much from my side as well thank you agnish uh

so next one is will be PR n test the flights hey guys yeah so on our end um we're basically just fixing some initial syn issues so there was a bug on and solving full notes so we have patched it on our end um we've also done some big optimizations to match column verification so that's done so now prison can sing a large amount data columns uh you know in less than a second rather than 3 seconds that was the case

before um on re basing on top of pectra it already works for us so it if you run prism with the pectra dep net with P we should be okay last I check yeah that's all would forecraft activation on F would be working for prism also um so this should be a new for is it yeah uh trickier new for you would you have the state having a new for verion yeah so we would have a new Fork

version at least that's what uh nimus is pushing for yeah that's a lot more work for us because it will essentially be you know setting up everything from off and you need this transition to transition the be sa from to full aish yeah but like I have a question so

like when prism was uh working like on pias as denb was in the beacon State denb and now that it's Electra I I I'm not like sure how uh prism is handling peas like uh I mean if if the beacon state was denb and still pust worked and then the beacon state was Electra and still pust worked shouldn't uh it still work for full like no but you're ask you're asking for a new state transition this is not like simply Petra and we have a soft for at the this is basically yeah you have to

add in the new fa version and you know want to schedule that you have to do like a bunch of other things so it's a lot more yeah it's a yeah it's a lot more work than just having a a soft po right right I mean um okay yeah yeah yeah I get it yeah yeah and thank you uh next one will be low start so can you provide with the context I

join late now we're just doing the CL updates yeah uh sort of we haven't worked on it since the last time so we have not much update from our site but uh with regard to get blobs uh so we were talking that get blobs will also be useful in uh uh imps but I I think that g blobs uh might not be as useful over there because uh with my get blobs experiment the hit rate is like 80% it's never 100% and

for you to reconstruct the data columns that you need you need all the blobs because a data column is a slice column slice across all the blobs so uh yeah but apart from that y that's it thank you do we have anyone from granding here and did I miss anyone okay um thank you so next we will talk about

the uh spec sorry def updates I think we could start with the updates from Babas tell me us about the current def progress uh yes so def three is pretty much dead we had over 100,000 slot reorg from Prism side week in the end of the week I don't think we can realistically recover from this so I'm proposing to shut it up today and uh we need to start discussing them four timelines I think oh thank you

oh um so the document is this this one right I just post it on the in the chat yes for the so so the def Force back would uh probably include the rebase unless uh people want to have another DEET before this uh which is the same spec as3 the thing is that uh the two branches are getting further and further apart so you should do the rease I think as soon as possible and I think most Cent teams are uh focusing on delivering

Petra right now so it might make sense to take a month off from having a p zet and just focus on uh working on the rebase and then launch pet 4 once we have pectra devet 5 uh ready to go and then build it on top of that possibly with the fork activation at fulu so that way we can uh have both words uh ready so that would mean that the P met 4 would probably launch around the end of November uh possibly a week or two after we can launch Spectra D 5 and ideally we would activate uh beard

death with Fu that point yeah I think most teams will be in Devcon uh that's why I don't expect to launch in like the next two to three weeks anything uh the question would be if client teams would be ready to do Dev 4 in the end of November uh assuming that they would also need to implement a new Fork new for transition I mean with the

FL thank youas this anyone have any suggestions or objection I I think I I would agree with this plan like to have a more comprehens full death uh death n instead of like pushing things Rush before diff is it uh fair to assume that we want to activate P Das at Fu are all the client teams okay with kind of hardcore

coding that because that would actually assume that we want to include P death in the full upgrade pretty much 99% is there any any other feature scheduled for f at the moment you have but not noite yeah I surfing just surfacing the discussion that's also just happening on chat um so if we activate on fulu um essentially

how we would handle testing is so there's eof that activates on fulu as well but we would just use uh for example nimus stable for EF testing and we activate fulu there but fulu means nothing for n stable for example um but it does mean something for the El for so the El will activate eof and the CL will basically use whatever electri and in the pias world um it we can activate fulu as well but we'll use get stable for example and get stable will just ignore the fulu fork because it has no logic and it will continue with whatever we've defined in Petra and um whatever the CL client is Will activate

pias so we're just essentially agreeing to use different branches for testing um as long as the client ignores a fork that it does not recognize so that's the one Behavior we need so if a CL fails when we provide it um a fork and it doesn't understand it then we would have to fix that behavior but otherwise I think we're okay are we sure that uh El can [Music] ignore we'll we'll have to sanity check this but I know G ignores it for example okay um but we we will have to sign it to check this yeah I know that CS would ignore uh unknown forks I have verified

that already okay perfect then we should be okay yeah but yeah that's essentially the one Behavior we're agreeing to today everything else we can sort out with using different images or Genesis generator or kosis configs but that behavior is something we're agreeing on and I think by the time we hit Devcon we should just agree on which version like which spec release of Petra we want to

base stuff on and just Target that release for a longer duration of time because I'll assume there's going to be more Petra spec releases and if each time we have to update multiple code bases that might be harder I think we should already agree uh to be devet 5 Spector devet 5 spec which would include 7742 because we would potentially want to mess around with different number of blobs so ideally all of that would be based on top of that yeah makes sense yeah I I I just don't see us needing to Target anything earlier than that

right um okay so guess techically for like including to with only P Des features easier for us and somewhat make the code base cleaner but it thinks it's better to have a double check at ACD with the El is that correct oh har yeah I guess double checking with ACD does kind of make sense because this essentially submits to pias as the next

for I we can always change that but yeah we're essentially saying pias is full which we have agreed in paper earlier but I don't know if we've made a strong commitment okay um yeah so let's double check at AC um okay we hold have still now we have more weeks for the definite so included more small P um yeah thank you uh so I'm going back to the the chat so Babas just that can

discuss 7742 yeah um so I think it is yeah so it would be good to discuss uh how we want to actually activate uh the Target and the max block uh whether we want to keep the Legacy values and then just introduce new one with the Electra and then Hulu or whether we want to do uh change the the already existing name to denb prefix or postfix as far as I know uh config values should

not be renamed so there was some push back uh to rename existing config values so it's probably just better to uh prefix them or postfix them with new Fork name e yeah so after the reest I

will go through again to see if if this uh yeah would uh impact I mean at least IP will impact the current P Des feature I think that would be included with this um max blobs per block change um but like yeah and I think I reply in the Disco that I think we would not change the current um d configuration so like the we will do the

traditional spec writing way add the prefix to the or surfix to to the parameters uh which should you know more redundant but um it seems the safe solution to the book changes unfortunately is there reason that D was not included in those names previously or was that just a mistake um no just when one configuration uh First Time created it

doesn't have any prefix but when it is updated it uh in the next for on other FKS then we will add the prefix but for example the max blobs per block that's not a value that is actively being used that was added four months ago only so I don't understand why we could not add the naap for this value that is listed under denb but it's not actually denb I think uh the plan just made a error including it under the

N was it a preset earlier and was moved to config yes ah okay but as far as I know it's not being used by any client things yes so if it the sorry I think for the BL uh Max BL that one would be P Des only but if there are ex some other parameters that will

be uh if this is the new ones then it will be you know we will have to add a such a Sur like I'm checking is that Target BL block Target block per block is that D configuration the Target no the target one is also not the configuration as far as I know like all of them are like BL uh sorry pure death only I don't think we have a Target

config even set and and what I'm mean is that the max blobs per block is not a config value that is being actively used by anyone it was it used to be a preset and then four months ago during uh Kenya we discussed that it should be a config uh parameter and app on made a PR the 3817 which introduced it as a config parameter but as far as I know nobody has actually used that config parameter till today so it's it's not likely the it matters what it's being called we could just rename it to being uh electure Max blops per block electure or Max blops per block daab six for

example um but like ma blocked a block that one was a d used one right no it's not that's that's my point it was AET earlier now it's a config so it actually hasn't been released on meet anywhere it was added four months ago and we already had the app four months ago I see this I mean the max BL per

black that one was using the D P2P protocol SP as a preset not as a config ah I see so if we introduce it as a config we might as well introduce it as a config with a proper name with the with the denet name in it yeah I think I don't know maybe client teams can chime in on that because I think moving from preset to config is easier than moving from preset to config as well as

renaming I guess we can take that discussion to Discord um I think just probably we put it to configuration so that my thingsam could have more flexibility to to tune this parameters for the loal definite also but this now it's became no F parameters which make it no probably that's

configurable I don't know that's as partic can continue the discussion uh at court yeah I think um another point I want to raate is um we want to avoid renaming um the those preseted configuration because um it's also a breaking change um for users of the beon node because beon node serves an m point that returns a list of configuration um of the current node so if we change a config it would be I mean if we change a config name then it will will be a breaking change but do you actually return this

value as a config um because this was a preset before so I don't think this is returned by any uh consensus layer client that's my point yeah I'm actually not sure about um I can double check okay uh um so let's uh check the speace check the Cent

information and then get back on the Discord uh sorry uh let's go back bit to the definite PR discussions um so think also like I think pop added one items on the agenda pop do you like to give a update on that PR uh yeah uh yeah I I want to bring the deou thing theoule sub thing back to the discussion like yeah it seem like uh everyone in the pr like is convinced

already like that that okay uh is somewhat useful maybe yeah I um uh is there any like uh opinion or comment before we can probably like much that many if C have this appetite to includ this

refractor to the defate anyone has any suggestions so come in if not we can get back to the ER um GitHub discussion later uh

yeah uh and also I have another PR is is about like uh because currently we don't standardize how we keep the the the data column like in the in the store and and it turn out that uh uh there is there are some issue in the spec like for example uh we we we cannot make any assumption on the on on on on what is inside the store like for exle uh what what is colum that is keep in the in the store is already valid or not so uh yeah so this PR is about like standardize it like to make sure that every data column

that is in the store is is already valid so maybe you can take a look in pr thank you pop sorry I I just forgot to uh do we have any other like features PRS discussion

he everyone uh can we discuss quickly metrics sure uh sorry can we discuss yes um yeah I I will quickly share the latest ER we're discussing and metric as well so um the we have two points to discuss uh do we need to use milliseconds or seconds for runtime metrics um we've discussed it already in

the chats so I would like to hear from the clients uh there is a concern from Prism that uh there should be more granularity for histograms and for runtime metrics so millisec milliseconds would be preferable um what do you think and this metrics will go to satle and will be used by research teams as well at least some of this matrics currently Lighthouse uses

seconds um and prison uses milliseconds so we need to agree what we are going to use is Jimmy here oh yeah Jimmy could you please comment yeah um yeah we don't have a strong preference so if MC is preferred then we happy to change our existing metric I guess okay then I guess uh anyone else if not I guess I will turn it to

milliseconds and the second um question is uh single or bed verification is as far as I as I got it it's the way of optimization and the clients do it in different ways so it's kind of difficult compare um maybe we can leave this metrics uh for monitoring not like for comparison if it's not possible but for mon for monitoring um Barnabas how do you think because some clients uh use only batch verifications some some of them use

badge but um in fact it's single so yeah who is who is reporting that it's doing batch but doing single because that sounds like a a mistake it's uh it's in this PR uh if you go down uh it's t SEC and Matthew we were discussing this in the pr so the last message so the client's concerned that

it's not possible to to compare this uh implementation b b verification I know that it is important for research team because we discussed it uh with migal UPS um and um yeah so I don't know uh I would like to hear from you do do we need to leave to continue work on this metric prison team unfortunately not

here uh copy paste um okay um I actually have a question so I I think I don't understand what is single seconds and bad seconds because um if we are using the same cap API then I think the only public uh function that is exposed is verify kzg batch I think and I think in that case we would just use uh batch seconds and not single seconds at all I mean I'm not sure how other clients do

it but uh this is actually what we do and I I did confirm it from Justin um he said that single verification is not probably exposed so I'm not sure how clients actually use that function so yeah or maybe I'm getting some it wrong totally so I I just need some more clarification on this yeah that what I would like to hear from clients as well if anyone uses single verification um yeah so for Lighthouse

um I'm just having a quick look now and I think maybe the issu is from our sides because we we we actually used the single metrix but underhood we're actually calling the batch verification function so maybe that's where the confusion come from so we could just change our me to use um the batch times and remove the single times any other clients what I've he now is it's only B verification so probably we can skip this

single I think we are all oh here something sorry I think I just raised my hand by accident sorry yeah so I think we are all constrained to use batch verification because we don't have access to the single verification yeah function that way like how ckcg is exposed so I don't think uh unless somebody uses um G's Library I don't think we have access to single verification okay okay I think if anyone

is agree so we can skip the single verification uh I think single verification just got removed from the API so it's just not available okay and as for batch verification um does the implementation differ um much or or not or it's how it is implemented in in the clients can we say that we can compare it or not uh for the functions themselves they should be comparable so probably I I think the

only yeah yeah I think the only difference is um ckcg calls a separate single verification function in itself like in the batch verification exposed function and probably Kev does everything in one function like that's probably the only difference so I don't think it's U I mean I mean technically it's comparable only yeah pretty similar yeah so um let's try to implement it and we'll see the results if they differ dramatically or

not um so the other group of metrics is go gossip sub metrics and the main concern is how to counter unfiltered U received messages do we need to count um invalid messages or not because some of the client filter them like even discounting um so that would be nice to hear as well um how you implement this filtering is it possible to um to count unil unfiltered messages and

um do we need to count invalid messages or only valid messages duplicated sorry k um I missed that um which particular metric are you talking about yeah sorry I will quickly copy the into the chat so we are talking about this um received counts and filtered so it means um

received gossip messages the discussion about it is also in the 13 PR um I think it's prison team unfortunately it's the prison team is not here today so it's a little bit

confusing to to discuss it but just uh it's it would be nice to know how how you count uh if if you count how you implement this um filtering part so do you include invalid messages or not include if we're talking about counting this messages is it possible to uh to implement it in the same way

ages um could you comment yeah so I think what you're like suggesting is at the high level once we run gossip validation you want a count of all the messages uh that failed gossip validation but just at the high level right because it's like it'll be really difficult for us to do it in the LI P2P layer because there are duplicate messages firstly and secondly it's because like Nim P2P is just not ethereum specific um it's used by others so we would have to

maintain a branch if we would want to count invalid messages in the lib P2P layer as well that would be like very like intense hard work for us so I think what you what you mean to say is in the app level and not in the LI b2p layer right uh uh so we we don't need to count uh invalid um invalid uh messages like uh separately uh so we just need to agree if we count uh all the messages including valid and invalid

uh in in this uh duplicated metrix so because um yeah it depends on implementation Dustin can you care to comment here once sure uh I I think very briefly I'll say and I know I've been I say outspoken in that PR but um I don't actually have anything necessarily against I think the idea of counting duplicate messages say it's

fine substantively um I don't have anything against that per se um it's a it's a process concern it it is exactly the way Agnes is describing um that these are not ethereum specific viar and it creates either engineering issues or sort of let's say political tensions like like like pushing like does a theory tried out and was colonized with P top um in any case the to standardize on metrics that require with PTP um based on a process that you know you know we in this room are not very many people compared to the people who are us the PTP overall and so that that so my

concerns are broadly speaking process oriented um as as maybe how I'll want to payid I do want to clarify I'm not really against any of these metrics as such I mean I think some of them are dubiously useful reasons I say I mean the gossip um messages have sort of near constant multipliers between you know um the bites and counts but things like that are sort of side concerns um but that that's sort of what I'm trying to get at I guess in my in that PR do we do we need this unfiltered metric in general um I think this is um primarily useful

to see um how much jate um you're getting um for example recently we've been looking at the net metric and we're seeing a lot of like a lot of like usage on the blob topics and with this Metro we're able to see that we're actually getting like the same blob five times for every slot on average so that that's the use of it like to see like how much duplicator we're getting um but I don't know like how like I don't know whether this would be um something that every client wants to add because it might be a bit difficult if it's if it's in the P2P layer the question is if you can deliver

P Des without the Matrix because at this point we have so many different for and different issues that without the metrics I feel like we are kind of blind in like what is happening so I'm really hoping that by uh pushing all the client teams to implement these specific metrics we're going to have a bit more uh a bit more oversight in what is actually happening in peer test and like which columns are received by which peers and which columns are uh provided by which other peers so so it's it's really about do we actually need this or is it nice to have and I feel like we

should focus on the metrics that are an absolute must and possibly make the other ones optional which just have to decide which ones are an absolute most um yeah so I think apart from metrics the one thing that we can do and I have seen this in a lot of load star logs is that load star like uh logs out the E to agent like basically the client the guy

who is uh like let's say requested something or uh sent something via gossip I think think we should all have that like uh at least that thing enabled in pias logs it just eases out the job a lot otherwise we just need to again continuously like filter out PR ID and all these things so I think E2 agents is something like or like not just agents we name it as E2 agents in Nimbus but like uh who is who I think just getting that on the P like logs would be really yeah gajinder has like agreed to that as

well yeah because it just makes the like it just makes life very easy to debug so yeah if if I have to ever interupt some really specific function I would definitely go with loadstar because loar just tells that I am the culprit if I am the culprit really so yeah I mean I think all the clients can uh try to have something like this in their logs what about um graphs I mean we can use this metrics

for grafana dashboards for monitoring for a long long time monitoring because uh as far as I got it uh sometimes um um dashboards help to to find the box which are not obvious in the logs well we have a lot of logs so it's difficult to to follow them all I think the duplicate messages or the metric integration I think it should be readily taken forward by clients who can afford to do it right now and uh I think that would in general

give us enough exposure to what is going wrong in the divets oh yeah even even if like two or three clients have it in their uh implementation I think it's still a win yeah I agree when we have like the the summary dashboard uh where we have two three four clients the it will be more obvious how it can help so I think that's it for metrics

thank you thank you uh so before we ending this call do we have any topics to discuss are we planning to have a peer death discussion in Devcon in two weeks I mean for this Cod or more in person in person meeting I yes I think will be sorry I think that will be a p

session and then I the session on the the two days have this only the times I think I guess we can probably cancel this call in two weeks right yeah probably makes sense yes yes we should oh they will have some impressive meeting during the defcom before

defcom and hope to see you there and yeah we'll canel the code on the 12 then the next course will be on the 26th yes or we can move one week earlier no I I I I can leave it on 26 okay so it's probably the the week that we will um finalize the def impation scope hopefully anything else yeah I I might have like an extra topic

to talk about um not that you guys mentioned about duplicates um we we did like previously an analysis of gosip sub duplicates per topic um my question is like how are teams uh are are teams now supporting ad1 messages on their current versions um we would like to rerun the experiment to see how or whether there was a reduction on duplicates but we still don't know um how teams are catching up with this implementation like um I know that te I think deployed it um lighthouse we discovered that Lighthouse was doing it in a a little bit naive way but I think this has been already merged as well um

could you guys do a small update on that sense yeah from te side uh te deployed it but it's uh it's now in master uh so uh we need to rebase on pectra to get it so uh in pdas in our pdas Branch we still have no I don't want any comments but I would say that it it would be

really valuable to have um this implemented before deciding uh whether we want to increase the blob Target in that sense like um the expectancy of the duplicate reduction I would say that it's High um but without numbers to prove it is kind of like blindly beon so yeah I don't know I don't want to rush anyone but yeah at least have a clear view of what's the current status will be appreciated by other teams as well um for Lighthouse um as you mentioned we we didn't have the optimized version in the the previous release um and um that the pr Optimizer

has been merged and it will be now next release um probably going to be released in a couple weeks time so I think it would be useful to test that out or even with our current unstable version which already has it now comment hey if not then think that's it for today

I hope to see you soon thank you thank you see thank you see guys you bye thank you byee byebye