PeerDAS Breakout #7

Transcript

e e e

hello hello hey Manu hey e hello hey hey hello

Goan Kev what can you say you know yes it's the gift that keeps on giving what kind of language is that as long as you do everything to go Lang way it works everyone Welcome to the P

breakout code number seven I think um so let's start the Cod we have any Cent updates yeah thank you um yes so for fre um actually so when we change the 32 dat colon subnets uh to one 128 it caus kind of lot of issues on Prison basically we had a race condition when we had a lot of goip subtopic and a very few NS like during like in a CTO

theet with only two or three NS so yes we had um R condition this R condition didn't trick before but now with 128 Pi it's used to treat not always but quite often um and so we we were not able to uh s um a full not it was okay for super but not for full n as well we had issues um where with initial syn because of that so we were not able anymore to uh start a nod not at Genesis but after uh this is fixed uh and the request is in review um about also we work on the de

colon validation pipeline before we had a multiple place in our code where we used to check if a data column was valid um now um everything is centralized we have only one Pipeline and this pipeline is used um everywhere across prism and also with Kev here uh it's more yes with Kev we K mainly work on for us um on the replace of C library to by go by the go library for prism and it seems very promising uh maybe nant has other thing

to add but for me it's uh that's pretty that yeah so uh one of the main things I've been focusing on is uh measuring our data column uh building performance in prison so over the past two weeks when we have run kosis Dev Nets we've run into a lot of issues with block building so uh with a lot of help from Kev I think we managed to Nar it down to the C kzg library so we're still aren't sure yet on why it's uh so inefficient for prison but when shifting over to go kg um you know prison performs a lot better the

devet can run in a much more stable manner so U we have to look further and why that's the case um do we sorry just to intersect but maybe we can do that later I don't know shall we but uh do if if go kcg fixes that do you do we still care do we care why it didn't work with ckcg for academic reason or because you want to have ckcg as a fallback well right now we use go kcg by default know so um we can you can use

ckcg in mainnet uh with the flag but you know by default it was a go kzg so but what we wanted we wanted to find out is that um like why prism can perform to like an acceptable manner with ckcg in the sense that just building the cells and the proofs took you know like in in some worst case scenarios it was like 5 to 8 seconds which is you know not reasonable so with with go kzg um the numbers are a lot more reasonable at least compared to the benchmarks that have been done on why it has been uh bad on our on for

prism we would uh I I've passed Kev some profiles that I took um maybe he has some ideas but yeah um yeah I guess it's just the personal curiosity for me as to why it's not working um the pr the flag that switches between ckg and gokg is currently in a PR um sound we can discuss it more later yeah thank

you uh lar yeah yeah so we U tried uh doing crois testing last week with other clients so lar super not to lar is working and uh but uh we uh so it was successful uh Lota was requesting um so it was not getting all of the uh all of the data columns uh so I for example tried lar superod with I think t superod as well as Nim superod I did not try it with

prism uh so I TR so I did that separately so just two nodes uh devet on kosis and it was not work it it didn't it was not successful I least since both was super Nots it should have been successful and uh uh so then there was uh this change by Barnabas regarding uh CSC in metadata CSC type in metadata and now uh I think we need I need to revert it and then maybe try with some other client that is uh that is compatible uh and then basically move

forward from there thank you and yep um yes J be here um so um in the past two weeks we've managed to merge our D Branch changes into our main branch so we have everything on our unstable Branch now and we've also picked up few minor issues um with Electra so we should be able to testra soon um I think the code is ready we haven't tried um we also upated our back test to

to work with Alpha 5 that's all working we've got metadata V3 implemented so we DEET two ready I think um at the moment the one's working on getting the max BL block configurable and I'm hoping to get checkpoint Sy working next week or so um we've also work on other small things like caching handling small thing um I've done a bit of testing on the fetch fetching blog from El test mentioned previously um so that um the super can fet BL from the e and computer data columns um on of other other notes um

I've tried testing increase BL count I tried 302 blob and tried as well um 32 seems Seems like a bit of a stretch I managed to have a few blocks created um from um proposed thisw at session deadline and that was okay but um I think the open rail Mis rate is pretty high it seems like 16 blob seems to perform much better than the current um yeah I'm going to do a bit more testing on this hoping to work so we can think about having increased Lo count the near future but at the moment seems like the main bottle next is um

the computation time and the network but um I think we're pretty close that's house thank you oh okay oh so uh the bottle neck at the moment is compute cells and kcg proof is that correct jimy uh no so I think it's both the computation and the network bandwidth um so that's going to uh basically going to be how we decide how much blob count we're going to increase to um 16 blob seems Seems okay um with some with the machine that I was testing with so I'm Keen to try out on some some more loow capacity machines to make sure that it also still

works for like four cord and eight cord machines um okay I see so if we in cre stud to blobs um I think the the time takes a bit longer and we've got like sometimes sometimes we'll miss some blocks but um 16 blobs seems bu so far my test okay see yeah so we basically need to get the block uh proposed and published within the 4 second marks so the other just kind test to it right um yeah I guess we could possibly add in that flag for a parallelism because if you can spare

like two to four Frets then um the time to compute like 32 blobs then goes down quite a bit yeah we've already doing um we're already doing parallel um at the blob level I guess you're talking about even more paration with The Blob right yeah I'm still not completely this is sort of out of my expertise but it seems like maybe the scheduler is doing something smarter if it has intra parallelism um I'm still not completely sure yeah you and

then yeah uh we were working on two big tasks uh first is uh implementing for Choice according to the spec this data availability check and we almost finished it but maybe we are going to to refactor it before merging and the other F we were working was uh refactoring custody uh it was too much prototype style before and we want to make it more suitable for for production and also we had changes with custody subnet count uh INR and metadata uh that's all

foro thank you gr hello so yeah the guy that was working onas from the fellowship uh program he he got some progress but he seems still cannot get cannot get granina to SN with with the rest of the C so so we will see depending on when the next devet will start we either will give him a bit more time or otherwise we'll just take over take this uh uh this feature and we'll try to get

it get it ready for the devet so so so that the update is is not very uh happy now thank you and um bus anyone find bus here no okay Al so let's move to the definite discussion so I think banabas can't make it today so anyone can anyone give a quick update of about what this the current status of

defate two is this is it still alive um I don't think it has been launched I think one of us was doing a bit of local testing and he ran into issues I see and um at for def 2 are you using um what what is the size of CSS CSC sorry um I think uh the latest discussions that we agreed that we're going to Red back to the previous u64 metad data um right I think once that PR's merge and then everyone's

implemented we should be ready to to again see so uh was did the def to was launched with uh 604 right so it's there was no change of defate to at all I mean during the past two weeks yeah so as as Jimmy mentioned defet to has not started I think we are still trying to get the local Deets working with curosis and I think we there are we are hitting issues over that and once

the curosis local Deets uh uh basically we are able to run them successfully uh then it makes sense to go towards defet to thank you um so yeah so we have a PR here um which is reverting the previous changes and also add more comments um to clarify the the type of E see so I think I we already got many approvals there and if there's no

objections then I will merge it after this go oh good you okay uh so I guess that's the latest death situation or do we have any other uh topics to discuss around death N I had a question um as to what was the status of uh just client syncing with

themselves I think Barnabas had a message in the Discord about it um so just starting up at devet and syncing just with your um nodes one super node I think it was in one regular node so basically the testing uh so the testing is happening in for example two three steps and the one of the first step is that each client supernode and normal not pair should be working in kosis and then basically we try other we try that with other clients that one one client super nde and another client has a normal node uh so there are these kind

of combinations that sort of we need to try before we need to move move to devnet uh so that is I think what baramas must have posted about and for example LSA super node and LSA normal node uh kosis pairing is working and uh and for me The Next Step was to try Lotsa superod with some other client superod and obviously then with a superod one client super node and then a normal node and vice versa uh so uh basically I think these are the steps We are following and uh so I am waiting for for example I first of all I need to

include this uh new PR 3908 and then I would like to try it with another client whose super node and normal node pairing is working uh that that's the way basically we can move forward on this um for load star does it do verific verification um I think I remember seeing just the compute cells and kzg proofs but not verif verifying like um columns being done yet so I think we have validation uh but we don't do reconstruction as of now in my opinion adding reconstruction is uh is not really a big step uh uh so basically

when I'll have some sort of a so my focus right now is to make sure that there is some working uh uh that working configuration with other client and uh then basically I'll add the Reconstruction step on it but most probably we are doing verification but if we are we are doing the validation and uh I'll recheck although MH yeah I think I might have just missed it then um okay cool um oh so it seems prism is also the same is that also the same for

Lighthouse as in they can sync with um a full node Lighthouse full node and a lighthouse normal node oh yeah um between super my house we can see we can see yeah uh we just don't have checkpoint checkpoint think we working and for as I brought in the chat we can think between super and full n but only from Genesis on the P branch and we have a branch which is called Fix puras discovery on which uh yes prism is able to to think

Super not from Genesis for so so running when starting after Genesis okay after Genesis does that just mean like from a some checkpoint no no no no it seems from from Genesis but sorry I don't know exactly exact for for that I think there one there one you just you you start a new n when the network is already running but not from a checkpoint okay okay I see right that makes sense maybe it worked for my checkpoint thing but I didn't I did not check actually right okay yeah that Mak sense

uh so is that the same for teu as well I'm not sure we have not tested it for a while it worked before but I'm not sure with the current build we should test it it work it worked like two weeks ago okay so it's currently a bit unknown as to whether the one super node and one full node currently works but it was working two weeks ago yeah yeah yeah if if Sy in the middle uh if if it starts in the same time no issues definitely but uh the syn

I'm not sure MH okay okay yeah thank you other isues topics around defate uh if not there any spake discussion not any open topics here

yeah um I want to discuss the the bottlenecks of you know Computing kg proofs for pias in comparison to for 44 so right now in mainnet um The Blob K kg proofs uh computed in the El and are fetched by the CL you know when proposing a but with pias that this thing uh this path is basically uh done in the CL while it is building a block so even with you know just six blobs you you could reasonably expected

to take know up to one second just to compute uh the cells and all these proofs with how main end is running right now do you guys think it's reasonable to have potentially such a large lag before you know you can broadcast the columns out into the network um I I have two questions um first of all why did you mention which layer does what like for the purpose of this discussion what does it matter

whether the proofs are computed by the or CL apart from yeah I guess that's my first question um yeah so the reason I'm bringing up uh the fact that it's computed in the E is because it is out of the critical path so like you know let's say it's a time to propose a block and you know I fetched The Blob transactions from the El the El has all the commitments and proofs ready to go I don't have to do anything from the sales point of view so I just you know uh do engine call get uh the blow bundle and pack it into a block and then you know just broadcast it out so with pias uh it this

changes we get the blobs from the e but we have to you know again compute uh all the cell proof so and the this is not cheap right so if you have six yeah six blobs it can take a while yeah yeah yeah I understand it okay I see it just I was wondering because you know if this is an expensive operation in general it doesn't matter whether it happens on the El or CL because maybe you did your engine call and you got nothing when so I was just wondering why we're discussing it from this lens now with regards to the time I think we also touched on this uh Maybe I'm Wrong

maybe I'm getting this wrong but I I feel like we touched this on the previous call as well where uh Franchesco was making a point uh on that the blobs are arriving even from the previous slot so you can start of like work already it's you you don't have to do the work in the critical 4 seconds window you can pre preor on the blobs you've been receiving on the mle does this make sense didn't we discuss this last time as well I mean I I understand that but like from the C's point of view how would I do that you know so like let's say there's a blob

that comes you know through the execution mle and it's it's the how do I you know before I know the time to propose is you know next slot so how do I know that I have to compute proofs for these specific blobs I see okay I actually don't have a good answer to this uh however it feels like um it has moved with this observation we're moving from a I don't have enough time for

compute problem to a what is the right API problem am I am I right on that or um well well I guess um you could you could frame it that way but wouldn't this be better to be done in the uh El you know rather than I see because we are already we we're doing this right now you know with Ford for for blob proofs our computered and verified in the E and with P we were taking this all out and you know we put this in the CL so it is kind of a you know big change yeah um right uh I I I see I see why you're I see why

you're saying this you're saying this because you're like okay the El sees the blobs at the earliest time possible why doesn't it spend the compute then but um yeah I'm I will have to think about because because it feels like even though the El takes the has the information earlier the CL has knows which blobs should be computed like otherwise the just has to do

everything thing kind of or I'm not sure actually my my knowledge kind of runs limited here of how this should be done but it is worth discussing yeah so I think I think the um the idea to compute um the proofs in advance it's interesting um we hav actually explor that um but like you said could potentially be a bit of waste if the block transaction doesn't get included um the other experiment I've been doing I I mentioned the last call is the um distributed blood building um I think it's probably not a good name for it um because it seems a bit confusing how that solve this

problem but um this approach we also fesh a block from the e menol um so whenever we uh um we receive a block from up here that's like a new block and we can fetch the the block from the e man pool and then once we get all the blocks we can immediately make the block available and then um and then for the super nde they could even compute the Darkon card and P sh Network um to help with the uh the proof building the super have better resources than band to propagate it quicker than like than any regular noes um but the older noes can can try also

try to fetch block from the the E mle if it's already there they they can immediately make make make their um block available and um that's the that's the experiment that we've been doing with 16 block 302 block and it seems to work pretty well so far at the moment with local testing um we're hoping to do a bit a bit more realistic testing more no and distribute set up yeah so like like the issue with you know you know having this super node wait for it is that you have this 4 second critical path and no matter how uh you know highly powered the note is you still have to wait for it to basically compete all these proofs

and you know send it out all in time so um it would be better if you know if we we we can like compute the proofs outside you know the block proposal time because that is what happens right now with you know in mayet I see okay um yeah um it feels like I agree with you it feels like what we have right now is not very well

suited to use the advantage of the blobs coming earlier basically it feels like there is at least two approaches here one is the the one you're kind of suggesting where the the does the kcg computations and then when the CL asks it at the Block proposal window the CL receives everything prepared basically and so the C does not need to do anything I think this is what you're suggesting the other is yeah this is for pus oh sorry sorry

oh no sorry sorry Kev I thought you were referring to something else um okay the other approach is some sort of API thing which I cannot really think about right now where the CL can ask for blobs ahead of time and the CL still does the proof computation I don't know which one makes more sense I also agree the first one looks it's a bit simpler but it feels like these are at least two ways we can exploit the earlier time window right does this make sense yeah I mean um both uh the solutions you know would make sense although for the first one I suspect um we would need like buying

from all the e client teams on to do that right so that might be ack thing uh um I I'm guessing the second one might be easier because uh in some ways we already do this right now so basically from the sales point of view if we know we have the proposal block let's in the next slot you know we send a call to the engine and the engine prepares a payload you know so if in that time you know we can also find out which blob transactions are you know most likely to be included um you know we can just build the proofs beforehand before it's time

for us to propose on the CL but it requires some sort of yeah yeah it requires some sort of API where let C can you know see okay these are the most high value blob transactions in the right yeah mhm right okay um I don't know which solution is best uh neither in terms of complexity nor in terms of you know um more economic Advantage basically so uh let me discuss this later on with

Franchesco and other people who understand that side better and I can report back what was discussed if that works for you okay great Kev it doesn't seem like we would need the CL blob mle I think it me I think the two solutions we're discussing is both in both of them the gets the blobs and then passes them to the CL but uh in different times and maybe different like in one case the prepares the proofs beforehand and the other not

I don't feel like there is a need for blob mle if I get it right if I'm getting this right all right okay I see yeah well um so we'll move it to offline discussion and we have

any open topics to discuss today uh yeah just one housekeeping thing uh babus is on V this week as well as next week so if you guys need something let me know um I'm not fully up to date with P stuff but I'll try my best thank do Perry um I had a general question to CLS about how they control Fred resources um is it basically just deploy as many Freds as possible and have the schedule

um figure out what's the best way to sort of handle the resources um I I looked at a few code bases and I couldn't see anywhere that was like explicitly saying this thing should only use five threads or something like this well for goal L it be done by the go run time so for us it's pretty straightforward you just spin up a go rout routine and um do whatever you need to do and then it's it shifted off into another thread maybe for I think other languages it might be bit more explicit so for Lighthouse we um we

currently don't have this explicit um allocation of um how many threats want Ed for the computation um I think that's something we they wanted to have in the future to have more control over like um the resource allocation because at the moment um in Lighthouse we have this schulle that um basically allocates like thre to different tasks but the kg computation is currently on its own and it can it can do whatever you want at the moment so um at some point we probably want to have some control over it but at this moment there not exessive location right okay um

te in Te we calculate how many cores available and and use uh some function uh uh to to to use part of them for the threads but we don't use in casg parallelism for Block production yet we use it only for for recovering the extended Matrix when we have uh more than 50% of data columns and want to restore whole Matrix uh we plan to add parallelism for Block production okay that makes sense um and I guess for load star I

don't know if JavaScript allows you to have the same access like that XX like what to control the number of threads um for a particular function uh I don't think so but uh I think uh we if there is an underlying Library if there is for example nappy uh kind of execution that is happening uh in the in the in the uh in the library then maybe over there one can figure out how many threats are there but I think

uh Matthew will be uh more knowledgeable on this and I think you are in touch with him Kev okay that makes sense yeah thank you any other open topics oh I've just mentioned one in the chat there's a um open consensus spec PR on clarifying usage of the execution get the1 end points um I'm not sure if um there's if everyone's look at this yet

um but it's been there I want to he if there's any any um feedb on this and whether we could include this um because I feel like that's quite um critical if we want to I mean if we don't have a solution to the the problem that we're discussing earlier regarding doing the proofs earlier um this this is the alternative where we we the CEOs could just fetch the block from the the E menol and immediately make the block available their view so I think this could be useful if we want to sh PE that increase block count uh I had a question so for this engine API call does it only retrieve the

commitment that you asked for can you like ask it for all the blobs um I think at the moment it's just it takes um I think uh version hashes and so if you have the cas commitment and you can can thr out the version hashes and look at both the they get the BL so you can't get any um any block of oh okay because if there was a way to also like you know get all or you know a subset of like the highest value blobs you know it would fit like I guess the situation we were discussing earlier yeah I think that's being

explored but it's it's a bit more um involved because um at the moment the SE land doesn't know much about the transactions um we could like if we really need to we could like that yel can could expose an endpoint that allow us to petch FL by transaction ID but the doesn't currently know anything about transactions gotcha yeah I'm I'm hoping to get some metx on how quick um other notes can um can make the uh the block testable and available before waiting for dark car to

arrive and hopefully you have something next couple of weeks well like for reconstruction um the numbers I've seen it takes quite a while so like what numbers are you seeing with r kzg um I think it's mostly around for 16 BL it's it's around 500 MC to one to one second um we don't have a more granular um metric but at the moment it's both between 500 MC to 1 second um for 16 BL and um I think with this change we we do less of with construction because most most of the know that get the block from the e will be doing the compute cells Co instead of

reconstruction oh okay yeah that's a good point well by the way um I think um so we've got a um Michael's got open here on ref for the executing API and it looks like nether Minds also already mered changes so if anyone wants to test out um there at least two clients that you can test

thank you Jimmy any other topics for today okay thank you for attending have a nice day byebye bye thanks bye bye guys thanks