PeerDAS Breakout #24

Transcript

All right. Um, well, let's get started. Plant updates. And I believe I saw Kimmy on the phone or someone from the White House that wants to kick us off. I can start if there is no light team member.

Okay. Yes. So uh during this week I mainly work on the validation pipeline of data columns. Uh what we do with blob is quite uh standard and what we do with data column is a lot more yellow. Um so I wanted to to clean this. Uh it was my main task also. Uh we have a version ready to interrupt with u grounding and lighthouse for pay with get blood v2 get payload v5 and um and

validator I tried it for 10 epox working um yep that's all for Great. Thanks, man. Uh, Tu, yeah, I'm here on behalf of Dimmitri today. Um, just reading his notes. So, I've been working on distributed blob recovery has been updated according to spec. Now, this the the recovery starts when catching either block and blobside cars.

um and uh run the task of any kind of node including full node and publish the columns we are required to custody seems like and then also started distributing self proof implementation. So adding uh get blobs v2 and get payload v5 and the logic is a work in progress behind this APIs. Then there was a bug in canonical and non-cononical side cars returned on the API side has been uh fixed thanks to thanks from Giron from Sonium and also some work with

Katya uh on the metric side. Um there was a another bug with matrix around the reconstruction with 50% plus columns. The last thing is there was a bug with validator custody requirement calculation and we are fixing it again. Thanks for from Giron on this most almost done the bug fix and will be merged soon. That's it from uh

yeah so last week we sort of paused some of our work in validator custody. Um so we are mostly up to spec uh except for uh some of the latest changes that were merged which is basically the finalized beacon state using that uh to get the validator indices or something like that. I mean I think something around that we we started working on get blobs v2 and we have some clarifications regarding the spec.

I think if Dustin is in the meeting then we can raise it later. Uh apart from that we last week we tested our new column syncer. um for a given scenario for the full node scenario I noticed a runtime crash but otherwise our um like I'm 100% sure that our super nodes can reliably sync using just full nodes and performing reconstruction while syncing um because last week we tested it twice uh with two separate hosts in pset 5 and it reliably synced up to head and I also did notice that it served range requests

to lighthouse get one. So that's pretty much update for me from me for this week. So that's it. Thanks. Um maybe we'll finish with blood and grand then go back to Lighthouse. So on the load start side uh got through validator custody. We still have to finish some uh work on the back filling though and uh PR is up getting ready to merge that and uh we've uh talked through implementations for both of the other two uh updates as far as the um

git blobs v2 and uh notify payload the new one or get payload rather. So we'll work on that this week um and likely have that ready for the next call. Great. And yes. So for the last weeks uh um not much uh progress on the implementation side but we have been working on testing and hardening the sale proof uh integration. So we get it ready uh interoper ready with lighthouse and we

have found a bug that uh cause the node uh to um um at runtime and it was a st overflow when we processing the higher blob count like uh mainly when we process like more than 30 blob so due to it's due to my uh paralyze process this bug. So this thing happens um uh testing with uh without par. So there's no like overflow issue and and we have to investigate it for the on the issue and another uh update is uh it it adds support on the multiple

KCG back ends on the pas. So it is um thank to uh the team on granding that implement the multiple back ends. Yes. And for this week I will try to investigate it further and um working on the syncing issue on the uh Det. Yeah. Thank uh Jimmy. Hey, hello everyone. Um so on the depth net five side we've um we've uh identified a a bug with our our uh DA check. So um probably noticed

that our lighthouse SCF node has been going up back down and and coming back up again. Um it was caused by a um an issue with our lookup. So it it regularly gets stuck and then it has to get into range sync mode. Um, we've got a fix for that, but we've got it um targeting the DEFET 6, so we don't have that on the DENTE 5 branch. So, hopefully that's okay. But if we need to make sure the DevNet 5 um keep running, then we can also merge that onto the DevNet 5 branch. Um, and for DevNet 6 stuff, we've got the selfp proof computation um for request wrapped up.

um done a bit of testing and it seems to work with uh GE Medine and Ref pretty well for a couple epochs. Um and we've also implemented the distributed block publishing spec for full nodes. Um so full nodes we now also publish the the the custody columns as well. We've also uh added the support for max block per per block fulu because previously we didn't have that in our configuration so it just ignores it but that's implemented and it's available on our devnetics branch.

Um yeah that's actually oh sorry uh there's also the uh we've also implemented the validated custody um the very basic so we've got we've got it um implemented with no backfield um and it's still in the process of getting be being tested and um yeah that's all from from LA side. Thanks J. Um Marius or yeah Marius you know jump in with death. Sure.

Um so uh I've been working today to fix all of the um all of the missing stuff. Um, previously my implementation was more like a proof of concept type implementation uh in order to get the uh CLS uh up and running. Um, now that we're moving slowly moving toward definite 6, um, I decided it's time to sit down and and create an actual implementation. Um, and uh, yeah, that takes a bit longer than I I thought because I still um cuz I have to like change the transaction pool and make

sure that we're validating everything everywhere. And um, yeah, but I should be done by the end of today. That's it. Great. uh FLCL or someone from another my team. Uh hello. Uh we fixed a small bug uh in um API and um are participating kind of in Cortis uh setup. Um no more news. uh we we we we want to collect feedback uh uh and uh uh

confirm that we implemented everything uh correctly and uh yeah uh and there are a couple of uh ideas how to improve efficiency of uh engine get blobs v2 uh we can discuss it in the chatter later. Thank you. Uh someone from the best suit. Uh here we go.

Uh Roman had posted an update here from Breath if you want to take the time to read it, review anything there. Uh and then Gader, did you have anything you wanted to add on the EthereumJS side? Yeah, on Ethereum JS. Uh so I was waiting for the PS to merge and uh I will basically uh then implement the new track transaction wrapper uh hopefully this week and as well as uh introduce changes for get blobs as well as the new payload uh the get payload uh new blobs bundle. So maybe next week uh EthereumJS should be able to participate and

support Excellent. Uh, I started the agenda here with a couple of notable wins or merges from last week. Uh, got some positive feedback as you probably heard on ACD last week on the BO concept or the blob increasing schedule. Um, saw that the PRs related to EIP7594 also got merged last week. There's one small draft PR that's open related to it with a few minor revisions. Uh just noting that Justin three or the uh spec team

Justin uh pre-released beta.4 and the API spec got merged related to self-proof computation. Um Barnabas or Pari, you want to jump us lead us off with Devet? Yeah, sure. Uh so I've been testing with different EL and CR clients and uh currently we have a few client [Music] command and wrath. uh gas is uh still currently work in progress and we have uh lighthouse and grenine and the latest

prism image was just built in. I'm going to be testing that shortly. Great. Um was there anything on the curtosis side that you wanted to flag Barnabas? Uh maybe just that spammer is now going to default to 100% v 0 rappers or preflu and 100% uh v1 type rappers post flu but this can still be uh modified menu.

Great. Uh uh oh and sorry we can do electro genesis but you have to specify it as electro for zero and then you can just specify for being bin one. Great. Uh, Sunnyside Labs. Uh, yeah. Uh, so we do have a another report coming soon. Uh, but I think most of the discussions are being done on Discord.

So I think we can follow up in Discord and share the report afterwards through Discord. But I think we can just skip skip it and move on to other discussions today. All right. Um, and Pari, I think I heard you note yesterday. I don't know if anyone from ProLabs on this call, but I believe you mentioned something, some related work that ProLabs working on with GitLab feed one. Yeah, I think the update from them will be postponed to next week, though. There's still a couple of details we want to check before we um talk about it publicly.

Okay, thanks. Um, I've got just notes here of open PRs that remain from last week. I don't know if anyone has anything that they want to flag on the uh salt proof competition uh two remaining open specs or if those are just in the process of review. also grabbed a note from Derek G on the bet be beacon metrics uh noting he'd be interested in seeing how frequently get blobs v2 retrieves all blobs requested

uh just trying to consolidate timely comments into a single location. Uh not sure if any further discussions required there. stand for beacon API and the draft releases. Um, with regards to the open discussion, these were probably mentioned earlier, but uh, I think discussion item one, this was noted out of a loadar contributor uh, slide. Sorry, I'm not familiar with the names. Yeah. Should I quickly talk about it or Yeah, please if you could

tee it up. I I tried to add a summary here. I don't know if there's anything. So, this is about the Yeah, this is about the validator registration. So, this was brought up I think sometime last year or even uh earlier regarding consolidating the existing two APIs we have which is prop prepare beacon proposer and which is used to register the validator for local proposal. So all this really does is uh register the fee recipient. Then we have register validator which was introduced uh with Belladrix when we added builder support.

Uh and the idea is basically to consolidate these. I think one big point was from the author of the issue uh that it's uh prevents an attack basically. Uh so he's running a rescue node where random validators can connect if their node is offline. uh and right now anyone can register fee rep uh recipient for proposal. So that was the initial I think biggest point for uh doing this. Um but I think now like in LOA for example we have two different caches uh for validator registrations for local proposal and for builders. Um the one local as said just has the fee recipient but for builder we

store the gas limit. Um, and another point there was consolidating would also allow us uh to eventually pass the uh desired gas limit down to the execution client. So we don't need to store uh set the gas limit in two different places to to bump it up basically. Um and I think now with pas we have a third basically functionality which is uh for the custody. Um, so I think maybe consolidating makes sense here. I'm not even sure what other clients use. So it looks like Manu on Prism said they use prepare beacon proposals to uh get the beacon node to

know how many validators are connected. But this seems like yeah uh you could even use other APIs like I don't think we should repurpose APIs like this cuz this will just uh cause inop issues uh if expectations on the beacon side are different between doing clients. uh one topic that was discussed there was the eviction or like how aggressively do we prune out the cash for example. So I think if this is not aligned properly and speced out then yeah we'll just get different behavior which is likely not what we want. So yeah it's just uh take a look at the issue. Uh I don't think there are big blockers. There are some

blockers I think that uh from Nimbus uh that Dustin brought up earlier or maybe it was Yeah. Um I I can talk about those at some point. I don't know now or as a bit later. I don't know. Preferences. Yeah, I think right now I I don't want to take too much time. Just look at the issue. Uh how do you implement uh this in in the client like where do you what cache do you use? Do you think like consolidating these caches simplifies things? Uh yeah, that's basically the ask. Yeah, in TEO we use them differently uh to with different caches uh because we also support some kind of

timeout in the register validator. If you if your value to stop calling it, it means that it doesn't want anymore to be using the build the builder flow. So at that point we time out them and only the prepared big proposal site remains on and uh I think like there is also some interaction with DVT because we support some uh parameters to to um to override the value in the register validator in terms of um the recipient but I don't recall at

this moment. So there there might be some interaction with uh with DVT to be to check. Yeah, I think the main blockers would be if the caches are pruned in different intervals um then consolidation maybe does not make sense and uh there's different frequency uh it seems in some client when they call prepare beacon proposal compared to register validator. Yeah, we do that uh probably in different uh slightly different. We are historically calling in different places. Maybe lately we consolidated to

the same slot. Uh but it's it's also executed differently. One is in batch in batches register validator calls is called in batch while I think prepare pre proposer is single because register is more heavy. Uh yes, I think it's slightly different the way we we use the API. The best thing would probably be just

comment on the issue if there are like big blockers. You see, maybe dad had something from the chat. I think one should do registration calculations period to period rather than having too much dynamic registration that could change every epoch. Great. Um, yeah, in the chat I noted for hanging that I've got discussion item two here. I don't know, Marius, if

there's anything that you wanted to add. You felt like the only thing that uh is probably undisussed is the GU approach to get blobs. Uh what do you mean the death approach to get plops? Uh hang are are you able to provide any more context on that? There's like there's no there's no discussion item

in my view. It's just a bug on GU that I'm currently fixing. So I don't think we need to discuss about it. Were there any other timely discussion items that people would like to take the time to discuss? Dustin? Yeah. Yeah. Thanks. Um, so this is relating to and I'm I'm ass sure that people are, you know, familiar with this, but I want to be explicit about what I'm what uh PR I'm referencing here. uh a number of people in this call have obviously contributed

to it. Um so okay so a a couple points and I made a couple of comments and the fundamental thing here for me is it's not at all obvious that the this new object it creates uh is how to say useful or distinct blobs bundle v2. So I I have two or three sort of broad questions in this regard. One of them is kind of naming and it's on the one hand not super important but I mean I will stand by that blob and proof v2 that for the sake of just having the v2 on it. Uh apparently we don't want what the plural

like I get the logic internally but it is not an accurate name. That said it's a name. It's not enormously important. Um fine but the uh more troublesome to me I guess is is blobs bundle v2 the existence of blobs bundle v2 it is not in fact a distinct um data structure and I think there's um how do I put it a a category error here of sorts um so the def it is defined as a specific set of data types array of data etc etc people can read The thing is those array those data types and names of fields and

ordering of fields and everything about it is precisely identical to blobs bundle v1. Now it is true that blobs bundle v1 and v2 have some different length constraints. That's that is the case. Um those length constraints are not part of the type. And so now on the implementation side or even on the spec side conceptually they are not in fact different types and they're not even really incompatible with each other. What? There's no mercalization that's happening on these things. As far as I can tell, just to to adjust Kajendra's point, the SSC container is these are not SS objects and and even if them SSC

objects. So I think this is also with regard to uh the builder API response and if we think that the builder API response uh can come in SS format then this is relevant over there but uh for engine API yes I do agree that for engine API this might not be relevant unless we at some point want to move to SSD based engine API so it would be relevant for that point as well. So I think if we update uh uh the container to a new version it should be fine as such.

I'm not sure I understand like when you say if we update the container to an version it should be fine as such like right that's what's happening now and I'm saying this is useless right now like it doesn't accomplish anything. So this this uh might not accomplish on the engine API side right now but it might accomplish something on the builder API. So if the builder API response is SS response then uh this uh this new type uh would be different because uh the the maximum uh length of the proofs uh array is changing. So in that sense uh it is relevant over there. I think we are

using the same type uh in the engine API as well. But uh even considering forward compatibility with moving the engine API to an SSD format, we should actually basically have this type where the array size is now different. Well, that's sort of the other aspect is it's not clear to me that the the limit as such on on any kind of SSC container is 100% artificial for this purpose. I mean that it's not being merkealized.

There's no hashtree route being done on this thing. Um so it doesn't matter what the limit is except that it has to be long enough. And indeed, if you look at the limit on the KCG side, it's something like 4,96, which is more blobs than anybody has proposed so far. But that's just sort of the theoretical SSE limit in general when when proofs or commitments are used. Th this is this is a completely arbitrary sort of game and you can just define yes and Ethan is working on Ethan has this by the way uh the unbounded SSC unbounded list. He he's uh this is part of his package of things. Exactly. So there's no you can

just define the length of this list as long enough to incorporate both like there's no there's no compatibility issue because right now this SS if we if we back change the previous uh DNO API I guess we can back change there itself the limit itself and make it unbounded over there. So that way I think the issue will go away if you don't want to basically do the engine version. So right I I get I take what you're saying that that there's a builder API is doing SSC right now and and or there

is an option sort of right now anyway uh in much more current development than the engine API and so this is a more salient concern um that this is true uh but at at the same time there's no yeah Um there there's no there's I I would say yeah what I would propose is um make there's a con and actually also resolve a conceptual lack of clarity at the same time which is that right now this blobs uh bundle v2 and blobs bundle v1 actually but they they define some a little bit of

semantics in a way where it's not clear where that semantics are supposed to be enforced right like on spec side that's actually really problematic like to just say they have to be the exact same that I I we can all read the text but later in there when you talk about say get payload it actually does have the enforcement mechanisms and where and how they're supposed to run that's great and like something like that does have to exist but in defining them sort of alongside the spec type is doesn't actually accomplish anything because it it is nowhere instead in the spec where where that is enforced and that means you can get consensus issues. Um and

so as a result like I would say it's also a conceptual cleanup to go back into denoun and say to remove the only thing that actually conflicts which is where it says that the all of these lists have to be the exact same length. Remove that constraint. uh make sure it's uh specified in one of the uh uh calls in terms of one of the semantics of the calls and so those are not randomly mixed anymore and at that point the data structure itself is the same then done like then there's not this pointless identical copy of Bob's bundle.

And you can make the SSSE merkelization limit anything you need because it's these are just on the wire things that sufficiently long is fine and there's no compatibility issue in that regard. You know one thing about uh the SSD on the wire is that we also basically check what is the minimum and the maximum uh payload bytes uh that we can receive even without uh parsing the data. So uh I I will not say that there is absolutely no difference even on uh SSD payload as well. So there could be some impact over there. But again if we just

go back and make sure that uh the list in Dancoon itself is is changed to an unbounded format then I guess it is also a problem because then we'll uh make sure that our clients interpret it in the correct way. Sure. that that's definitely one approach. Um that seems um reasonable. I don't know if the current SSC setup has truly unbounded. There's always some kind of um mercuralization limit built in, I think.

But it can be large enough not to matter. Great. Um, I just want to check with others if there's any other outstanding discussion items or for Dustin, do you feel like you have direction on this? Is this something that uh Francis is required to be able to speak to or just Justin?

I think it would be in what sense? like be required to be able to speak to. I guess I'm not I'm sure. I mean, I I would it would be nice if they could comment given that it's Francis's PR. I don't know if he's here, but I don't believe so. No. Um Um any other discussion topics there? The last thing that was on the list was just this the meeting time for this call. Uh it had been suggested on the Discord yesterday of trying to normalize this with uh ETH clock and the every

other week cadence. Uh trying to get it at 2:00 p.m. UTC for uh kind of a a broader audience. Um, it sounds like historically at one point the time adjustment was due to conflicting with Bosel and I believe the history expiry have breakout calls every other week at uh 2 p.m. UTC. Um, just curious if people are in favor of this idea or if there's any concerns about movie or indifference.

Sounds like there's a internal guest call at 2 p.m. on this day. Marius, do you know if that internal get call is something that could be moved to accommodate this call? No. like we're not going to move this this call for for for this call here. Uh either some people will not attend the internal guest call um and that's probably likely.

Okay, great. Um, I will plan on moving this call to the 2 PM Um, are there any other items that people would like to discuss today? If not, yeah, there like um a bunch of discussions on Discord during the call uh about the response to uh get blobs. Like currently, I think the spec also um gives a response when you don't have all of the blobs. like you'd kind of have this partial response with some of the blobs which isn't really useful. And so there was a bit of a question of do we actually want to have this response versus just like an empty response uh whenever you you have something but not everything. Um so like basically you

only give a response with something if you actually have all of the blobs. Um yeah so that that's part of it and then a bit of kind of uh side things around it uh about what exactly do we want the yel to do like maybe want the yell to be like more aggressively trying to request things once they get a get blobs um request and they know which blobs exactly they should be looking for but I think that's maybe um yeah I mean it could also be like it doesn't have to be like agreed upon it could be each client does something different there uh more about the yeah this response thing I think it's something we should agree on what are the options for the response

right now and what's the status quo uh so I think the status quo is that whenever you have something you give that so like if you have three blobs you give three blobs and and so on um and uh yeah the I guess the one option would be that you basically only give the full list of blobs or an empty response. Um like in any case where you you're missing some blobs. Yeah. Like whether it's zero or three. Is three is three useful at all for the CM? No, it's not.

So that's I mean now at least I know like Lighthouse I think is just dropping the partial response. I I I don't know what other people are doing but u but like I I don't know. I don't really see a reason why you you would use a partial response in any way. Um, sorry. What is a partial response? Uh, like a response with some blobs but not all of the ones that were asked for. That that was exactly Oh, wait. Oh, okay. But like we might not be able to answer all of the blobs, right? Yeah.

Yeah. So the point would be like if you cannot then you could just be responding with an empty response or you I don't know some some response that just says that you cannot um give all of the blobs um but doesn't kind of unnecessarily send like just a subset of them um basically just because a subset of them isn't useful in pure dust like in I mean in the current design in packra it's useful but in beer dust it's not useful. Yeah, I guess yeah, like it's been said in the chat, the partial responses eventually will be useful. Um, but just like not in Fusaka. So, I don't know. There is a question of if we want to have them now just because they'll

eventually be useful. I don't know. Feels to me like we can introduce it when it's useful. But exactly. Feels like we should optimize for what we have today, right? I mean the the API is so if if if you if it's a local API it wouldn't be a big difference but uh but I think we should we should optimize for so adding a flag or or having something later on is shouldn't be that difficult and then you have to get those blobs uh from and and give the partial when it's not useful. I would I would for now I would define it as as uh all or nothing and then maybe outline how we make it more complex API

later on. I also agree just because not using different use cases you just add data and DL is looking up data and everything. So even in future when we'll have an API that will give us uh the columns that are custo basically blobs that API would be different because it's response format would be different and it would not be named for example get blobs or it could be something else. So in that sense I think we should not worry about the future API and the current API empty or full response uh seems like the correct behavior as the time is very

critical at that point when we are making this call any latency is very making this call. So if there is no gain to be had from this API, it would just be better that we have a quick empty response. Give me I saw you had your hand up for a bit on this topic. Is there anything you wanted to say? Um no, it's um yeah, it's been mentioned and it's discussed in the chat as well.

So yeah, it's covered. I just I just want to say that like at least in in our implementations like we what this would require us is to uh like basically we're computing everything and then we have to check if we are missing anything and then we will return an empty response. So yes, we might save some uh some like time during the the JSON RPC uh during the JSON marshalling, but otherwise this is like it's actually more code for us uh

to to to to do this. What do you mean by computing everything? Because isn't the transaction sender the one computing everything to begin with? No, I mean I mean like pulling everything up from the block pool, right? All of our transactions, all of our blocks are in the block pool and we need to pull them out. We need to make sure that they are um uh uh that uh that we can decode them and everything and uh we need to turn them back from uh from from their encoding in the block pool into uh into data and then uh then we we return it to Are these uh concepts that

exist just for guess or do you think this is how everyone's implementing it at the moment? I think this depends strongly on your implementation of the block. Yeah, Marson maybe. Uh yes. So uh for Nethermind uh we have like full blob transactions uh saved on disk. So pulling them is expensive. Uh but we also have uh uh some some parts in memory uh and uh one of the parts is low version hash

hashes. So actually it's cheap for us to check if we have uh all the all the blobs and uh uh and return null response if uh if uh some is missing and uh I think uh from from what I remember from uh before canon fork like Peter from GF have uh added uh is is storing some some part of transactions in memory. So uh maybe adding blob version hashes there uh can be helpful.

Um do do we want to change this in both V1 and V2? I think in blob v1 I mean we do actually use a partial response. So that's not really uh as critical there. Uh in B2 is where we need the whole thing or it doesn't work. Um and I think it should be said that like it's not a big deal on the seal side if we're going to do a length check from our request to our response to make sure that we got everything back. uh but regardless of whether at least in node states and I'm ass sure every client's the same like there is some uh amount of uh burden in order to be able to go across the wire even if you're going locally to the E in

order to pull that data it's still relatively quick I get that but uh but there is still data transmission there is still some performance uh that should be uh at least accounted for uh so if you know if it's us I would say we'd rather just get a null response uh and save the bandwidth and I know it's you know it's not a huge burden in order to check a length but um it just seems like the right thing to do. So sorry am I understanding this right? If one like if you are asking us for the for all of the proofs for all of the transactions for one block and we don't have the proof we don't have proof for

one of the transa one of the blobs in one of the transactions in this block then we will just return an empty response. Exactly. because the only way we can calculate the cells is if we have all of the blobs uh cuz you need all the rows in order to split up in order to actually build the build the columns. Uh so getting four or five is useless to us. So why send any of them? Then block v2 will never be useful. Th this is the whole thing that Ginder was saying and this is actually something I like we have a metric that uh he brought up that currently uh v1 we get uh and we don't pull blobs uh like we don't call blobs

v1 until about 2 and a half seconds into the slot. So we we have a delay period where uh we wait for stuff to come come across gossip. So as much stuff that's going to get pushed to us gets pushed to us and then we'll call V1 in order to pull whatever blobs we have locally at the EL before we start to wreck rasp the rest of the blobs from the CL. And currently we get about 60% of the blocks that hit that that threshold. Uh we get all of the blobs that we need in order to be able to validate the block. So it B v1 is relatively uh effective. But the issue becomes is that the spec is basically written such that the first column or the first or the block that

comes across the wire which is going to be much earlier in the slot. Uh we question whether all the blobs are going to be there. So, like I'm actually starting to look at implementing two polls where we'll have the first poll comes uh right at the beginning whenever the first column or the first block comes across, but we're going to do a second poll uh at that same time where we do blobs v1 right now, which is again it's about 2 and a half seconds roughly into the slot. Uh and and hopefully by then the rest of those transaction, the blob transactions will have made it to the E. But uh realistically like we're going to put a metric in and this is one of the things I was going to ask the other CL teams is that you know if and I know Katia is listening uh if there's

some way in order to do like a unified metric so that we can kind of figure out if V2 is going to actually be effective because I've talked with a lot of people and we're starting to wonder if like you know it's a great and I it's a great idea but is the reality is like are they going to be there because you need it's all or nothing and That's kind of a Yeah, I mean I'm preaching to the choir. I'm sure you guys understand. Uh yeah, I just want to say like something I think it's it's a bit misunderstood about this is like yes in a future design this is meant to be uh kind of become better and partial blobs

are or like partial response are are going to become useful again if we move to some form of cell gossip. For the time being, most of the utility of this is really for local block building. Like it's it's not actually for facilitating um blocks that are built by builders and and the thing is we don't just don't really care about uh what happens when blocks are built by builders. They can just they have the bandwidth to to just push the columns directly to the network. So it does just doesn't really matter if this helps them much in in that situation. I mean it would be nice sure if we also had this uh you know helping a bit in that situation but it's really mostly just about local block building. for local block building there is no private blobs. So that's that's most of it. Like it just doesn't make

sense to look at hit rates on mainet when uh you know whatever like when a lot of the slots that you're looking at are actually slots with u basically me boost blocks and and private blobs because like of course yeah like some of the blobs are not going to be there but that just doesn't tell us anything about local block building. Um, so yeah, basically just I think you really have to like contextualize what is this useful for and like what is the situation that we're actually uh we should be paying attention to to understand why whether or not it's going to be useful. But if this is only But if this is only for local block building, then why don't the CL just call this API only if they're local block building?

Like if we No, no, it's uh it's it's a testers dur during a slot where there's local block building. Like you don't know if the if the proposer was a local local block builder or not. Like it's not about you as the proposer, but it's about like the testers in that slot is to have. And so like when you like when you do local block building, it's not so much that you're publishing on the on the uh subnets for each column. It's that you have to have a peer that's connected that's listening to each one of those subnets. And that's trickier than it sounds uh because we already have a lot of peers that we need to maintain for other reasons. So really the issue comes in is that say for instance like I'm I'm doing a local

publish and I have you know I don't have anybody listening on column 110 or on column 99. So like I'm going to publish them nobody's going to answer that call and those two columns are just going to be missing from the network. So you definitely have to rely on either super nodes that are or or nodes that are custodying more than 64 columns so that they are going to be doing the reconstruction. Um or this is a second backup where uh you could pull it from the E in order to be able to do the reconstruction from the E as opposed to from the CL side. But this is where I I you know we were talking about this again last night and you know the reality is is like there's going to be an awful lot of nodes on the network that are going to have more than 64

columns I think. Now again this is where just going from what we've heard in these calls. So is it like is the added complexity really worth it you know for something that you're not. So, this is why we're going to add the metrics. And then, and that's why I'm also suggesting is that I'm going to do two polls and add metrics for both of those polls because I I wonder if all of the transactions are going to be able to hit the E by time we do the first poll. Like, I I I don't know if that's going to be the case, but metrics will will definitely bear that out.

But I think Franchesco's point was that there's no reason to even look at hit rate metrics, right? I mean, I'm curious. Sorry. Go ahead. I honestly like I don't like I don't know if there is metrics currently and somebody but like do the blobs all propagate fast enough in order to be able to all show up on my EL by time I make that pull at you know 200 milliseconds or 500 millonds into the slot whenever the first thing shows up.

That's the part I think is mushy. There there are people measuring how fast pops are pops are popping up. Uh and blow up is doing measurements on that. Uh they're just not fully ready with that. Um question. Yeah. So I I'm just like I feel like we're like microoptimizing something that is that is really not the bottleneck. The the and it should rather be that the CL is smarter about asking

these questions than us not responding in some cases. So it this would require us to do a significant changes to our transaction pool in order to um in order to support this in a clean way. And if we are thinking that we are going to uh block V3 get blocks V3 quite soon um then it makes probably no sense for us to to to move this especially if we are also saying that V1 is not uh is is

sticking with partial responses. um I can just implement it in v2 but then uh as I said most of the uh just in the endpoint and then I then as I said most of the gains are just lost u because we are still pulling uh up everything from the blob pool. So yeah, I don't know. I I don't think we need to specify this. There's no need to specify this. Uh I think every E can make their own decision uh what they uh what they want to support and the CL's need to be h able to handle both cases, right?

So to to me it seems like what seems to be more flexible is actually to specify it such that you you're supposed to give an empty response and then for example if in g you think that it's not worth it to do the optimizations to to kind of uh not do the work to to like do these pre-checks or whatever like you don't need to do it but you can just do whatever you're doing now and then at the end just send an empty response like I but you know in some other yell it might actually be useful uh to like they might have a architecture it's easier for them do these pre-checks and then they can actually take advantage of this. Like I don't really see why we we wouldn't like I don't I don't think that it can ever make it worse to to specify

this. Like if if your architecture just doesn't really support it being useful, you just send an empty response uh at the end without saving any work. And yeah, another small point is like um the get blobs v1 thing. It's it's not that we would still have this thing around that uh that uh where partial blobs are useful just partial blobs are not useful in fusaka and but the thing is get blobs v2 is the only thing that's meant to be used in fusaka so it's it's more of a like once we get to fusaka just partial blobs are not useful and uh yeah I guess I guess about v3 like yes we eventually want to move to something where partial blobs are useful but like this we're talking about like next fork uh so it's kind of I don't know I I at least I wouldn't And

like we don't even have a you know a clear like uh exact design of how this is going to happen in mind. So I I wouldn't make decisions based on this kind of eventual retransition to partial blobs. Uh Michael then Ari and just to note we are over time. Yeah, I just wanted to mention that we are actually taking a look at how uh blob transactions are hitting the the meool at the and what we saw is that 86% uh of the blob transactions are actually in the public mele and actually 82% of the total proposed transactions arriving before the start of the slot where they are included. So even though it's not

100% I still think that there is something valuable to get from there is like if I'm a local builder or even if I'm a me builder um there is no need to propagate at least not at the same time all the blobs especially because if 82% are already present in most of the mess like I could start just the propagation to the ones that I know that are more likely to be missing at the others meut. So I think that there is a lot of value in having like such a engine API working like I think that it can speed up like not only the processing but also like allow us to save some bandwidth on redundant information that is already present at your local host and is someone uh able or do we need to

open up PR and track this conversation beyond this call? No, it's fine. It seems like I'm the only one arguing against this. So, uh we can just implement it. It's it's uh I haven't heard anyone else uh complaining about this. So, uh I guess every everyone is kind of in agreement. Uh I one followup point that might help. Um, right now do CL's actually wait for all of the response or timeout or whatever and then verifying before they go out on their own gossip layer to fetch u missing blobs or missing columns sorry or is are these things happening in

parallel because if it's happening sequentially then Mario that probably answers your question of give an empty response ASAP because then the CL has more time within the atistation window to go fetch what it needs to um because if you take 2 seconds to respond with a partial response now you've taken away 2 seconds from the CL so this in my opinion is worth specifying um but yeah the question is is this happening serially or is this happening parallelly so in loadar I can answer this that uh some of it is happen in parallel that while we do get blobs uh so basically gossip blobs can still come and uh so basically there is this sort

of parallelism there but we don't start an aggressive uh blobs pull uh unless get blobs return to us. So in that manner get blobs uh latency does matter. Okay. Uh any final thoughts on this before we close the uh Agnes?

So for Nimbus what we do is uh whatever the blob count is u for every every function call of blob gossip validation we spawn a thread and we call get blobs across threads. So we we don't retry after every time out but we retry in parallel and whoever wins like it's basically like let's say if the blob count is nine there would be nine parallel retries for get blobs and there would be one uh blob gossip validation attempt whoever would win would apply the block. So the latency

wouldn't matter for us but but in general it could be problematic if uh this model could be problematic if the blob count is raised uh a lot higher because the CPU utilization would go through the roof. Uh so that needs to be worked upon. Uh so yeah. Okay. Um I will get the meeting minutes published and try to summarize this uh as quickly as possible. Um but I think that is about all the more time we

have for this morning. Yeah, just to uh summarize on the topic, are we then tending towards specifying uh default behavior as in respond with empty or are we agreeing to discuss this async later? I mean, it seems like everybody or almost everybody aside from Marius, I I still love you, Marius, but wants empty response because it just cuts down the bandwidth. Yeah, it's it's fine. I'm already implementing it. It's not a big Thank you.

Okay. Um, then maybe we can agree on who's updating the PR for what the behavior should be. I can try. Okay, thank you. Thanks all. All right, thanks all. Thank you. Byebye.

Bye. Bye. Bye.