PeerDAS Breakout #28

Transcript

Give folks a minute. Okay, let's get

started. How's everyone doing? Uh, KOS breakout room 28. jump in with client updates. Can everyone hear me? Okay, looks like it. Uh, Manu or I'm sorry, is Pawan on or someone from Lighthouse? Doesn't look like it. Uh, Manu, do you want to lead us off with Prism updates? Yep, sure. Hi, good morning. Uh yeah during the last week um we work on the new uh data column by root request

uh for the next devet. Uh also uh yes um we the validator custody still use the head state and not the finalized states to determine the total stake. No, we use the final state as well as um um we did don't use we we we we did use um sorry the balance and not the executive balance. Now we use effective balance as the spec states. Uh also we notice some issues on the sun signets. Uh the first one is is we have

a lot of uh reorg still why I know why. The second way one is sometimes um when there is too many rearchs like rearch at slot n and slot n plus one the node just just gets stuck and which is bad and I don't know why and the third one is um the second one is may not be a pure issue but more gener general issue and the third one is we had some trouble to rethink uh from from genesis and we solved the third M and and we improve the sync uh the sync speed by quite a lot. Um yep that's pretty all for prism

for this week. Great. Thank you. Uh teu Dimmitri uh hi uh not much progress. There were a lot of holidays but we are 100% DNET 7 spec compliant and currently working on BP Fox feature in the middle of implementing it. That's all. Excellent. Thank you. Uh from the Nimbus team.

Yeah. So we are DevNet 7 compliant. We got the new data columns by root identifier in um we had a couple of issues in the sunnyside network I think with get blobs v2 but they are fixed and uh we didn't have a column back fill completely ready uh because we were testing it for devet in devet 5 but then uh like the spec change was pretty big so we didn't merge that backfilling branch. So we didn't backfill uh for a couple of thousand slots. Uh so we didn't have uh columns for some thousand

slots. Uh but then we were fine. I mean we were we were working fine as a super node for node with get blocks v2 with payload v5. Um we have implemented validator custody. uh we have implemented all of the back filling needed for excess column back filling for validator custody. Uh but I think I have a few queries uh regarding that. I'll ask them later in the call. Um we are still working on improving column sync performance. Uh we had a column sync or like sort of a column cache uh that we used uh to validate like the

blocks against them and kind of persist the blocks and the columns gradually. Um so we improved uh that cache. Um so overall super node performance should be uh much faster than before and this should uh consequently affect the column sync speed as well. uh for validator custody backfill and for genesis column syncing we use reconstruction uh to rely lesser on the network because from the experience like every devet uh there are cls who are unable to provide or serve all

columns and eventually sync gets stuck and uh yeah so so that's that's pretty much uh from our end Excellent. Thanks, Agnes. Uh, loadar, we are just about DevNet 7 ready. Uh, we have the last piece is the updated um, Recras format. Uh, we have that PR ready to merge, but we found a couple bugs that we think are related to how we're uh, doing indexing in our DB uh, that's serving over Rec Rasp. So, we're trying to debug that first and then we'll merge in the uh updated uh uh rec rest spec

and then uh we're ready to go. Excellent. Um Grand Hello. So Hungling was working on um on some sinking issues and also he he's working on Deet 7 uh redness. So there are still some things to complete. That's all. Excellent.

Um, I don't see is someone from GU on the call? Perhaps not. Uh, never mind. Hi. Uh, we found uh important bug in a block building. Uh it was uh like uh when we were building a block uh uh when having a lot of blobs in blo like more than 512 we were uh because it's it's like our capture has 500 blob transactions and uh if there was more in

the blob pool we were loading them from disk when building a block. So at some point uh it was causing a time timeout when building a block. It is fixed now and uh it should work fine but it would be great to to have some tests like with with having uh network spam with drop transactions to have more than thousand of block transactions uh in a blo. uh and uh we had uh uh we have updated uh matrix uh we had a PR from uh mini from sun labs uh so it's now merged

so um our our uh gets v2 metrics are following convention and uh yeah that's that's uh all from Thanks, Marco. Um, from refu uh we're yet to disable uh DF on our branch, but otherwise no updates. Great. Um, so anyone from Bessu or Arago?

SA are you able to speak to get updates? Uh just a little. So I I know that uh Marius was working on removing the related things. Um that's the the main thing I think which which happened. Uh he's also looking into into performance related things but I don't have the details. Um I was looking at at more uh get blobs statistics in the meool. So that's that's more on the on the desert side of things not really implementation of of the new standard.

Gotcha. Thank you. Uh and Gender on the Ethereum JS. So this is testing with definite 6 builds because on side uh nothing has changed 7 on that front I found the whole of issues. Uh uh one of the issues was that uh lighthouse was doing get payload retries and on payload retries we were not uh sending execution request again which was which is not really appeared as thing but yeah we fixed that and uh further I'm seeing some more issues regarding uh basically again uh

self which we look at and uh we most probably most likely fix in next two to 3 days. Okay. Uh just a heads up, your audio is a little garbled. I think we caught most of that, but uh just so you're aware and great. Going into testing DevNet uh Ari did, is there anything that you want to kick us off with uh regarding DevNet 7 status? I think Barnabas might have the update this week. Oh, I've only been um loosely playing around with the curtosis configs and I think we have one or two clients

that are sort of working uh but I haven't tried the latest branches today. Okay. And uh Minnie, is there anything that you wanted to highlight on Sunnyside Labs? Yeah. So, we've been running the network for the past week and last week we had some issues uh with few of the nodes uh going over the uh data DB size disk size. So, we had to manually uh increase the DB and restart the nodes and it might be uh rel sort of

related to us restarting the nodes. uh after that we uh noticed some of the nodes uh go like not sinking further and the network is not finalizing anymore. So depending on the plans for DevNet 7, I think uh we can either keep our the Sunnyside DevNet as is for debugging like syncing issues from the client and then we can uh uh recycle the DevNet and launch a new one. Great. Thanks.

Um male Michael is there anything you had noted that you wanted to discuss this recent uh E3 research post? Um yeah, this falls more towards like what SH was discussing like we've been analyzing the get blobs B1 um engine API call. Um and I have some slides prepared if I can share the screen. Maybe it's easier. Yeah. Uh let me stop sharing my and you should have the ability to share now.

Yep. Can you see it? Yes. Yeah. So, it's it's pretty much like an extension of the um research post that we uh presented like few weeks back and the idea is that the previous post was more on the theoretical side like understanding like which of these blood transactions were actually present on the um EL mele on the public one. Um so this falls more into the empirical um heat rate that a consensus client could expect when it's requesting it from the Y. Um some information is that I was running this on my local setup. So just to be aware that this is only like the

the point of view my local setup which I think it's still representative but yeah I was running Netherman in prison. I actually have like a a small fork of prison that will extend like this like more extra information about this call. Um and yeah like the data it's pretty much for the last four days and ideally we'll keep running this uh for petrol also to see if this increases or or changes over time. Um feel free to interrupt me if anyone has any question. Um, so yeah, like something that I was worth that I thought it was worth mentioning is that we still have like 44% of the blobs or blocks sorry not having any kind of blobs. Um, which to me it's always like surprising to see

like how we want to increase the target on the value but we still have like almost half of the blocks without any blob. And if we see the distribution of well this is like the number of requested blocks that the CL ask for the EL but it also means like this is the number of blobs that were included on the block and actually the distribution falls more into the one to three blobs rather than four to six which should be like the expensive side um but yeah like only 55% of the blocks include block transactions. So things that I saw um is that on from this only focusing on this 55% of blogs that included block transactions uh what I was able to measure is that over 68% of those RPC

like sorry API calls were actually responded successfully with all the blocks. So this means that the heat rate is over 68% um like like all this ratio is pretty much like bandwidth that we will be saving from the consensus client topics. Um while only like the the remaining 31% falls more into the partial responses. Um, I'm not 100% if this is only due to Nethermine. Like I I've heard that there are some execution clients that are only responding if they have all the blood transactions. At least Netherman was responding with partial responses, which um once I was digging deeper on the reporting results,

what I saw is that pretty much all those partial responses were only missing uh one blob out of the total that we were requesting. Like this graph over here, it's the distribution of the responded blobs that we got out of the requested ones. And it aggregates by the number of blocks we were requesting. might be able to call you but yeah it's pretty much like when we were requesting six blobs um 40% of the times we got the six of them and around 40 something 45% of the times we were actually like like we got only five out of six uh and this can be also related to to the arguments um there were a few questions raised on the

uh telegram channel that we have and it's like what's why are we actually only missing one transactions like this is something that I'm still trying debug. It could be that those are just private transactions. It could be that my local node uh didn't get aware of those transactions on time or didn't pull the blog on time. Um so yeah, this is I probably will update this on the final report and I'll extend also the it research post. Yeah, still work in progress. And this was also discussed a few calls back like the the time like the time that it took for the to reply to these requests. Um 99% of the time stay within 100 seconds.

Um I thought it was worth noticing then in the case like that people want to discuss about the timeouts. At least we have a plot here. Um it was also as expected like the more blocks you request the longer it takes to to reply. Um yeah but quick summary is that I think that we like I still I still think that there is like a big value on having like these partial requests because as we saw they're pretty much always useful. Um, and if we were able to at least on a pre-pectra um stage, if we were able to push forward those block transactions that we know that were like the last one received at the meool, uh, this could also speed up the time it

takes to process a a block a block on the chrome census client. So yeah, this is pretty much the summary on on our end on the problem. Not sure if there are yeah any questions there. I don't want to jump on the Zoom call and make a loop. Uh yeah, Marson. Yeah, I would just like to add uh that uh about partial responses uh for uh get blobs v1 uh we have partial responses in our mind and we will have uh we have all or not for v2. So uh v1 will will return

partial responses all the time. Cool. Yeah. I don't know which is it. Yep. in the chat. Gadinger noted that 60 70% is still a pretty good hit rate. Um yeah I mean absolutely like um like to me it was just more like remarking that these 31% are still really useful uh at least to process faster like data and also like it all this falls into the lines that the the motivation of this study was to like benchmark how effective could be like distributed block building and I think that the combination of both these studies like

the theoretical and the empirical one show that it would be highly efficient at least with the current rates. Um but yeah like ideally we'll keep running this at least in Spectra when we see that the block count increases and see whether this decreases or increases or if there are any changes in the dynamics of the network. But yeah um Mark is asking can you please go back to the histogram chart I think uh page three. Oh, maybe that one five. I'll make them I will share the link of the slides either way like afterwards.

So feel free also to reach out if if there is any question outside the call. All right. Thanks very much, Michael. Uh, Barnabas, I think you were just joining when we were going uh around the horn asking for updates from Panops and DevNet 7. Is there anything that you wanted to add or provide an update on?

Yeah. So yesterday we started looking into Nimbus and uh Vernine and there were some issues already uh showing. Uh last night we saw that Deku also has a working branch and as far as I know nobody has tested it yet. Uh sorry I haven't been really on top of launching that 7. We are kind of focusing on uh PRA that's launching tomorrow and I think once uh once we have a successful fork then we can fully focus on good and maybe we can target a Thursday launch.

Uh Muel, could you uh mute please? Thanks. Sorry, was that Burrows? So I think we can target a Thursday launch for per is that you're saying and then there was some chat in the or some discussion in the chat about sync testing uh a basic suite for sync test uh regarding I don't know if this is anything that you want to speak to So, uh, so we we have a a quick uh setup for testing uh different clients and seeing

how long does it take for them to sync, but that's mainly for the EL. I'm not sure what Barry was referring to regarding the sync test. Yeah, we've only I only have like a basic spec on my thing. I can share it with you, but it's the one that we still have to build basically. And once Petra is done, we would have a bit more time to work on it. Great. So now we do have this alpha version. So this ktosa sync tests. Um I would suggest client teams at least use that for now. Um and then we'd be refining and making it better.

Yeah, we can we can definitely add support for like net syncing as well. So right now I just have a basic PR that's open for pulley and hoodie and minet and it's going to be going around the different ES and try to sync up as fast as possible but none of it checks for the CS. It's only using a single CL and uh uses checkpoint sync see the CL because we mainly want to collect the statistics for uh the side I don't think we need to worry about genesis sync to be honest on the CL side so unless anyone would like explicitly want to genesis

sync I think maybe by tomorrow or Thursday we should have uh checkpoint C instance or fully support as well then we could just do checkpoint syncing uh on all the future full networks. Great. Um I guess moving on to open spec and EIP discussions. Um Justin, the first two looks like were merged recently. I just wanted to see if you wanted to discuss anything with these before I remove them from the list. Uh, nothing to discuss on those

two. You can remove them. Great. Um, and then I know that there's just sort of a I saw an overall call to action requesting a review of the blob schedule. Uh, you're hoping to merge that on Friday. Gabe, I don't know if there's anything that you wanted to say, but I've got all four of these related PRs here. Um just trying to move forward some closure on the blob schedule. Yeah, cool. Thank you. Uh yeah, just please review that. I would like to merge it after uh making the upcoming stable release. So it'll be included in the next alpha release for the blob schedule PR that is

and I would like to include the PR from uh JSON site uh regarding the blob side cars deprecation support. uh planning on merging that like right now unless anyone I suppose 4286. Yep. That one. Okay. And then the lastly uh I think you noted your next release of the consensus spec you're going to push tomorrow a couple hours after pector goes live. Uh yep. Uh v 1.5 uh zero. Yep. about an hour after uh well after we're sure that the upgrade

is successful. Excellent. Um Francis, is there anything that you wanted to discuss on the uh beacon API? Um yes, I think everybody like all the team is here. I really would love to get a kind of like a direction forward for this PR. Basically the idea is that either we stick with the current get block sidec cars uh PR to change its output to not return the proofs and etc just return the data or I think as like a gender proposed we just deprecate this

p this like function entirely and implement a new one called get blobs that only return the blob data. Um, I'm fine either way, but it really would be great that we can kind of align on a decision here. I believe Gender and Dimmitri uh also voiced support for the new get blocks API, but would love to get thoughts from for example Prism, Lighthouse and Lowar or Dimbus. Yes. Prison, Lighthouse, Loar, Nimbus.

Any thoughts on that? Um so barring any feedback uh Francis do you are you able to take action?

Um I don't think so. Um I think we just want to kind of like decide on which direction we want to go. If we are going with the current direction, just change the get blob side back. If there's no objections, we can just merge it. Um, but if people want to go with the other direction, which is change to get blobs, then I need to change the spec and have everybody review it and then merge it. It doesn't quite make a big difference. It's mainly a like user experience and like client preference side of things. So I I I kind of like want to make a quick decision on that and not think about it anymore.

Manu Dmitri Agnes any concerns with uh that being merged? I don't really have any concerns. I just think uh where we are with dev velocity is going to be a little while before we can actually get it implemented. Not sure if the other teams are in the same boat though. Okay. I see a no concern in the chat from many of their Francis and an affirmative from Agnes.

Okay. Can I just confirm the affirmative is on merging this PR or go with get blocks. Okay, seems like it's merge. Um anyway, like we can merge this PR if we want to go to get blobs later, we can like change it. But uh I believe it has some benefits to merge this now so that everybody can at least like have time and if they have time to start implement it. Um okay, great. Uh was there anything that you wanted to mention on the builder spec uh change?

Um is Alex on the call? I would love to see if there's any feedback from the MUV community. Yeah, not yet. Everyone's pretty busy with Pekra. Um the consensus so far just from some informal chats seems to be people are very interested in changing this endpoint. Uh that's kind of the last open question here. Um yeah. Um great. I I think that makes sense. Um just curious if we could maybe put a some kind of deadline on that. maybe two weeks after petrol we can make a decision. Do you think that's

reasonable? Or is there any way that we could potentially speed this up? Yeah. Well, okay. I mean, one option and I guess I'd be curious to hear from people here like do we just want to merge this in as is? Is that helpful for development or would we rather keep it open until we get to the final solution, let's say? Um my current feeling is that uh having kind of implemented this like in prism the current spec basically requires returning both blocks and execution payload and due to like and based on the current discussion like we either only return execution payload or

return nothing. I feel like based on the amount of changes and API definition stuff, it might make sense to implement it after we kind of like nail down what exactly do we want to return. Okay. Yeah. Looks like in the chat Francis, Justin, and Matthew Ke and Gadinger uh and Manu all agree to merge it and update it later. Well, Francis is saying, yeah, he's saying the opposite, but yeah, whatever.

I think fine like if you want to merge it now and if clients have time, they can implement it, but expect some changes a little bit later. Yeah. Okay. Um I think I'll lean towards Well, okay. I guess I'll think about it. I I don't know if it's really going to change much to merge it now versus later. Um because again I don't think anyone like I guess and this is what I was asking like is anyone blocked on this right now? I would guess the answer is no because they're focused more on core pure things and not necessarily the builder flow right now.

I don't think that's Oh, so sorry. Yeah. Okay. I don't think anybody specifically like blocked on this. Uh although I would love to have kind of like Yeah, it's fine. Like I we can test with the existing kind of like structure and there's some changes that needs to be done on the builder side for like integration testing as well. So the quicker we can get this kind of like uh finalized the quicker everything can be ready for like for official testing. But right now I believe Justin uh has done some changes for the like builder side of things and uh we could start testing with the existing like PR spec.

Yeah. Um okay. So then should we just go ahead and merge this and then people can work off of that if they want to. Sound good. Francis. Yep. Okay. Okay. Yeah. I mean, any modification we do might even be optional for relays. So, yeah, like there's probably a little more design work um for the git payload endpoint.

So, we do know that, you know, assuming you have enough bandwidth, it will work as written right now. So it's not that we're merging like a incorrect solution. Great. Okay. I can probably merge this today. Let me make a note. Are there any other open spec or EIP related discussions?

Um, if not, there's a few things I flagged in the open discussion just from the ACD call last week and from the chat, the Discord chat. I don't know if I'll just kind of surface them here, but Agnes, you had uh posted a link to I believe a proof of concept of implementing validator custody. Oh yeah, sorry your uh audio is cutting out there.

Yeah, I I do have Is there anything that you wanted to say on this? Yeah, so I had like a couple of questions like at least five. Some of them may be a bit naive but I think I would ask them anyway. So um there is a there is a fair amount of time uh in Hima 2 where the blocks don't go into the database and uh they are in a cache uh but they have passed DA and let's say at a time stamp my DA is 10 columns and while they are in cache I detect that my validator

custody bumps to 15 columns or let's say even worse let's say eight columns um are my blocks valid now because u in that way I have to literally uh tune the cache to be a function of time uh and at every slot I should record validator custody and then I think I should probably claim that okay at time stamp so and so my custody was 10 so I'm going to follow 10 so I need some clar ity over this and I think it may be it might be different for other CLS but this is one design challenge architectural challenge that we are

facing right now I understand that you should make updates at least every two epochs yeah last finalized yeah that changes something okay um okay that's the first question Um, do I need to halt validator duties while refilling excess columns or can I keep proposing a testing and refilling in the background and once refilling is done, I'll update my ENR. So, do I need to pause? Hi. Do I need to please? Yeah, sure. Come to the elevator.

Yes. Sorry, Nazar. Was that intentional answer to the Sorry. Go ahead, Agnes. Yeah, I I think somebody answered. I couldn't hear. Okay. Okay. Okay. Okay. Yeah. So, um Yeah. So, let's say we increase it and um uh do I do I really um not perform validated duties that time? What should I do while refilling? because refilling can happen every if it happens every two epochs um like if I pause then I basically I'm I'm always paused uh from

proposing a testing so what should we do uh Demitri noted in the chat that teu they only increase they don't decrease and if a validator was removed they ignore it and then recheck the requirement or v1 epo epoch okay Uh okay. So I think I can assume that I will update my ENR u while refilling is done and uh and I can I can basically refill excess columns in the background keeping a test like this. So anyway just three more questions. Does

validator custody change in a non-finalized network? You know, right? I mean, should it change in any way? Uh, no, I don't. Okay. Uh, okay. Okay. The next question is, what if I update my validator custody? Um but I don't refill and I don't advertise new ENR and uh but I but I subscribe u to the additional subnets from that slot onwards. Um but since I have never updated my ENR, I am never uh really discovered by somebody else some other CL that I'm really malicious at my ENR

right now or like maybe I'm advertising a subset. Um, so I can practically get away with never advertising the updated ENF and still have honest validators attached. Well, technically it wouldn't be an honest validator. I mean, the specs state that an honest validator should automatically update. Oh, okay. But like can there be a way of penalization? Um, not really. Like how would that work?

kind of a complicated problem. Yeah. Uh yes. Okay. So, and uh Okay. So, the other one was how often in a practical network can the validator custody change? I think it's every uh block or so. Yeah, that's pretty much uh all of the questions that I had. But um I think u like I think there should be I don't know how or maybe If there is a way in which we can um basically check advertised um ENR like update or like whether it's increased or not by like maybe checking

uh what subnets the other node is subscribed to. Can we do that in any way? I'm not very sure. But technically like we can get away with not updating liners and okay that's the honest okay yeah I mean these are the it's worth exploring some of these ideas but I think they might be complicated so we can look into it. Yeah. Yeah. I think it's it would be great if you can look into that, right? Um Gajendra is right. It might break break privacy for now. So like let's say if I'm if I'm a minority client, I can never ever ship this

feature and I can still participate anyway. Okay. Yep. You can participate. You just wouldn't technically be an honest validator like There's a ton of things like this. Ashley, [Music] uh maybe just another question. Um can there ever be a room for a reward? Um if uh so so there is a very significant storage uh cost or difference if my validator custody is 63 and 65.

because I'm assumed to serve all columns. Um or I think Yeah. Yeah. So, but is there is there any um is there any additional reward that can be thought of later for having a validator custody above so and so number or something? Yeah. Is there any like has has anybody ever thought about it because it's kind of very altruistic.

Uh yeah, I'll just say I think it's unlikely that we'll have an extra uh award for that or reward for that. But I get what you're saying like it's a lot of extra responsibilities and data bandwidth the whole thing. All right. Thanks Agnes. Appreciate all the questions. Um and then I think Stokes you had noted on the issue uh just trying to arrive at some consensus whether or not validator custody will be implemented in purest devet 7.

Yeah, I mean we had discussed this some with the timeline last week. So I would say people can implement it. That would be good. I'd probably lean towards including it in the spec and we can always take it out. Um let's see. I think also yeah it should be fine even if it's like part of the spec but you don't support it. So like the network should still be good. So yeah, Barnabas is saying this in the chat. Great.

Um and then yeah, just going back to the plan as discussed last week was I think people were noting to ship Gudos DevNet 7 sounds like uh towards the end of this week and then a DevNet 8 which would include the BPO feature um and then move on to Fusaka Devet Zero based on the DevNet 8 spec. Oh, yep. That makes sense to me. And then I think in the testing call yesterday, um there's discussion brought up about whether or not uh continuing this call makes sense or

if the meat of this call just moves into ACD. I don't know if there's any thoughts or feeling on that. Um and when and if that change would happen. Uh, I would be in favor of moving this to the uh, testing call on Monday. Just my opinion. Oh, sounds good. Uh, Mario.

Hey. Hey. Um, yeah. Um, normally that call it's lead by Barto and myself. um on Monday. Um I think you're doing a great job moderating this part of the discussion. Um how we should like and you keep track of the of the issues that are currently u surfacing. Um do you would you uh help us like uh bringing up the the discussion topics for peer like this or should we take up on that? Um, other than no, I'd be I'd be happy to.

Uh, yeah, more than happy to do that. Um, yeah, just as also question for you, Barry. How would you feel about it? Yeah. Uh, I think that would be perfect. Also, in the past, we had we were always Danning Tago uh tagging Dano in when it was the EOF topic. So, we can continue to do that for Will and then moderate around that. Yeah, exactly. Like co-coordinate. All right, next week this is kind of the next fork to ship. It makes sense to move most of the discussion not on the pas breakout room but rather um all core devs testing.

Great. So, I'll just plan to remove this calendar, sorry, this meeting from the calendar moving forward and we'll move these conversations to the Monday call and uh work to consolidate like the same issue list and things for that uh agenda issue. Y sounds good. I'm good. And we should also mention this in ACD this week so that everyone's aware of the change. Makes sense. Uh I'm going to open up the uh PM issue for the Monday call and then I think you can will you can leave the the comment for Pieras specific topics

in there. I'll send you the link once I created the issue. Great. Uh Francis um hi. Yeah, I just have one question regarding timeline. Uh if you are doing DevNet 7, DevNet 8 and then Faka Devet Zero, um is the plan to launch Devet Faka Devet Z0 before the Berlin interop or does the plan change a little bit? I think our aim is to still do it before the Berlin um interrupt event, but I also don't really see a reason for Pieras Devet 8. um that can just be Fusaka Devnet zero and I mean considering that EOF is

not a part of Fusaka anymore um the Pas devet is kind of the Faka devet anyway yeah I kind of agree like we could potentially like test like infosaka net0 as well but I think it's technicality like the naming thing okay Excellent. Um, all right. Sorry. Can I just say one thing? Um, yeah.

Uh, from my current observation, it seems like I I I think like clients are getting more and more back to this, which is great. the the situation currently that there I think there are some kind of uns specified edge cases for example for by data custody how do we back field and how do we uh custody columns like based on the balance change and if there's like four three advertised uh columns etc and those things make things super hard to debug and uh to figure out like what is actually happening I'm just curious like from a learning perspective And from all the like experienced client devs, is there any way that you do that can help

speed up this process? Like make for example make reproduction easier like make linking who is the custoing columns and who is kind of like missing the columns like much easier to visualize and debug. I'm just curious if there's any like suggestions on that. Uh I think it might help to get better tools for visualization. Um one of the things is that ProLelabs is working on a tool to help with exactly this. Um so that we can sort of probe which node is supposed to have custody of something

and get a response and be able to proactively tell when something is going wrong. Um I'm not sure about the status but um I see Jiannis on the call. Do you want to maybe give an update on that Jiannis? Um yeah sure. So we do have work on this uh kind of mildly mildly started but we haven't done much progress yet. Uh I think we can uh we're planning to have that delivered within the next month or so. maybe uh a little less awesome. Um if you guys have a repository or some open tasks etc maybe you can also just share that and Francis

if you guys and team have some spare capacity maybe it would be good to collab on that. Yeah, may maybe it would be good to catch up then and perhaps schedule a call. I don't know if your availability has changed, but last week you were asking for opportunities to jump in. This might be something. Yeah, sure. Um, I'll keep track of that. Great.

Um, are there any other discussion items for this week? Marson, I have a question. Uh, uh, are the blob schedule changes in a scope of devet 7? I mean this max max count of blobs per transaction. Uh, no, the blob schedule stuff is not in defeat 7. Okay, thanks. That's my understanding. Right. Right. But the goal is to have a final review so you can merge that by Friday.

Correct. for it to be included in the next DevNet. Awesome. Well, unfortunately I will be asleep when Hector goes live tomorrow, but I'm very excited to be a part of the team for this first hard fork under my belt. It's going to be a big one. Best of luck to everyone.

It's a lot of work went into it. All right. Talk to you soon. Bye everyone. Thanks. Bye.