PeerDAS Breakout #26

Transcript

Hi. Hey, R. Hey. How's it going? Good. Uh, R is a new contributor. Uh, I will leave it at that and I just wanted to give him an opportunity to introduce himself to the rest of the the call. Yeah, absolutely. Thanks. Thanks for that. So, my name is Raul. I led the libby tob project both on a technical and strategic level between 2018 and 2020 while I was at protocol labs. Um I think I've collaborated with with a ton of you um over the years uh specifically around the time where ETH2 chose to adopt libp. So definitely have some some

good times and and war stories there. Um well the protocol labs I also led some other relevant projects I think um around like test ground the falcon virtual machine and other things. So so yeah I think those could be relevant in in different ways to the work that we do here. Um and I've just joined the the Ethereum Foundation to help advance and and accelerate peer-to-peer efforts. Um so we'll be working very closely with Marco um who also joined recently and and a few others. Um the the scope and the mandate is pretty broad. Uh but I'm very excited about the outlook and and the opportunities around long-standing topics that we've been we've been kind of like a little bit sweeping under the rug for for quite a bit of time like

things like privacy censor censorship censorship resistance interop and and a bunch of other threads as well. But of course um the immediate focus here is um is DA and blob scaling. So, I've been listening in calls and reading a ton to to get up to speed with with all the efforts there. Um, so I'm going to I'm going to be spending a few days getting the lay of the land and then I'll start making contributions across threads. I've already made a bunch around for example generalized gossip gossip sub and a bunch of other things. So during this time if you wanna if you want to bring it anything to my attention uh whether it's specs designs implementation strategy wherever you think I can I can be of help uh feel

free to reach out to me on on telegram um I already posted a message on on the telegram group for for this effort. Um, so you've got my details there and yeah, I'm happy to to connect and chat one-on-one with with every one of you to get a firsthand feel for where things are, any difficulties, risks, uh, timeline, uh, complications, testing, challenges, anything where uh, me and and and Marco and others can can help. Yeah, it's pretty much it. Very exciting. Thanks so much for Marco.

Welcome. Also, Marco, did did you want to give a quick introduction? Uh, sure. Hey, hey folks. I've been lurking here for for a bit. Um but yeah, you guys might know me as the Golo P2P maintainer and recently just started focusing on Ethereum side of things and just trying to understand I guess the lower level details of Pure DAS and seeing how how we can optimize that in the short term and then in the longer term there's there's a lot of stuff we can do.

Brilliant. Thank you. Glad to have you both here. Um, I will share my screen and if we want to uh jump in. I don't believe anyone from Lighthouse is on the call. Um, I'm ready for tomorrow. Oh, Paul on. Hey. Uh, yeah. So, it's been like a slow week for us. Jimmy and Lion both are off. So, I've been mostly working on trying to fix some of the sync bugs that I've found and also uh been working on uh some other metrics related issues. Uh and yeah, I've found a few sync bugs

with Lighthouse where we were requesting data columns from peers who were not synced. So, I've made a fix for that. And I've also found that some issues with other clients who were not serving data columns by range properly and I uh messaged on the pod testing channel. I think it was Nimbus uh loadar and uh grand uh so yeah mainly been trying to fix sync issues this week. That's pretty much it.

Great. Thank you Paul. uh Manu or someone from the prison team. Hello. Uh I'm Radik and uh yeah I am here today because Manu is out of office for vacation. Um seems like just the day before everything went uh down so before he he left. Um and um yeah so I have an update that he shared with me before going on u on vacation and um what he's been working and ference have been working on recently is um okay so working on backfill um but

it's still not yet finished. Then there were some disconnection bugs where when we were completing initial sync and this has been fixed by Terren. Um then we also properly implemented the blobs bundle v2 structure for communicate for um getting um data columns from the execution client. uh previously we just uh there was a hack where we used get blobs v1 and extended the maximum ss size of the strct.

So now we have the new uh v2 structure in news and uh also a bit of unit test and spec test that were missing uh got added to the branch and tangentially manu started slowly merging chunks of the pure branch into develop mostly around uh kg library and uh our RPC structure. So nothing core uh just yet. Um and also I saw a message recently uh from Manu where because of some um refactoring that he did we managed to uh decrease the verification time of

columns by 10x. So so before it was like 100 milliseconds and now it's around 10 milliseconds. And I guess that would be everything for our update. Excellent. Thanks, Rod. Uh Demetri, uh the it was so weak for us to uh we have finished and merge distributed cell proofs feature both block production and execution layer reconstruction with get blocks with two.

Now we we are working on builder flow. So local production is done but builder is missing for looking to finish soon. That's all for us. Great. Thanks Demetri. Uh Agnes DM me his update sound like both he and Dustin were not going to be able to make it to the call today. So yeah, their current focus is investigating issues uh related to DevNet 6, improving column syncing for Nimbus E2.

Um Matt or the load. Hello. Uh got uh get Bob V2 implemented uh implemented the first poll uh for u the first object that comes across. Gossip got a PR approved and ready to merge for the second poll that happens a few seconds in the slot so we can understand if there's any discrepancy between pulling uh right at the beginning or waiting a few seconds. Um also got get payload v5 uh implemented uh that PR's gone through several rounds of review.

It's just waiting for uh final blessing but uh it's basically ready to merge as well. uh caught a pretty nasty bug with our uh super node flag getting overridden by the validator custody that we implemented last week and actually I think Agnes uh brought up that uh he saw that as well or actually it's Pawan um but we had already were already starting to work on it so uh getting that knocked out and then um put up a couple more PRs uh for a refactor, a pretty big refactor we're doing on just our uh data availability pipeline and uh maintainability unit

testing. Uh so was a was a very busy week. Uh we're still having issues obviously with the DevNet. Um and I have to run uh for uh a pretty important call I need to take. Uh but uh uh looking forward to hearing about resolutions for uh the DevNet. Um, so, uh, good luck. Talk to you guys soon. Great. It's on the agenda. We'll circulate notes with you shortly, uh, so that you get all the context you need.

Um, yes or someone from the Grand team. Yes. Hello. So it was also a slow week for Grandina in terms of PDAS and uh uh Hungang returned from the holidays and he started to look at the various issues that was reported uh last week. Uh but I I think there isn't uh apart of the fact that uh that the work on fixing stuff started. I don't think that we have anything significantly significantly fix fixed so far. So that's the update. Uh the small one. Excellent. Thank you. Uh anyone

from the death team on the call? Okay. Uh I think I saw Yeah. FLCL or Marson from another mine. Uh yeah uh I'm working on some uh potential optimizations. Uh it might be better to have this network part as s container actually instead of bunch of um uh run p encoded proofs blobs and so on. And uh uh uh I'm trying to learn uh whether it will be more performant uh uh

to uh store all the data being flat instead of uh uh separate uh by trace and stuff. Um but uh um like in general we have a branch which is ready uh and uh is under review. Um, what else? Maybe Martian has more news. No, no update from me. Great. Thank you. Uh, Ramen Roman, do you have anything on the rest?

Uh, uh, nope. So on Friday I restarted all of the RA quine pairs and that's how I got them unstuck but uh I did not I did not get any more insight on on the thinking issues because I was not able to reproduce them. Sounds good. Um anyone from Bessu or Aragon on the call? Uh Gader, do you want to do you have an update for EthereumJS?

Yeah, I will. Yeah, thanks. Uh so basically uh as mentioned last time uh I've did uh I worked on the PR uh on the basis of CKZG and uh I just I'm mostly done. And I need to add some test cases uh so that uh EthereumJS will be able to participate in the devet and uh and once microsigner is ready then we'll remove ckg and shift back to micro signer uh library for handling the cell functions.

Uh but yes uh hopefully this week uh we should be able to start participating in the dev nets. Sorry muted. Um excellent. Thanks Gajender. And speaking of that that's Ari Barnabas. You want to lead us off? I know that you've got uh some items that you want to discuss regarding the current status of DevNet 6.

Uh yeah, I've been digging up uh digging into DevNet 6 earlier today and maybe a quick TLDDR. um about the end of last week so I would say 17th or 18th April uh the notes started degrading a little bit and the root cause at least one of the root causes that we found was out of memory issues that seems to have started mostly with death and death on the 17th of April however um OOM itself wouldn't have been a big deal um the problem seems to be that every time G om it started sinking about 30,000 blocks in the past. So even though head was around 40 something thousand uh blocks get was

going reverting from 18,000 and syncing back to head and sometimes it was during this reync again oming and going in a reboot loop or om loop. Um however there were some periods in which most of the get nodes were syncing up the head and that leads to this sort of spiky behavior we were seeing with finality. So we do hit finality sometimes and sometimes we didn't. Um things started to get really bad once um this happened I think three times and I think the third time um Netherman joined in the party. So we roughly the 18th of April.

Um and then we started having Nethermind Os as well. Um this led to basically most of the network being offline and a lack of any good slots. Uh that combined with the bugs that various CL had in either advertising or sync meant that there was just no realistic chance that we were um healing the network itself. Um it seems like the first two times the network did heal itself. So it's roughly the third time when u more things go wrong that we uh see bigger issues. I think every CL probably already has a list of things they have to work on slash already have worked on. So if you have PRs you want us to deploy please

let us know. Um, however, one thing I do want to mention, it doesn't seem like the API on all clients is um as reliable as would be expected purely cuz when I started debugging earlier today, there was just um yeah, API would say something, log would say something else. Error codes are all over the place, not standard. And I think we should spend some time fixing this. I can make a list if required. Uh so yeah the TLDDR uh devetics is quite broken. Um one of the fixes would be bumping up the RAM um on all the machines to 16 gigs instead of 8 gigs.

However, 16 gigs of RAM on a weakish dead um DevNet is kind of a lot. Um so we do have uh just looking at the chat we do have rats uh restarts as well. So if you go to the root cause pure devet 6 thread, uh there's the OOM log. So that's the output from um the message pass on Linux as well as a script to parse it for whatever date you want and it'll give you a combination of which clients have um how many restarts on a certain day as well as which hosts have how many restarts on a certain day. Um, so you should be able to independently check what's wrong with your client and everyone should have uh SSH access to

their machines. So if you need PRO or whatever to verify what's going on, uh, happy to provide that to you. Um, yeah, short-term fix we could try is bumping to 16 gigs of RAM and then trying to resync a few nodes. I think it's worth a try. However, um yeah, that would note that 16 gigs of RAM for a weakish devet is kind of absurd and is way more uh bandwidth required than before. And I do think we need to fix the we need to fix OM handling to be a bit more graceful than this.

Great. Well, thanks for the update, Ari. Um, anything that people should keep an eye on this week other than that thread that you shared? I think the big question is if we then just aband like beyond the basic trying to recover DevNet 6 if we should just abandon it and do DevNet 7 cuz I'm assuming there are enough fixes and a lot of noise that we're not going to find everything anyway. Gotcha. Great.

Um, so we jump to Sunnyside unless anyone else has questions or feedback that they can share regarding what they're experiencing on DevNet 6. Okay. Uh, men here. Yeah. Hey guys. Uh it's been a slow week for us as well. So we don't really have an update report like before and we are running the DevNet 6 specs with like 70 nodes. Uh but we are also waiting on the like the fixes from once the DevNet 6 issue is resolved. But we will continue running

the DevNet 6 notes uh and report on discord when we get to dig on dig into some misbehavior of some of the nodes. So right now it's just running and we'll continue to monitor and uh get back to you guys when we find out something wrong with it. Great. Thank you. Um, I flagged a couple of things for the spec uh discussion session section and then I saw that pop added something down below.

Um, I asked Justin about this uh FLCL there was a conversation that was in the discord. couldn't tell if this had reached conclusion or not that regarding the formatting for blobs bundle v2 is there any open discussion there that you wanted to uh surface for the rest of the group um uh kind of uh I I'm not yet sure uh we want to make benchmarks uh to uh confirm whether it changes at all anything. Um but I can describe briefly maybe someone will be

will find it interesting. Uh so um when we include the um network part of blo uh transaction encoding like when it's encoded in uh networking form we have uh commitments blobs proofs encoded as LP strings uh in arrays and you have like uh six,000 uh uh and something of them. And uh uh for uh in managed languages it means uh a lot of locations which not a big

problem but um then we want to copy this data um in a flat structure and flat structure to pass it to CKG uh and uh it means more copies. Um so if uh uh this uh data uh was passed as a like SSD list uh of um such fixed insized containers of commitments, blocks and proofs.

Uh maybe it would uh or three lists of uh of such vectors right it would u uh allow to save on copying and um allocations uh but uh it's a big change and uh uh still uh I have no proofs that it will be uh way more efficient. Yeah, that's it.

Okay, great. I just Yeah, wanted to give you an opportunity to talk through it. Um, okay. And then another item, uh, I guess this was discussed on ACD last week. Uh oh yeah there's regarding the CLB uh BO blob schedule. Um Justin shared this and let me actually do it this way. Uh yeah, I I suppose wanting to make sure that you know confirm agreement on

whatever was discussed. I I missed the last a uh ACD call, but it sounds like there was sort of a soft agreement reached for the configuration for the blob schedule for both CEL and E. So just uh trying to get make sure that we have an opportunity to talk through any uh last second thoughts or uh flags for us here if you don't believe that we've reached consensus. And then uh Justin wanted to discuss a PR that he just opened related to that. There was one particular comment that you wanted to flag uh asking whether or not this should go into DAB which I believe is further down. Justin was there any voice

over that you wanted to give on this topic? Uh sure. So yeah uh last week we discussed uh the structure of the blob schedule configuration being a list of records and that's what we have implemented here. Um yeah, could you click on files changed? You can just kind of quickly see what this might look like. Uh scroll down just a little bit. Perfect. So uh that's currently what we have. We have uh added the net to the schedule even though it's not uh necessary. It in my opinion feels weird to leave it off though. And then we've included three uh example BPO forks for 18 36 and 72 blobs at undefined epochs.

Right now uh yes uh Barnab Bar Barnabas the minimal spec was different than the mainet or the minimal configuration is different from the mainet configuration because on minimal you can only have 32 blobs per block. That's the maximum that's defined by like commit uh max commitments uh per block or whatever. So, uh the limit is 32. We can't do 72 or 36. Uh yeah, I mean I think this looks pretty good. Uh, someone asked whether or not we should uh pretty much just change the DLAB specifications to use the like get max blobs per block

function which would use the blob schedule. Uh, I can see pros and cons to it. I would prefer that we all discuss that before making any decisions. Now, I guess a pro would be Yeah, a pro would be that uh just like simplified like we don't use different constants and we just purely rely on the blob schedule. And a con would be that we uh we're just modifying old specifications and old code which is a bit messy and that's all. Thank you.

Uh, anyone else have comments on this or on the BO schedule configuration consensus at all? Um, curious if there's a a radic. Yes, maybe just one comment. These uh three BPOS's which have uh unset epochs they like do you see this go into the spec because I expect that if people see that there might be

some um expectations that this is what we are actually going with. Uh sort of just a placeholder at the moment. Uh the the blob limits come from Alex Stokes's uh blog post. Uh we could reduce these or like we can agree on whatever. We also don't have to necessarily make it uh we don't need to decide on it right now. But I can remove those if that's a concern. Great. So looks like from the thumbs up maybe people are in agreement uh they like the idea of removing those

for now. Okay Paul, is that what you thumbs up for? Yeah, pretty much. And also uh like what uh Alex had mentioned earlier about having a fixed blob schedule. Uh this PR doesn't do that, right? It just uh sets up the uh like basically YAML config for setting the blob schedule, but it doesn't commit to any specific thing right now as far as Yeah, that's right. Okay. Yeah, we can we can remove this.

I think the idea would be that uh we should be able to configure as many blob schedules as we want in like a definite configuration where we would just be like epoch one is uh 18 epoch 2 would be like 12 go up and down and up and down in field if we can go back and forth in different uh explos if you're finding yourself in a situation of having to go to devet seven because of issues with DevNet 6. Do you include uh this in DevNet 7?

I would like it. Yeah. But I'm not sure how long would it take for clients to implement this BO. I think it's best if we keep the scheduleuling discussion separate. Um let's first figure out the PR and there'll be spec releases and so on before we talk about devets. So, um, anything regarding the BPO config schedule from the EL side? And Raul asked in the chat, are we expecting clients to adopt this schema config format as is or is this just for illustration

purposes? Justin, any thoughts on how tightly it should be followed? Yeah, just um just a little bit of context cuz I'm thinking that if this is a config format that we expect clients to simply reference download slashverify somehow or just offer some utility so that operators can just update and refresh their configuration based on like some canonical uh agreement. Maybe this is a specification that should be versioned.

Um, and there should be a canonical schedule published for each network that we spin up over time such that clients can simply download the thing and maybe there's a signature. I don't know. It's just like I'm just thinking through the process by which the operational process by which these values would be updated when there is a decision taken to upgrade a network to some value at some point in time. Yeah. So, uh we did sort of talk about this last week. Uh to answer your question though, I I think I would expect clients to adopt the specific schema and configuration structure. Uh when there's an update to

the pod schedule, there'll most likely just be a client or there'll be a bundle of client releases and everyone will just update accordingly. There shouldn't we shouldn't need to do it like super quickly like it people should have time to update. Also, just one more additional point. Usually updating this file is associated or when we expect the clients to adopt them there's also a consensus specs release. So there is sort of versioning built in in terms of which version of the spec you reference as a client.

Got it. Very useful. Thanks. Great. And any thoughts on this topic from the E side of things? Okay. Uh Pop, I have the PR open that you noted in the uh comments on the on the issue. Is there anything that you wanted to say about this?

Uh yeah. Uh yeah, like uh we found some small bug in the in the spec about the custody sampling. So uh yeah it just a small buff. So it like I think it like should be like a one line change in the in in in the clients. Yeah. Like like Yeah. Like like scroll down a bit. Yeah. Yeah. This this like Yeah. Like like like the the the function get custo should should uh should receive the the the number of custo uh in the second argument. But but but uh you can see the variable called sampling side uh

currently is is is the number of columns is not the number of custo. So so yeah so this this PI just change it to be like the number of custody groups so that it can be put in the get custo function and yeah that's it. Great. Um, anyone have any questions on this item? Cool. Thanks for flagging this up. Okay. Um, yeah, there's still uh really no change to the open uh PRs from last week. Uh I believe that one the

blob count move from draft to open. Um which I don't know if there's anything related to um yeah Francis is there anything that you wanted to flag on these or ask for review um or just speak in general about the status of this group? Yeah. Uh thanks. Um so for this PR I've asked uh Felix and like Male to help review it and male have officially given a plus one on it and I believe I'm kind of in contact with him to get it merged today. So hopefully

that that will get merged and there's two things I want to call out on the execution API part. Uh there's a little bit like change to the spec but it's mostly minor. The first one is that uh we added the um kind of like for unsupported fork logic to get blocks v1 and uh get get payload v4 because like after flu we are using the new versions. So it's kind of like before we just added that in the kind of like uh in the change and the other one is that uh I think male gave a very good uh failure scenario of the get blocks v2 which if the if the client somehow gave us a blob h version hash of the previous version

uh then we in the spec we specify that we return now as well. basically we treat that as like not found. Um and those are the two like main changes to the PR but otherwise uh everything should should be good and we hopefully I will contact them to get it merged today and yeah that's it. Excellent. Uh, Muel, is there anything that you wanted to add? Um, great. And then I I don't know

if there's any of these remaining uh that are open. Uh for those two issues, for those two issues, the PR we discussed earlier, we'll close them. Okay, great. That's the first two on this list. Yes, that's right. Okay, perfect. And then I think Alex Stokes had this open. A little polishing for the actual EIP

and there was there were no open discussion items um submitted. I don't know if anyone has anything else that they would like to discuss. Uh see a little bit of conversation uh in the chat regarding the VPO schedule. I don't know Ari or is there anything that you wanted to voice over here for others? I have a quick question unrelated to that per se like have the have the ELS decided on a block schedule configuration structure?

Uh I don't know FLCL Roman Gajuner can anyone provide a temperature check on the E side. Uh Marson. Yeah. May maybe we can just uh choose uh uh the best two ones and uh select between them in the chat. Uh finally and because uh uh all the cases look uh quite similar and

uh look look nice and uh there is no huge difference and we just need to choose and uh go with something. Okay. Yeah, we can discuss the sync. Yeah. And yeah, Barnabas, I think the CLS should be decided ASAP. I think we've decided on it though, right? Pretty much sounds like the the format and the structure is decided on. It's just a matter of noting the max blob count. Is that the open item? Uh yeah, the open item is

we'll remove the BO forks from the current bomb schedule and that PRI. Gotcha. Um regarding the API, is that all clear everyone like the future uh max per block should be returned in a JSON format? Um I'm confused. What? Uh so there should be some API where we could like query the specific uh max blobs per block, right?

Oh yeah. Only for the current one. Isn't that just the config API that returns the entire config? That returns the entire config. Yeah, sure. I think there should be an extra one for like returning like the current mix blobs maybe. But why? It's probably going to be used for some somewhere someone I think you're saying that you could just get the blob schedule from the the other the other one.

Yeah, I think the argument is essentially that in the past everything's been tied to a fork. Um, so if you knew you were in DAB, you knew what the blob max was, but in the future you could be in Fulu but have a different blob max. I guess that's what Barnabas is getting at. Yeah. Like I wouldn't necessarily know if like where am I at right now? But I need to query which maybe we can just copy the CL on this.

Um they have a gas limit. It changes it's valid data was voting. So I guess the API takes care of that somehow. And we can do the same approach for the CL. Sure. Uh that would be in the beacon APIs uh repo there, right? Not here. And the specs. Yeah. Yeah. Okay. Yeah, we'll look into that.

Cool. Good call out. Um, are there any other uh open discussion items? So, it sounds like uh diagnosing pure DevNet 6 is kind of the the most pressing item right now. And uh Francis trying to get a couple of those lingering little uh issues related to your spec sorted out. Am I missing anything else or glossing over anything else?

uh in the chat follow um Justin noted that there might be any update on the purv builder topics. I I think I uh updated rustic builder to uh have fulu support but uh in the curtosis uh mock builder setup it also requires me boost. I don't know if boost was updated or not Justin. Uh yeah we we made an update for that like I don't know if it actually works but yeah it might work. Okay I haven't checked then I can check it today. Okay. map was actually rebased on top of the latest

develop because there has been some works in the latest develop where the SSV bug would be fixed. Uh it has not been I will do that today. All right. Thanks. And the same for the relay. I think uh the relay should be supporting SSD as well. I'm not sure if this is a stick builder. I don't think there's a relay involved. Oh yeah. Okay. But the builder does support SSC, I believe. Ready?

So, you have your hand up. Yeah. So, um just wanted to mention one more thing. It's probably better suited for the next call. But if you go to the um consensus consensus dev channel on discord, Terren from our team posted a question that because each data column has a block root in it, then that block route is basically duplicated in all data columns. And Justin commented that um having some kind of like a different container that would have a list of columns and a block route only once would reduce the maximum request from 640 kilobytes to 132. So it's more than

uh reduction of more than 75%. Although that would um in some way make semantics more different because if you have a list of columns then you would need to somehow be able to say some columns are not um present. So maybe you know that's there's a trade-off here. Just wanted to raise awareness that there's um such a um comment on discord and anyone interested can go there and uh add your perspective.

Maybe I can just uh forward the message to the pier testing channel for clarity. Thank you. Yeah, thanks Justin. Okay. Um last call. I just have a quick thing to uh highlight which is there's a gossip sub API change like a change for implementations that can improve pure das uh publishing performance and I have a branch out in go P2P pub for this but uh it would be good two things one to

have other implementations start adding this and then two once we get this in to have a client teams use it. It's the links in the chat. Sorry, just seeing the chat or I just saw the link. Thank you. This this extends the gossip sub badge publishing um ETH research posts.

Great. Uh, Saba, I see you on the call. Is there anything that you wanted to uh mention with us? Only asking looks like it's related to research post you authored. I would just thanks to Marco for linking up with this linking up with this and I think some teams already implementing it so it would be good to to see it implemented but nothing else at the moment.

Cool. It sounds like White House is implemented. Uh Katcha, I saw you had your hand up. I don't know if you had your question answered. Yeah. Hi everyone. Uh I just want so from my updates I added a couple of metrics uh regarding to custody group and I would like to know from the clients mostly from Prisman and Lighthouse is the logic fully on the beacon node side. So no need for additional validator clients um or custody group metrics. Um probably we can also discuss it async but yeah if if you know yeah I think on prism um metrics

are not yet fully fully there. We have a peer does um uh issue that has all um sub issues and uh I'm pretty sure that Manu mentioned to me that metrics are still in progress. Yeah, I'm working on other metrics. Yeah, my question was about custody groups. uh is it is one metric on the beacon node side enough or you need separate validator metrics so both for pick node and validators regarding custody group oh sorry uh I don't have an answer to

this unfortunately okay we will we will just discuss it I think I will open the thread for for this thank you great thanks kaja I'll be sure to surface this on the call next week uh in case it doesn't get resolved in the meantime. Um anything else? Okay. Uh good call. Thank you all.

Thank you. Bye. Bye.