ePBS Breakout #16

Transcript

more or less fine to start now. Uh it might not be a long meeting. Uh perhaps just reporting on where we are and um what are the challenges that we're facing and perhaps some updates from clients. Uh we are now uh syncing uh two client devet on Corttosis. This is Prism on Techo. Uh we we see that we can actually sync on top of each other's block. We can fork into EPBS without problems. But then soon afterwards we start forking uh which are start reorging our own blocks

and it seems there's some some problems at gossiping of execution payloads. Uh we have implemented RPC requests and and a bunch of other things and have been improving quite fast. So I mean all of you know that once we start iterating with a different client uh the pay starts picking up. Actually in fact I have now a devet running which I should have stopped. Let me just stop it. All right. So uh some comments of the things that appeared from diagnosing this with tech. So, just to make it uh officially, we agreed on adding the slot to the payload envelope and removing the payload withheld from the payload

envelope. Uh so that that was a message that became useless once we switched to Franchesco's for choice and we decided just to remove it once we changed the the the structure to add the slot. Uh I will open a CL spec uh PR with these changes. I'm not sure how painful it's going to be to fix the current state the current tests. I suspect it's not so bad. uh the uh the so for advice if any client is here uh trying to to to build on this and and sync there's a couple of things that that are essentially required otherwise it's it's impossible to sync which is uh a client should be

able to sync a block for which uh it's uh it hasn't sync one of the two parents that means if you sync if you receive a block so currently We all have paths for when the parent block route is not known. We verify the signature. We save the block and we call by RPC. We request the parent block. So this becomes harder with the PBS because you might have the parent block but you may not have the parent uh the parent block hash. So now there's two different parents. You have the parent block and the parent block hash and you you need both to sync a block. So it may be that you find yourself in a situation that's

what's happening now because we're not receiving the payloads in which you're you have the parent block route but you do not have the parent block hash and you need to implement an RPC request so to get that payload envelope to process that envelope immediately so that you can actually uh sync your block and the same thing happens with payloads. It's quite common when you're running a corttosis on your own machine that uh you will receive the payload earlier than the block. So that means that you're receiving a payload for which you have the parent block hash now. So this is the opposite with blocks. You have the parent block hash because the parent block hash is from a previous block but you do not have the parent block root

which is for the current slot. So in that case you need to request the block by root sync the block and then you can sync the payload. So these two things uh were things that we hadn't implemented when we had that finalizing devet uh predef on on prism and we were forced to implement a sync uh tech blocks uh what else uh we added the slot we I already covered this uh oh I see so uh another thing that is currently not covered in the spec is what happens when you're syncing a block from when you're syncing a block that reorgs the fork block. So that happens

quite often and um well that used to happen quite often. So if you're syncing a block whose parent block is from the previous block from the previous fork, you need to declare that parent block as full and you need to treat it as a full block. That is any block prepps is a full block. It has both the block and the payload. And this was a source of bugs for prisms. uh so I assume that this may be a source of bug for other clients as well. Uh the final one that I wanted to discuss is uh we had some PRs open for so both Enrico and Stefan open PRs for uh to add payload payload envelopes by range requests on the P2P side and uh

PCON APIs for payload envelopes. uh both of them uh request the payload envelope by the block route and I wonder if we shouldn't request them by block hash. The reason being I suspect that block hash might be better uh and even to store in your DB might be better to store them by block hash instead of block root and and it's because it's typically easier to get the block route when you have a payload. you have the block route because you have an envelope and the envelope shows the the the block route than to get a block hash. So oftentimes to get a block hash you need to first get the block to read

the header and then read the hash to see what hash should have corresponded to that block route. So I think to avoid this lookup for the block it might be better to store pay everything that is payload based uh in terms of block hashes instead of block roots. Uh well that's about it. That's um that's the situation we are now. We are uh having some issues in the devet because we are regarding each other after uh after some blocks. We don't understand what's going on. We we start the fork we sync we sync for a while and then after a while uh payloads start disappearing for some reason. Uh, so that's a peer that's a peering issue

surely and we need to just uh check on peering. All right, so that's all my updates. Uh, I'm not sure if Stefan wanted to add something. Uh, I think it pretty much covered everything for this. Uh, apart from that, uh, yeah, I'm going to start adding some more P2P logging to see what potentially is going wrong when we send the payload. Um yeah, other than that, not much else from me.

All right. So, is there any other comment from anyone else? I was hoping that we will have this finalizing today. Uh well, yesterday and I was hoping that the RPC would just solve all of the issues, but it didn't. Uh it it also shows one of the reasons to have these dev nets early was to prove one of my biggest fear was that we couldn't handle the blocks the slot time and we can in the happy case but it's seems that it's tricky when you're syncing a block for which you hadn't seen a payload because you need to request the payload sync it immediately and then sync the block and uh this this I think it's what this definitely is

showing is that is very tight. So I think we're going to do this preemptively like Teo is suggesting Tech is suggesting to request a payload if you haven't seen it in a while. Uh this is something that that is interesting because uh in the current beacon chain you cannot request a block when you haven't seen it by 4 seconds because you don't have a block route to request it and we don't request blocks by slots. But uh in EPBS if you hadn't seen the payload by a certain time you can actually request the payload because you have the root and you even have the hash because it's committed already in the beacon block. So I think it's a good idea. Um we should experiment with this of start sending early RPC request

requests for payloads that we haven't seen. uh I'll implement this and see if this saves the the devnet but also it's because I think it's going to be impossible to sync a block for which we haven't seen the payload earlier and build on top of that. Uh I think the timelines of the the slot the attestation deadline being at 3 seconds forces us to request preemptively the payload and we should think about those things. I don't know. So Enrique is here.

Terren, do you guys have some comments? Um no comments. I guess like um what we need to work on next from the client code perspective it's probably um um payload by range implementation um such that we can test um syncing um from genesis right besides that are there still any like um high value test or low hanging fruit that that so the devet showed that we can actually sync blocks without payloads which is great because uh that that is the

biggest change that you can actually reorder the payload. So that that I think is already covered but I'm pretty sure that for choice is completely broken because I know that prism will not uh will not survive even in the second epoch now. So like we if we fork into ppbs only a prison devet now with all the new changes will have a problems in in for choice and will start reorganing itself in the second epoch of epvs. So I want first to solve the the issues that we have with TEU then for choice at the next epoch then have this finalizing and after we have this finalizing I will start worrying about uh syncing uh I think uh Enrio's PR is fine the one

for for payload by range we need to like also change our state generation to so that I mean the state replayer so that it includes the payloads uh That's that's a bit of a change. U besides that I'm not sure. When do you slide in Franchesco's four choice implementation in this like road map? So Franchesco's for choice implementation if we don't implement uh inclusion lists and we don't uh uh and we are still based on top of Electra instead of PR. It's it's essentially nothing. It's no changes to

port choice besides what we've already done which is dealing with blocks that don't have a payload. Uh so you also need to count you also have to count attestation which we don't have. Yes. But but I think that's I I think for a long time we can continue just ignoring PTC attestations. I see. Uh because uh they they won't change the honest behavior of clients. If you if you see a a payload, just build on top of it. If you don't see the payload, just don't build on top of it. I think this is this is very minor uh to add it or not. I will add that people are requesting this on the on discord. I will add that CL uh spec moving into

Franchesco's port choice, but I stress Franchesco's news. of portraits without inclusion list and without uh and without PRS it's no changes because you don't need to enforce PTC because the PTC is not telling anything about GA and it's not telling anything about IL satisfaction with Fossil and with PRAS then for choice changes because then you need to actually enforce the vote of the PTC because you cannot have validated DA unless Unless you're a super node and you cannot well and and there's some reasons there's some other things about the the inclusion list but without PRA

the PTC is kind of irrelevant for this dem just to update the spec so we can just see how simple it is so that like yeah I guess the discord the feedback was ready for choice complexity blah blah blah yeah if you open the spec maybe that I think it'll be easy to see how simple that is. Yeah, it's at this point it's really a matter of bandwidth. Uh I have been essentially like probably 20 hours a day here and I'm not going to spend them on Python until I have the go code complete.

Yeah, that makes sense. Besides TU and Pris, do we know if any other client teams are working on EPBS? Absolutely. can I can ask that on Discord cuz I mean I'm sure there are ETF like fellas that will be interesting working and like Sunai Nimbus Lowar and um Nimba low star and Lighthouse may need help here anyway. Okay, so T said so Dustin said that there are some people working in uh in Nimbus. I'm not really sure how far uh that's that's going, but apparently there are people working

on Nimbus. All right, so we can close. I was hoping to have this finalizing by now, but it's uh I think it's going to be soon, either today or tomorrow uh as soon as we solve the peering issues. uh besides that uh I think the next course of action is uh having this finalizing actually first second uh as you said implementing syncing and then I'll I'll make the changes to the Python spec after those two things are done and then we'll just move on from that that I'm sure that

that's already going to cover the next two weeks. All right. Uh, is there are no more comments? This is closing on one, closing on two. Closing on three. Thanks, guys. I'll see you in a couple of weeks. Let's go back to the debugging thread. Yeah, good luck.

Bye, guys. Bye. Bye. Right. Bye. [Music] Heat. Heat.