All Core Devs - Testing #74 (ePBS Breakout)

Transcript

Do we need to do something special or can we just quickly get rid of this of what we have to discuss today? The PTC, you mean? Yeah. Uh as far as I know, there's only one two items in the agenda. The first one is we need to decide what to do with the PTC. Uh I haven't heard any any feedback. It's not really clear. Uh there are two open PRs. It's not clear if we're willing to do a third one, if we're going to choose one of those two. I want to hear what client devs have to say about those.

No one has an opinion. I can probably put people on the spot. I mean, for Lodestar, we are fine with either of the PRs right now. I think um NC is preferring the What was it? Uh 4992, I think. Uh I have implemented that in Lodestar, and this it's fine for us.

Um yeah, but just to know that we will still need to serve the whole epoch worth of PTC duties or assignments. So, um lazily computing per slot will likely not be possible. Is that something people are considering? Unless we change something in the um in the beacon API spec, but I think for duties, this I think we need to serve the whole epoch, maybe someone has thoughts there, but uh that would be the first time if we only can serve a duty at the start of the slot.

we are I mean I think this this two PRs are completely independent of the duty thing. Uh both of them allow you to cash and serve duties at the beginning of the epoch for the whole epoch. Um I think mostly the question here is whether or not uh we want to cash in the state or not the full epoch to serve those duties. So I I I guess I don't know if uh Teku hasn't seen this

perhaps. >> Nico, you've seen it? No, no, not yet, but I'm I'm I wonder what is the benefit of having those information in the state rather than caching us out outside of it. Yeah, so let me just quickly go over what the issue is that Nico found. Uh the PTC is now dependent on the effective balance. And the last slot of an epoch, its attestations are being sent.

And the PTC committee that is being computed for that slot might not be valid, might not be the same in the next slot when we just call get PTC with the current helper. Because the state has changed the effective balances. So we need to somehow cash in the state, at the very least that a So last slot the last slot from the previous epoch. There are two options. One is to cache the entire previous epoch and the entire current epoch on the state, and the other option is to cache the just previous slot and current slot in the state. There are other options we

could explore, like just for example, caching only the last slot of an epoch, and there's trade off trade off there that I don't want to even get there. Uh but between these two particular PRs, either caching the full epoch uh for the previous one and the current one versus just the full slot for the previous slot versus on the current slot, the trade off is either one trades size in the state. Uh it's 8 kilobytes on one versus uh 256 kilobytes uh on the state that would be added on every single state.

But, on the other hand, the advantage of caching the full epoch is that anyways we're going to have that cache in memory, and the problem is bugs of the form keeping that cache in memory that we have in sync with the state that we use to validate. So, if we actually have it in the state, then we're sure not to have those problems of synchronizing caches in memory with the state that we're using. So, the question is whether or not that synchronization safety uh is worth having 250 CX 256 extra kilobytes in the state.

Let me see. Uh as Nico said, this issue of like hashing Hashing I I doubt very much that hashing is an issue because this cash will only change once an epoch. So, I actually suspect that hashing is worse on the first slot situation than the per epoch situation. I mean, we're sure to rehash when we change every slot and we're sure only to rehash on epoch transition when we change the full

epoch. My feeling is that people haven't seen this in detail, so we're not really ready to make a decision here and it's an easy change. So, we are in no rush. I guess I'll pass perhaps before Barnabas. We should decide what to do with Devnet 1, when tech was going to be ready, when Nimbus is going to be ready.

If we keep Devnet 0 going for longer, I would propose not to make a decision on this issue until we actually need it. Yeah, so what we can actually just do is there's already a bunch of other changes in offer three. So, I was I would actually just propose that the PTC change could wait for Devnet 2 and we just go ahead with the with the smaller changes in Devnet 1 and we still try to launch maybe next week with that. I'll I'll disagree with this. I don't understand why I mean, unless there's a hard fork, I don't see why we should be

changing the Devnet. So, the PTC thing is a hard fork, that's why we actually need it. And I would expect this change to actually be on Devnet 1. But I don't see any reasons to ship a Devnet 1 without a hard fork. I'm also happy to just include uh all the changes in Devnet 0. Uh it's just a lot more messy. And uh Why so? It isn't isn't all of them not hard forks?

The spec doesn't change. What's the problem? The social like this this spec changes in terms of which kind of spec version we are targeting. Like we said that we're going to be targeting Alpha 1 for Devnet 0 and then Alpha 3 for Devnet 2 or whatever. Uh if we're going to be saying that we're going to be targeting, you know, Alpha 1 uh or Alpha 3 now for Devnet 0 it can be done, but it just a lot more messy in terms of uh people knowing what version that they should be targeting.

That's it. Like not everybody is so hands-on with EP yet that uh they keep track of all of this uh unless there's a new actual Devnet happening. It's it's really for bookkeeping reasons uh that I would push for a push for a new Devnet. But if everybody is on board on just rolling out the new Alpha version in the current Devnet, then that also works for me. I I just want to make sure that, you know, uh I'm I'm not sure if for example we have uh a claim there from every single team here uh to make this decision, uh but if we do, then uh then we could just make the decision about that here.

Do we really have a timeline for when Teku and Nimbus are going to be able to join? yeah, I'm still uh we have issues with uh networking. Um so, we are we we're looking at those. and I have some version of checkpoints in which I will merge a bit later.

so, yeah, I'm hoping in the next couple of days we have something. Yeah, the actual problem is that the issue that we have uh is not fully understood, so well, yeah. We don't know. To be fair. I don't know if if you agree, Stefan, but yeah, hopefully it will be mm in the coming days when we found the problem and we'll be stable there.

Is there anyone from Nimbus here? Didn't say. Yes, this is I'm from Nimbus. So, yeah, it's is it and my audio work? Yes, this seems to be. Yeah, we can hear you. Yeah, so my current our current state it is um we we have test with cuddles is locally that's the the sinking which mean calling by range or to to request envelope is working.

But on testnet it seems to be having a lot of Easter. Not sure is it prism or because right now I'm still not quite sure which clients provide envelope by range. But it seems like if for some reason if Nimbus successfully requests envelope from prism, I guess. after a period of time, prism seems to be will have um disconnected from all the peers or or what whatever reason. I'm not sure quite sure. But yeah, after that we kind of not able to sync.

then Wait, wait, I I didn't understand this. The fact that prism gets isolated, why how should that affect your sinking? Because we have not done the sinking yet. We still have been a few and those >> if that were the case, so you're not connecting at all to other clients that are serving? Yeah, because it seems like by ranges seems like it is the most stable if we request it from prism but not other client clients. We found that for one client, I'm not sure which one, they return um zero envelope from by range.

But we got all the blocks from there. So, it's also making us not quite able to sync. So, yeah, that's the case one but we start looking into the checkpoint thing which we still having some issue because we didn't implement that. That's for the devnet zero and I we still having some issue on the on getting data to fill in the database. So, yeah, that's the another problem we are trying to fix.

We tried. We mean >> we can we can discuss. I don't I don't think prism should be the one that is giving you zero payloads because I'm experiencing that myself. >> that I believe not that is not Prism, but I'm not sure. >> case that if Prism is down scoring you somehow and and not serving you anything, it shouldn't be the case that you can't sync. You should still be able to receive even from Lighthouse payloads. Yeah, I I will have to check, but it's not quite stable right now.

But at least from local I mean if I test locally with Cuttlefish it's working quite stably, but there might be some edge cases that I didn't catch in in Devnet, so I will still have to check. Right, and and so Nico is confirming that Lodestar should be serving envelopes by range. Uh is that now live on the Devnet, Nico, or is it uh something that you have in your branch and haven't uh shipped?

Uh no, that should be live since last week last week, I think, but we also checkpoint synced a bunch of nodes, so they are not actually serving that. Uh we only synced one node from Genesis. Yeah, this is the problem that that's why I'm trying not to uh I'm trying not to uh restart Prism's nodes because of the same reason. I don't want to lose history yet. Uh Aethon is saying Can you Can you just speak up, Aethon, on your Prism? Oh yeah, no, uh Lighthouse is serving envelopes by range. Um they're just not guaranteed to be canonical. Um we have a we have a small bug in our implementation where we're not checking if the payload's canonical already

served it. Uh yeah, I'll I'll get back to you in a second about this because I I'll have a complaint. Uh but um can can you say something about like this issue of like moving to Devnet one or that or staying on devnet zero? Oh, yeah. I think for us devnet staying on devnet zero makes sense. I mean, we already brought up that it's not a hard fork. The spec changes for alpha three. And we're working right now on trying to stabilize our fork choice implementation and introduce checkpoint sync. And if we try to launch devnet one like next week, um that there's just going to be churn involved that's just going to slow us down from doing I think what's more important for us right now. So, I would

rather stay on devnet zero if possible. Sounds good. How about Lodestar? Um yeah, for us it's also fine to keep the current devnet. So, there's a lot of code review we still need to do which probably going to take most of our time. It's It seems to me that that's the most reasonable thing also for Prism. We have a lot of bugs. We can't restart now on an existing DB. We can't checkpoint sync or we can sync from genesis, but we're having trouble restarting. And I think it's going to take us a few days even to look at the current known bugs.

So, I'd suggest we keep devnet zero. That also gives time to Teku and Nimbus to catch up because they know what's running now on devnet zero. And let's not worry about devnet one until we actually fixed everything we can do on devnet zero. Um on the peering side, uh and this is my personal grudge, but but but this is uh it's kind of making our life much harder to debug. Uh I don't know if I should remove the down scoring or I'll just put it again, but but the reason is the thing is the following.

I am requesting blocks and payloads from peers. and I check the consistency of both the blocks and the payloads. So, if the blocks don't follow the parent root hash check, I down score the peer. If the payloads don't follow the parent chain, like each payload parent hash is the hash of the next payload. Uh I down score the peer that gave us that. I suspect that no I haven't seen a problem here. But, if the same peer gives us blocks and gives us payloads, I down score the peer that doesn't give me

this in a consistent manner. So, there are two cases in which I'm down scoring everyone. One is peers that serve me a sunset. They serve me pay blocks and they don't serve me payloads. They say that they have zero payloads for the for that Then, I immediately disconnect those peers because that's malicious behavior. And something similar is happening, I suspect now, with Lighthouse, in which the the blocks do not agree with the payload chain.

so, I'm not sure what to do if if this is going to take a long time to fix, it might be better if I just stop down scoring for for now and then reinsert it when clients are ready. But, I'm just giving a heads-up that we are as soon as Prism goes out of uh out of sync for a little bit, then we're isolated because we end up down scoring because everyone is sending um sending inconsistent payloads. It might be even the Prism is serving inconsistent payloads because we run out of peers, so it seems that we're not even peering with Prism.

Yeah, so Light Lighthouse has a fix for this. The issue with us is we just don't have checkpoint sync yet. Um but, we we have the fix, like we know what it is. So, as soon as we have checkpoint sync, we'll push it up, but I won't downscore you if you don't downscore me. I see. Yeah, but it's most probably what's happening is that we're down scoring lighthouse and this this is what's giving us isolated. So, can I ask if people are backfilling? Um can can you guys serve payloads and blocks if you backfill them?

Okay, so Nico's saying that not low, so Uh I suspect we also aren't doing this. I'm not aware. Yeah, so Terrence is confirming that person is also not doing this. Yeah. >> Yeah, I I You're also not? Yeah, so Barnabas, I think this the only safe way of keeping testing here is we fix everything we have. Um we try to be in a situation which we can't restart nodes and serve historical data, otherwise this is useless. All right, that's a wrap. I'll I'll ask I think then on the PTC stuff since there's no rush, but please give it some

time and choose one of the two. It's a matter of like size on the state versus um versus having the full cache or not on the state. Um I don't think there's absolutely anything else to discuss in this meeting. Do do you guys have anything to to add? All right, then. I think we can wrap it up.

All right. See you guys. Yeah, thank you. All right. Good night. See you. All right. Good luck to you.