All Core Devs - Testing #93 (CL Breakout)

Transcript

Justin Traglia: Hello, just one minute, and we'll get started. Alright, let's get into it. The first item on our agenda is, just… we're gonna discuss the builder specs and, builder… sorry, not builder… builder specs and the Beacon API. So, the R's. Jason, are you here to give us an update on, builder-specs 165? Yeah, I see that they're merged. That's great.

Jason Vranek: Yeah, like Nico said, I haven't had a chance to review the changes he pushed this morning, but it looked like some… just nitpicks, but it's been merged now.

Justin Traglia: Awesome. Remind me, is this the… like, the… the big builder specs PR? Or, like, BNs and stuff?

Jason Vranek: Yeah, this is… Yeah, completing all three. So, last week the Key Manager API was merged, and this week it's Beacon API and builder-specs.

Justin Traglia: Got it. Okay, out of curiosity, how are clients doing regarding, direct bids? Have you guys tested this on DevNet 8 yet? Are you able to test it?

Nico Flaig: So for Loadstar, I didn't test it on a DevNet yet, but on Kurtosis with Builder. Yeah, not sure anyone tested it on the definite. Maybe Brism, I don't know.

james: No, definitely not us. We're still missing a couple PRs on it.

Justin Traglia: Crap.

james: It's in the works.

Justin Traglia: Eitan says that Jimmy's been testing it in Lighthouse, and they've uncovered a few small bugs on our end. Eitan, are those bugs in Lighthouse, or are any of them bugs in the spec? Do you know?

Eitan Seri-Levi: They're Bucks and Lighthouse. Not, not speculated.

Justin Traglia: That's good. Nice. Yeah, PK Shader linked to the public builders. Okay, Jason, would you like to give on… I mean, are there any updates you'd like to give on the next PR, the Beacon APIs one, or just the same? Like, it's already been merged.

Jason Vranek: Yeah, no big updates. The latest changes, since the last batch of people approved last week was just some… renamings across both. Like, signed request auths make sense in the builder-spec repo in isolation, but across clients generally, it was confusing, so it became… builder-signed Requista, or assigned builder request auth, so just a little more, clarity on the naming.

Justin Traglia: Got it. Alright, thank you. And, lastly, Luca, are you here to give us an update on BeaconIPIs 627, and what needs to be done to get it merged?

Luca | Vero: Yeah, sure. 6.30 was merged just now, so I'm gonna update 627, just to get rid of the conflicts, and… I'm gonna do that today, and then, well, basically, the only thing that needs to be decided is whether we want to have it as a separate endpoint, as it's proposed now. Or if we wanna… make it part of the normal produce platform. My personal opinion is the separate endpoint is much… more, elegant, simple, Teresa?

Justin Traglia: Alright, how do others feel about this? Do clients have opinions on whether it should be separate or… The existing one, Nico?

Nico Flaig: Yeah, there's just… so we made the builder-config required in Produce Block before, so maybe that's an argument for having it separate also. But besides that one thing, so how does that work if you weight the bit that Vero selects, for example, against the local payload? Right, you would pass a boost factor to the Beacon node, I think, right?

Luca | Vero: I'm not sure if there's a boost factor right now, but… If there would be, you would just, yeah, boost the local value, and if it's higher, then you could override it.

Nico Flaig: Yeah, so…

Luca | Vero: So it's.

Nico Flaig: the…

Luca | Vero: Makes sense for this endpoint, actually.

Nico Flaig: But you select a bit, right, and then you pass it… to the Beacon, how would you ever pick a local payload in that case?

Luca | Vero: You wouldn't. The only time you would expect… a local payload? is if the circuit breaker is active, or something like that. In that case, the Beaconaut can use the local one. But if you are bringing your own bid, then you expect to include that bit in a blog in most cases.

Nico Flaig: Yeah, I'm thinking functionality-wise, you should probably pass the boost factor, or the… maybe it's enough if the Beacon node applies some default, yeah. I guess I'm not sure, is anyone gonna implement that, besides, like, Vero or Vouch? Maybe Nimbus? I'm not sure. I think there are not many valid clients that behave like this, where they query all connected Beacon nodes.

Dustin Brody: Interesting question, we haven't really… approach this, but obviously Nimbus does, in general, have that behavior, and might exploit that.

Luca | Vero: The question is more about if… The Validator client wants to, fetch bits directly instead of letting the Beacon do it.

Dustin Brody: Yep. So far, Nimbus has not tended to have that approach. It's a pretty… it's much more, kind of, lightweight. Thing.

Chris Berry: Just on this, so how do we intend to switch between trusted and trustless bids from the multi-client side? Because if we're getting bids from the relays, and we propose a bid, dire, however it works, whichever endpoint we use. How do we know that the trustless bids are not better than the MEV one that we've received via the relays? And who makes that decision?

Luca | Vero: Well, as a Validator client, you can subscribe to the trustless bids over SSE, so that's one way To approach this.

Chris Berry: Right, okay, so you're basically in a world where you're gonna ignore… the local block, whatever it's worth, and it's up to the Validator client to differentiate between anything that comes over the gossip Trust Plus path, or the Web2 MEV Relay path. And essentially offer up one to the… To the Beacon node. That makes sense.

Nico Flaig: Yeah, but I think that's another detail, maybe, worth to define on the spec in your PR Luca, because it's not clear whether the beacon node should bring in the P2P bit, or… Like, we expect the Validator client to consider them by using the SSE event instead.

Luca | Vero: Yeah, that makes sense, I'll edit, I think. In this case, the Validator client should take those trusted bits into account as well.

Chris Berry: Because it's… it really is… it really is dependent on who is deciding which to choose. If we take the path that the… the Validator client knows what it's doing, and whatever it sends up, that's what you have to choose. Ignore any local, ignore anything else, that's what's going to be used, then yeah, fine, you don't need to add any kind of… Boost factor or something similar to help guide the Beacon node to make the decision you want. And that works, I think that's a reasonable path to take, it's just that as Nico said, we should be explicit that that's the intended behavior, that you're basically gonna throw away any decision-making from the Beacon node side, and it's basically gonna take what the Validator client offers up.

And ignore the rest. I think that… it makes sense in that use case. It's just a shame that you can't really consider the local block, but I think in today's world, you can't really consider the local block either. well, it's something that happens on the Beacon node side, right? It looks at what comes on the external builder endpoint, and obviously could build a local block, but I don't think any decision making is made today. I think if one's on the builder endpoint, it is what's selected.

Luca | Vero: Yeah, unless we would want the local payloads also to be broadcast over SSE or something like that.

Chris Berry: I don't know who would do that, because presumably right now, that's only done if a valid proposer asks the Beacon node to build them a blog.

Luca | Vero: Yeah. Well, maybe one more thing is, since… The bit that the Validator client brings could be invalid on one of the Beacon nodes it's connected to, because it's… On a different head, or something like that. So, it may be good to clarify what the Beaconode should do in that case.

Chris Berry: I mean, that's a good point, right? Because, like, what we just talked about, offering up a bid, and that's what the Beacon node takes regardless, you basically say you're not allowed to do a check, right? But I think it does make sense to say, hey, if this is an invalid block in our view, in the Beacon node's view, throw it away and build a local one. But, oh, hang on a minute, you can't do that because you've already signed the blog. At this stage.

Luca | Vero: Yeah, no, I think the BR actually says something like, if the Beacon considers the bid invalid, it should disregard it.

Chris Berry: Right, but then it can't replace it with a local block, right? It's just, you've missed the slot.

Luca | Vero: Well, that's gone.

Chris Berry: Because you've already signed the block at that stage. Have you not?

Luca | Vero: I don't think so. This is during the Block production, before the signing.

Chris Berry: Right, so we take the bid, we offer it to the Beacon node, then the Beacon node returns a block with that. A proposal to sign based on that.

Luca | Vero: Yeah, but before the Beaconaut returns the block, it can validate the bit, and if it's not valid in its… viewed, and it can return a block with a different bin.

Chris Berry: Okay. Well, as long as that's the behavior of the PR, then great, that works well, so it could actually ignore the… whatever the validator selects and send back a local one, and that's because it's failed validation. That makes sense, if that's how it works.

Luca | Vero: Yeah, that's how I imagine it would work. It has to have some leeway, too. Replace it if necessary.

Nico Flaig: I think one more concern could be also the timing of the API call, because I'm guessing you will not call this at the start of the slot right away, right?

Luca | Vero: Right, yeah.

Nico Flaig: And, at least how CL clients behave right now, they will only start building the CL block site, or most clients do this. They start this only when the API is called to produce a block. So if that is laid on the slot, that could be potentially problematic.

Luca | Vero: Good point. There's no easy way to solve that, right?

Chris Berry: Well, I think you just have to control it on the Validator side, right? And just make sure you have a deadline of something sensible, and then you definitely call the API within that time to make sure that it's… got enough. I don't know if we have an idea of how long that is. I think… I think on production today, we generally make that… our deadline is about one and a half seconds into the slot, I think. I don't know if that would need to change for this.

Luca | Vero: Yeah. Do we have any idea if the time that it takes to produce a block is dominated by the EL payload, or is it the attestation packing, or… Because obviously, yeah.

Nico Flaig: Yeah, I think the packing is probably the bottleneck here. at least there's some worst cases, at least in Lodestar, where it can take, like, 800 MS. I think Teku mentioned something earlier about this, too.

Luca | Vero: Okay, but I think this is not such a huge deal, because it's… It's… kind of an advanced Beacon API endpoint, so the callers will just need to make sure that they're calling it on time.

Nico Flaig: Well, what was the issue of you just calling this on every Beacon node, and you just, sell, like, just get the local payload that the Beacon node produces? And then, only Vero calls the builders.

Luca | Vero: Oh, that you replace the bid directly in that block that is returned?

Nico Flaig: Yeah, and then you… Let me see…

Luca | Vero: It affected the block's state rule, so that didn't work.

Nico Flaig: Alright, and also it would not work with the circuit breaker.

Chris Berry: Is the circuit breaker designed to be within the Beacon node, such that if we call it with, whatever, a MEV bid, but the circuit breaker's on, then we always get back the local block instead, regardless of how valid the bid is?

Nico Flaig: Yeah, I mean, that should be on the Beacon node side, I think. There's also, another factor, which is, we have this EL flag today, which says, like, should override builder, which is, if the EL detects censorship, it will also use the local payload. But yeah, I think it's hard for you guys to implement a circuit breaker, because you need the full review of fork choice and stuff, so…

Chris Berry: Yeah, I think, as it is today, we should retain this logic where it's the Beacon node giving the validator client which block to use, and then circuit breakers, validity, everything else can remain in the beacon node. And it's always… then it's a simple path for the Validator client once we get something back from the Beacon nodes themselves, that it's basically whatever we get, we sign it, we submit it, and that's… that's kind of done. So it would make sense in my head to preserve that, that everything… essentially, the last steps for the Validator client are, whatever you get back, sign it and send it.

Luca | Vero: Yep, agreed on my side as well. So, yeah, like I said, I'll update the PR today, and I think we can move on to the next topic.

Justin Traglia: Okay. Thank you, Luca. That was very interesting to listen to. There are no other topics, officially. I'd be curious to hear. updates on circuit breakers within clients, maybe? Like… Have we tested this on the DevNet, or are the implementations Merged into those branches.

Parithosh Jayanthi: We had a bunch of discussions about the Circuit Breaker, on Discord last week, when we had non-finality on the DevNet. But we haven't gotten, like, a clean test yet.

Justin Traglia: Okay. Are you aware if, all of the clients have a circuit breaker, which could be used on the DevNet. Like, have they been merged?

Parithosh Jayanthi: I think Nimbus and Grandine might be attending? But the other should have something.

Justin Traglia: Okay. Cool, that'll be fun to test here soon. Oh, Nimbus does have a circuit breaker, they said.

Parithosh Jayanthi: I got it, okay. Then, oh, oh. Yeah, Be Nico says not the builder.

Justin Traglia: Oh, okay. I guess that's it for that. I'm also curious if clients are providing Like, a way to operate a builder. That's not build OR by PK. I believe Nico's been working on the Summon Lodestar, and I'm curious to hear if other client… if clients are doing something Similar. says that Lighthouse has one too, as far as he's aware.

Eitan Seri-Levi: Are you talking about, like, a trustless builder?

Justin Traglia: Yes, exactly.

Eitan Seri-Levi: Oh, yeah, we don't have that. We do have, like, a closed source. like, trusted builder-client thing that we've been testing on the DevNets, but the trustless client, we haven't implemented yet.

Justin Traglia: Okay. Do you know if there's an intention to provide that, or out of scope?

Eitan Seri-Levi: Yeah, it's on our list, we just, haven't started it yet.

Justin Traglia: Okay. No worries. Is anyone else gonna work on this, too? I'm just curious.

Nico Flaig: As a side note to that, I noticed something, at least in Lodestar, how our bit publishing validation works. So, as it is right now, you cannot even publish any bits that are not HAT compatible, because we apply the same gossip rules as for normal gossip validation on the API. that's maybe something we need to fix, because I noticed if there are reorgs, there are sometimes no bits. Maybe that's also builder not doing it correctly, but I think Lodestar would also ignore these and not publish them. So I guess I'm not sure what clients should be doing, so my idea is just skip the… only basically run reject rules, and all the ignore rules, you can basically not do them, and then you can also flood publish the bits.

So that, you basically cover more branches, even those that are not HET compatible.

Justin Traglia: Okay, yeah, we should test this more. And determine what the actual… Ugh, problem or change needs to be.

Nico Flaig: So does Builder already do multi-branch bidding now, or… Because that's kind of something we need to test this.

pk910: Yes, it's true. It builds payloads for the immediate parent and the grandparent, basically. So it tries both. And also variances with, An empty payload and a full payload, basically.

Justin Traglia: Have you seen this working on .NET 8?

pk910: I have seen bits for these, but it's hard to… have them included afterwards. I mean, we need such a situation to have that bit included. But there are bits for all four situations I've seen.

Justin Traglia: Okay, cool. That's all from my side. Is there any other topics we'd like to discuss before ending? All right. Well, thank you everyone for joining ACD Consensus Breakout. Talk to you guys next week. Bye, everyone.

Eitan Seri-Levi: I see.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.