All Core Devs - Testing #99 (CL Breakout)

Transcript

Jihoon Song: Alright, welcome to CL Breakout. The first agenda we have is ProduceBlockV4 with Bid Beacon API. There's a link to the PR. It's Luca on the call.

Luca | Vero: Yep, I'm here.

Jihoon Song: Yep, the floor is yours.

Luca | Vero: Thank you. Does anyone need introducing to this endpoint, or do you want me to? If anybody is unfamiliar with it, let me know. Otherwise, let's just jump into the trade-offs. One of the concerns that has been raised regarding this endpoint is that it could Palm. Be suboptimal for the network under, bad network conditions. Like reorgs or non-finality. So, that's… that's probably the only concern that I've heard. Otherwise. Is there any other concern? Okay, guess not. So…

Nico Flaig: Has anyone looked into that?

Potuz: Oh, yeah, so actually, let's try to go about this the other way around. I didn't even know about this, and I'm just looking at this, and it looks like the VC asking directly to… contacting directly to builders. Oh.

Luca | Vero: that.

Potuz: Which my first impression is that this is absolutely crazy. So instead of like, yeah, I would have liked to know about this before. I know it's my mistake. I'm pretty sure that most people don't know about this.

Luca | Vero: I have been, trying to raise awareness of this. It's been open for 2 months, and I've been mentioning it on these calls and on the Discord. But, okay, so let me just give a quick intro. The idea is indeed that the validator client would contact builders, and then it would pick a bid, and it would supply a bid to the connected Beacon nodes, and those Beacon nodes would then Use that bid in their block that they're producing.

Potuz: Yeah, just like two days, one hour, a few hours before a fork in Sepolia and a few weeks before mainnet fork. This seems like a huge change for me.

Luca | Vero: Yeah, I don't think anybody thinks, well. It doesn't seem that necessary for Sepolia, that's why I haven't pushed for it earlier. I've always imagined it would be useful starting with Hoodi I have been testing this endpoint for a few weeks on devnet-8 as well.

Enrico Del Fante: Can you go back to the reason why we want to do this? Seems like your question, your first question you asked. The answer should be yes, because I don't get why it's required at the moment.

Luca | Vero: I think it's useful when you have a validator client connected to different CL clients, like, even 5 or 6 different ones. That way, you don't have to monitor all of those CL clients metrics, you just need to monitor your validator client. Makes these node operations easier when you have multiple clients involved.

Enrico Del Fante: So it seems like the VC should be in charge of selecting the bid, because this is where the logic of selecting the most valuable thing resides. So, when you have multiple nodes, the VC wants to know about all the bids around and select the one that it likes. And submit that one, along with the block production request? Is that the reason?

Luca | Vero: Yeah, that's basically how it would work. But why can't you have just one?

Potuz: one Beacon node connects. So, I mean, today the Beacon is the one that requires. Why don't you just make one of the nodes contact the builder on all of the other nodes to not do that?

Luca | Vero: Well, because if that one node goes offline, then you're losing access to bids.

Potuz: And isn't it the same if the one VC goes offline? I'm a little bit confused. I mean, you have one endpoint that calls.

Luca | Vero: Yeah, but it's different, because you can have a failover for the VC. or.

Potuz: Why can't you have a failover for the Beacon node then?

Luca | Vero: You can, but then you have to… You do still have multiple Beacon nodes involved. But, like I said, you would need to track How all of those beacon nodes are doing this. And all of those beacon nodes would basically be contacting the same builders. With the same request. So it just seems…

Potuz: I don't know.

Luca | Vero: I'm sorry.

Potuz: You want to avoid many calls to the Beacon node. To the… to the builders. Which, if you just put it… if you put them on only one Beacon node, you achieve this. But then you say, okay, no, we cannot… we may not do this because that caller might fail, which is the case for the VC. If you have a single caller in the VC, that caller might fail, and you say, okay, you can have a failover, but you can do the same with a single beacon node. You would just say, one of the nodes is going to be this node that is a Prysm, and if that one fails, you fail to get bids, but okay, so then let's put another Prysm that is a failover for that Prysm, and it only acts if that other Prysm is offline, and then now you only get called once. I don't understand why

Why this cannot be achieved on the Beacon node and yes on the VC?

Luca | Vero: I'm not sure what you're proposing here in terms of failover Beacon node.

Potuz: just the usual thing, that you have a Beacon node, and if that Beacon node is offline, then another Beacon node takes its place, which I assume You are telling me that you have to do anyways this on the VC side.

Luca | Vero: Yeah, well, multi-node validator clients usually work with all of the Beacon nodes they're connected to, so they don't work in the failover way. They would ask all of the connected Beacon nodes to.

Potuz: But if you don't have a failover for the validator client, then you still have a single point of failure to the calls for the bids. Which you would have, therefore, the same single point of failure if you just put one Beacon node to call for the bids.

Luca | Vero: It's better if you have the failover validator client. Because it's easier to failover in that case. You can… if you have a shared slashing protection database among multiple different validated clients, you can, in theory, have them running at the same time. But I don't want to get into these details, I should say. Assuming there is a reason. This endpoint is beneficial. Do you have concerns, Potuz?

Potuz: Well, yeah. Changing the validator client to do this is a big change. And it makes the validator client more fragile, therefore, which is something that we want to keep lean, we want to keep small. And actually, I want to hear Nico, because Nico implemented this, and he has already some concerns.

Luca | Vero: Yeah, just one thing, it would be optional. So validator clients don't need to implement this.

Enrico Del Fante: Yeah, I mean, is the… is the Beacon node should implement that, and Validator Client will be custom Validator Client consuming this API, in my understanding, correct?

Luca | Vero: Yes, so the Beacon nodes would only need to implement a very small change. very… Small change to the produceBlockV4 endpoint, which we have.

Potuz: The change would be that you provide yourself, you provide yourself a bid.

Luca | Vero: Yeah.

Enrico Del Fante: Yeah, like, fixing the bid, and we probably need to do the validation of the bid online in that case. It's just a fixed way of saying, produce this block with this bid.

Luca | Vero: Yes, and the Beacon node has full. It can refuse to include the bid if it. Isn't valid or anything.

Enrico Del Fante: What are the concerns from Nico? Sorry.

Nico Flaig: So mine are really technical, I guess. So the first thing, like, just from an implementation point of view, is that Vero needs the payload attributes to know what bids to request from the builder. And we currently emit them in the previous slot based on, like, a prediction, right? So it can happen that Vero gets a bid from the builders that are not even compliant with our proposal head, which we call at the start of the slot. That is one thing. The other problem I see, and this is also implementation detail that could be optimized, is Vero will call the ProduceBlockV4 API later on the slot. Because it does it after playing the timing game with the builders, and users will definitely play timing games at some point, I would expect. So, it calls the API late in the slot, and I'm just not sure, what is the implementation, like, the problem here, because right now, on the BeaconNode side, we just assume this is called at the start of the slot.

At least in Lodestar, we are not optimizing, this to be caught late on the slot yet, but it could be done, it's just an implementation thing. And then I guess the third concern is more implementation, which is you… Basically, at, I think, 2 or 3 branches in the ProduceBlock API. I don't think it's a big deal, that one. I mean, I have it implemented, but… Just something to consider. I guess the other idea I had for how could we do this without duplicating too much… I guess Vero could also just fan out the URLs to different, beacon nodes. But I guess my main, like, concern is, or, like, the benefit here is that we dedupe requests to the builders, mostly.

And I guess, we make the monitoring easier, and the fallback is maybe a bit better, as Luca mentioned, but… I'm just not sure how big of a deal that is, to be honest.

Jihoon Song: Okay, you're on that.

Potuz: Wouldn't it be the same if we I mean if I operated this I can just call different builders with different beacons, and just put those bids directly on the P2P stack within my nodes. Right? I mean, those bids will be accepted over P2P.

Nico Flaig: But.

Potuz: strictly about the API. Yeah, the trusted payments, yeah, so.

Nico Flaig: This is really only the API.

Potuz: I I don't I I personally don't see anything wrong with allowing you to provide your own bid. I mean, I'm sure it's going to enable some off-protocol markets that are not going to be desirable, but. There's… it's not something that we can prevent in any way, I think.

Nico Flaig: I think the core design is good, but I'm just having, like, the technical concerns I mentioned, implementation-wise.

Enrico Del Fante: And if the bid is not accepted by the Beacon node, everything will fail or… Do you?

Potuz: Yeah, but what's the problem? I don't understand why.

Enrico Del Fante: No, no.

Potuz: Why is that a problem?

Enrico Del Fante: No, no, no, I'm just asking.

Potuz: say no.

Enrico Del Fante: I'm just asking if it's desirable to have defaulting to a bid that the Beacon node wants, or is just a behavior that is preferred? To not… to error out, because maybe there is some… there are some other failover mechanism.

Luca | Vero: Right now, the idea is that, if the Beacon node doesn't like the bid for any reason, it falls back to either a peer-to-peer bid or a local build.

Enrico Del Fante: Okay, so an actual recovery is, is It's, it's good for your… from your design.

Luca | Vero: Yes, that's right. The bid inclusion is currently referred to as best effort, so it tries, but it's not required to include it.

Miguel Tenorio: Also, there is a use case for this, so if we don't standardize it. with such an endpoint that Luca is proposing. We might risk people implementing their own custom integrations. Like a proxy seated between the Beacon node and the Validator client. We could avoid that.

Luca | Vero: Yeah, maybe one more thing. Even today, people use Vouch as a… Sort of, MEV-Boost. So, I think there are definitely node operators that are used to having their validator clients connect to builders. In that way. It's a little different than what I'm proposing here, but… Also a bit similar. Okay.

Jihoon Song: All right, so it sounds like.

Luca | Vero: Yeah.

Jihoon Song: We don't have enough consensus to merge this PRSS, but it seems it does bring us some benefits. So… perhaps the right way to go forward is discussing more on the design, whether it be this PR or slightly different version. Or on the Beacon node, or whatever. And perhaps we can have it on Hoodi.

Luca | Vero: Yeah, let me create a thread on this story and we can continue there.

Jihoon Song: Thank you. The. Next agenda I want to discuss is… these two PRs is when a builder wins the bid and exits. And at least one epoch is missed. We have two solutions to this problem. And 5695 has had support from clients. The reasons are… It is a less breaking change, and 5594… sorry, 5694. Draft the payment to the proposer of the last block before the missed slots gap. So I'll suggest merging 5695, but I wonder… If there is any concerns or comments?

Nico Flaig: What do we need to make a call on this? That's mostly the question, I guess, right?

Jihoon Song: We just need more… I just wanted to make sure there is no pushback. Because there was some last time. But if not, we can just merge this in.

Nico Flaig: Because, yeah, if there's no pushback against my PR, I would favor that over Potuz' PR, but… I'm not sure everyone has taken a look, and I think it's quite something everyone should at least have looked at. At least more people should have looked at yet, so… I'm not sure Potuz is happy with my PR or neither happy nor unhappy, just neutral, right?

Potuz: I'm not, but I'm… I'll be more unhappy if we don't decide this now, so I think it's fine to merge it. At least I do not know how to attack it today. It might become attackable later, it might constrain the type of requests that we need to. Whenever we add a new request to the future, it might make us think of more edge cases. The other one seems much simpler. But anyways, I would rather just merge anything and just get be done with this problem.

Nico Flaig: I do think it's strictly safer to reorder this as my PR does, considering if we add later more requests.

Pawan Dhananjay: From Lighthouse side, I haven't looked into it personally. Last time I checked, Michael slightly preferred Potuz's PR. I'll check with him again today, and maybe we can merge it.

Potuz: No, no, no. Michael explicitly said, the last we heard from Michael explicitly said that he prefers Nico.

Pawan Dhananjay: Okay, then let's go, I guess.

Potuz: And I'm fine with merging Nico. So I don't know how to attack it. And I really did spend some time in trying to attack it.

Jihoon Song: Okay, I'm gonna work on merging this, Nico's PR, 5695. Thank you all. That's all the agenda we have for today, but Barnabas said in the chat that we have 6 open PRs to pick on APIs. Which are these, so I wonder if you want to go over these PRs?

Barnabas Busa: They're all Gloas related, so I think it would be good to hash out as many as we can. In the next 15 minutes. Yeah, we have some time.

Jihoon Song: I'm here, so we can go over the…

Nico Flaig: So, I mean, the current state of all of these is basically… I think I already approved all of them, and I even have them implemented in Lodestar. I just kept them open to get more approvals from other clients. Spec-wise, they look good.

Jihoon Song: Have other clients had time to look into these PRs?

Nico Flaig: Also worth noting, these are trivial. This is Compared to the other endpoint we discussed with the bid, these are way more trivial.

Jihoon Song: Pawan in the chat said, we'll take a look into this, and… Leave comments. Can we have… can we have other clients to also have it? Look into this.

Enrico Del Fante: Yeah, I will get them all. I believe I have only looked at the last one, which I provided some feedback previously, but the other one… I will, report back into the team and take a look. Seems like, nothing harmful. At first sight.

Jihoon Song: Thank you, Enrico. Right. Do we have anything left to discuss?

Barnabas Busa: Are you going to have the CL spec release?

Potuz: That spec release, we're gonna need it if we want to merge the change for the ordering.

Nico Flaig: You don't need a spec release for that. It's like applying this change in the client, you still pass all spec tests.

Potuz: Oh, the problem is that there's no regression on the… on the… there's no regression… the regression is only included in your PR, right?

Nico Flaig: Yeah, my test will catch if you don't implement it. But it's green on whatever we are on, like beta 2 or something.

Potuz: So we're gonna be like forking probably Hoodi without… without a test that actually covers this change. We're gonna actually make a consensus-breaking change, and… We're gonna go without, yeah, okay, anyways. I'll open the PR on Prysm for this now.

Jihoon Song: Yeah, as soon as we can merge this PR, we can discuss and make a release. regarding… focil images in other clients. I believe we have Lodestar and Teku, and that's it. Lighthouse is working on it. And yeah, that's pretty much it.

Enrico Del Fante: But what would be the effort to add in that PR just a reference test that will break if you don't, implement that way? would be safer.

Jihoon Song: From my understanding, it will only break when… The builder that won the last bid. Request for exit, and we missed at least one epoch.

Enrico Del Fante: Yeah, I mean, I know it's complicated, but I don't know if, from the testing perspective, it's a doable test right now with the tooling that we have. to recreate the scenario or not.

Jihoon Song: You mean in reference?

Nico Flaig: Wait, wait, I have a test. What I was saying.

Enrico Del Fante: For instance.

Nico Flaig: No, I think maybe that was misunderstood. So if you implement this reordering today in your client, you still pass the current beta 2 vectors. Because no test covers that, but in my PR, there is a new test that would catch if you don't…

Enrico Del Fante: Okay, sorry, sorry. Good. Okay. Good, yeah.

Jihoon Song: Okay, cool. Any other questions or comments? It seems we're good. Thank you, everyone. Let's go ship Sepolia.

Pawan Dhananjay: people.

Nico Flaig: Yeah.

Potuz: Bye bye, guys.

Jihoon Song: Bye bye. Thank you, Will.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.