All Core Devs - Testing #101 (CL Breakout)

All Core Devs - Testing · 19 Oct 2026 · 35:42

A breakout session for All Core Devs Testing call #101, focused on consensus layer client testing. It covers a spec change for heze-devnet-2. There is no new spec release for heze-devnet-2, which uses the previous beta as is, and quick slots stay on a feature branch only, not included in heze-devnet-2.

Speakers

Topics

From All Core Devs - Testing, hosted by Ethereum Foundation.

Transcript

Justin Traglia: Hello, everyone. Just to… 30 seconds or so for everyone to join. Okay, let's get into it, since we only have, like, 5 minutes left. Luca, are you here to give an update on produceBlockV4WithBid again?

Luca | Vero: Yeah, I'm here. So I posted some additional motivation to the PR about 2 weeks ago. There hasn't been any feedback until the start of this ACDT call, so… Yeah, I'm looking for feedback. That's the gist of it. I would love to make this an experimental endpoint. I've been testing it for over a month on devnet 8. I'm not aware of any real technical issues or technical points against it, so… Yeah, I would just like to hear some arguments against it if we don't want to do it. Because there are good reasons to do it, which I've described in that comment from two weeks ago. But the gist of it is that it's just very useful to have this functionality for, multi-node client-diverse setups. So think 4 different CL clients.

Nico Flaig: Yeah, so the biggest issue, I guess, aside the fact that we need client approvals and, like, clients implementing it. Which I guess I personally don't like about it is… The fact that you basically have to trust the builder to give you a valid bid. So, currently, how the spec works, basically one single builder can grieve all other builders. Even if it's a trustless bid, by just sending an absurd high value. stairway, did you think about how we can address that? Because it would…

heluani: Why does this happen? Isn't the operator that is providing the bid in this endpoint?

Luca | Vero: Yeah, the Validator client fetches a bid from some kind of builder, doesn't really matter, but the problem is that the Validator client doesn't have a good way of verifying that the bid is valid, like… It's difficult to do on the Validator client side.

heluani: But we'll… What's the pro… I'm not following what's the problem. Like, we have many of endpoints in which the Beacon is trusting that it's getting good info.

Nico Flaig: But not from the builder, right? That's the thing. Like, currently a single builder connected via the Beacon node cannot grieve other builders by sending an invalid bid, because the Beacon node has all state and fork choice info to reject that bid and just not select it.

Luca | Vero: Yeah, while the.

Nico Flaig: Validator client cannot, like, just has to trust what the builder sends, which kinda is not aligned with ePBS in general.

heluani: This is…

Nico Flaig: notion.

heluani: But this is just the usage of this endpoint, right? So if that Validator client does not have a way of trusting whatever it's providing, then it's the problem of that operator. It's not… It's not the problem of the chain. I fail to see the problem. Like, I already trust that my opera… my operator will just give me good info to produce good blocks.

Nico Flaig: Oh, no, the problem really is that those operators cannot connect to builders they do not trust. That's the problem. Whereas if you connect via the Beacon node, this doesn't matter, you can connect to any builder via URL, you don't have to trust them.

Luca | Vero: Yeah, I don't think that's a problem with the endpoint, right? It's just a problem with the approach, and We… as long as we make it clear to the node operator that this is a downside of this approach, I think it… Still okay. It's kind of what operators do right now as well, right?

Nico Flaig: Yeah, I mean, it just favors relays over other builders that are not trusted, which is my main thing. Like, it pushes more on the current status quo, rather than in the trustless status quo.

heluani: And so, so your argument is that people that are going to use this system will not be able to contact directly builders. contact directly builder, and they won't even be able to take the P2P bids. So they won't allow the Beacon to provide their own bids.

Nico Flaig: They can use the P2P bid. So, as a fallback, right, if the Beacon rejects the bid the VC sends, then…

heluani: Right, then… but only under rejection. So the Beacon will not get from these operators ever a trustless bid. They would only get bids on the endpoint from trusted builders.

Nico Flaig: Yeah, so I think over the endpoint, like the builder API, you can only connect to builders you trust with this approach. Which, if it's a small amount of the network running this, that doesn't matter, but if it's, like, more than 20%, I'm not sure what the vouch and Vero shares, but this would be bad for the ecosystem, in my opinion.

Justin Traglia: So could we get some approvals on the PR, or… I mean, it sounds like there's… Support for this?

Luca | Vero: Let me just add the experimental side, I think. I completely understand if people don't want this to be a, like, a production endpoint right now, but…

Justin Traglia: Yep, thank you. Okay. Yep. Clients, please take a look and review when you can. On the agenda, there's one more question from Paul, asking if the inclusion list, deadline should be changed for heze-devnet-2. I think we can just say no, right?

Jihoon Song: Why… why is it known?

Justin Traglia: Well, I mean, it's not included in the spec. Currently.

Jihoon Song: why it's not included is it's already merged, and we talked about this in ACD Breakout.

Justin Traglia: Okay, it's merged into the main branch, but it wasn't included in a release, right? And if we're gonna.

Jihoon Song: That's weird.

Justin Traglia: a specific… Please.

Jihoon Song: Yeah, we didn't cut any release, because… There was some contentious discussion, and it seems like We just decided to not include quick slots in heze-devnet-2. We can't just cut a release with other… PRs, if there are some changes to be made. Of course, an out of bounds.

Justin Traglia: Sorry. Other than quick slots and this PR, is there anything else that we should cut a release for? Or, like, any other reasons, too?

Jihoon Song: Not that I'm aware of, unless I… Missed something from last week.

Justin Traglia: Okay. And it's just this.

Jihoon Song: 50 feet. then… then… Without this change, and no new release. what's the difference between heze-devnet 1 and 2?

Justin Traglia: But does heze-devnet-1 include the two CFI'd EIPs? I don't think it does.

Jihoon Song: Right, so Beta, Beta 4 already included them.

Justin Traglia: Yes, I.

Jihoon Song: Yeah, yeah. Yeah.

Justin Traglia: I would, argue we don't make another release and use Beta 4 for heze-devnet-2, which means that this EIP… sorry, this… change to the deadline is not included in the next devnet.

Barnabas Busa: Yeah, so… It's just a con… it's just a config value. It's really whatever… It's not important.

Jihoon Song: Yeah, yeah. Yeah, just wanting to be on the same page.

Justin Traglia: Yeah. Okay, so we're already 3 minutes past, Regarding those two topics, does that sound fine, or is there anything else that we need to discuss there?

heluani: So what is the scope in the end for devnet2? Are we doing quick slots in the devnet, or we're not doing the quick slots in the devnet?

Justin Traglia: I would argue against it.

Barnabas Busa: I would argue for it.

Matthew Keil: I'm with Justin.

Justin Traglia: I mean, I always wanted it to be optional, to where we could just test with kurtosis locally. It… Barnabay reasons not to do it. we'd have to cut a new release, and figure out some testing issues, and it would make it harder for other clients to join devnet 2. Those that don't have quick slots support yet. If we launch the devnet with a different slot duration.

Barnabas Busa: Didn't we, like, discussed a couple of weeks ago that if… clients would be able to ship it, then we would do it. Otherwise…

Justin Traglia: Yeah, I think I did say something about that, like, if 3 clients…

Barnabas Busa: We could. Oh, that's fine. I don't know, I mean…

Justin Traglia: things have changed, and I feel like there's not as much support for that right now.

heluani: I don't mind testing this. I don't know if we can support it, but I don't mind if this is tested in the devnet.

Barnabas Busa: Like, how can we merge in the PR if we don't test it?

heluani: Yeah, that's a whole… the whole thing. I don't want this to be… let's not go in circles. I think people don't want this to be… or some people don't want this to be merged into the main Heze branch. And we can test, and we have test vectors in the feature branches.

Jihoon Song: I mean, I can make another PR that… puts this quick slots in a feature branch based on the latest Heze, if that's what people want. I think the first question that we need to answer is whether we want to have quick slots in devnet 2 or not.

heluani: Yeah, I agree with this. And I'm fine either way.

Matthew Keil: I think I'm also fine either way, but with the specific, like… The nuance that the spec gets merged to a feature in the specs, not to the main Heze.

Jihoon Song: Yeah, just to be clear, it is already in the feature branch, so the new PR, we're gonna… be the one that's updating it with some comments in the… those two PRs.

heluani: So, so the current spec is based on Gloas, Right?

Jihoon Song: I believe it's based on Heze…

heluani: then… then why are we arguing about merging or not, Barnabas and yours? Barnabas seemed to be moved too ahead.

Jihoon Song: No, no, both are… no, both are targeting to Heze, and Barnabas PRs contain some more changes on top of mind.

heluani: Okay, so then, then they are targeting Heze, okay.

Jihoon Song: Yeah.

Justin Traglia: Yeah.

Jihoon Song: Yeah, I can change the target to the feature branch and add some more tests so that clients can test their implementation.

heluani: You're thinking?

Justin Traglia: We need to keep quick slots as a feature branch for at least, like, another month or so, until, like, after Devcon.

heluani: For us, it's very simple to test that way, because then… then we could just pull in on the branches that were implementing this. We can pull in those test vectors. Whether if you're merging on the main branch is quite disruptive, because then we need to start skipping stuff that might be important in our production code.

Justin Traglia: Yeah. Right. And Barnabas, I think we… well, just Barnabas and everyone, I do think we need to change how things are done to be more of that. CFI EIPs are just features, and… we test them in a devnet by themselves, and then after we all agree to it, then we merge it. So then we SFI it and merge it into the Heze specs, or into the upgrade specs. I think something… something needs to change. Francesco?

Francesco D'Amato: Yeah, I just wanted to say that that's fine, that seems like an okay process, but I think, yeah, we just need to kind of clarify this, probably, maybe, like, in the main ACD on Thursday, or whatever the next ACD is, because it's a bit confusing right now. I think, like, there's an expectation that, you have to get on a devnet in order to be SFI'd, but now we're saying that you need to be SFI'd in order to get on a devnet, like, or, like, an actual devnet on, like, a feature devnet, so I think if if we want the process to be featured devnet first, then we make the call, and then we put it on an official devnet, that's fine, but we just need to kind of…

Yeah, make it clear that that's the process.

Justin Traglia: Yeah, I agree. We'll make this more clear on a better call, not in the last, like, 5 minutes of a breakout. Potuz?

heluani: No, no, I disagree with Francesco. I think that the problem is the last part of the assumption, that I think that comes from Panda. I think we can test on devnets, and it does not need to be on the main fork branch to be tested on devnets. I think the whole procedure should be, we have feature branches, we… for CFI EIPs, we test them on devnets, from those feature branches, we agree on here, or whatever, what goes into a devnet, we test this, once we think it's fine, we specify it, and we merge into the main branch. It doesn't need to be in the main branch to be tested.

Barnabas Busa: But this is not what we did till now. That's… that's why I'm just…

heluani: We did both. We did merge prematurely also before into the main fork, but now it's getting more… it's getting tricky. Like, ePBS itself never merged into the main fork until it was CFI. Like, especially for large… for large EIPs, we should be careful.

Barnabas Busa: headliners, of course, but small EIPs, they were all merged into the main branch before they got classified. They all, for Amsterdam. Every single one of them.

Justin Traglia: That is true. Jihoon?

Jihoon Song: I just want to touch Barnabas and NC question in the chat. I… I think we… already covered this in the previous room, but I just wanted to be very explicit, like, do we wanna… pull off the other two EIPs from Heze.

heluani: Please don't pull anything. It's just more work. We're already passing test vectors in whatever it is on the main branch.

Justin Traglia: I would also say no. I felt like there was really strong support for those, EIPs. End. They're… yeah.

Jihoon Song: Yeah, yeah, thank you.

heluani: But we can probably try to SFI those EIPs, and then XICD, so that they are there and they are already SFI.

Justin Traglia: Yeah, I would support that. If other clients do. Okay, any other topics we'd like to discuss?

Barnabas Busa: I don't really wanna… waste anyone's time, but I would like to see a change to the existing EIP that we included to disallow the 0x00 deposits. Basically, make them inactive instead of burning the… Deep value?

heluani: I'm sorry, what, what's, what's the change?

Barnabas Busa: So, basically, if you make a 0x00 deposit after, D has a fork, then they would become, basically, inactive. And, they would need to do a 0x01 BLS change in order for them to… Erb EF. Instead of burning the ETH, because right now the current EIP basically describes 0x00 deposits post as a fork to be invalid, and… the 32 ETH would be just burnt.

heluani: So what's the… what's the mechanism, you… You… when you process the deposit, you're gonna set a very large activation epoch, or something like this, or…

Barnabas Busa: Yep.

heluani: Or you're gonna have, like, different logic to activate those.

Barnabas Busa: No, no, no, just recycled, original. Ideas, like, the ease eligible for activation would be just, set high. I mean, we can… They can come up with a… actual good, design… down the line. It's just the idea that burning ETH It's a default for a process that Like, a mechanism that was, valid before is not the… not the way forward, in my opinion.

Justin Traglia: I actually like this change, because I think it's just a bit simpler than the other one. -Oh. I don't really care as much about, like, the burning ETH.

Barnabas Busa: Also, we still have some tools that do allow 0, Validators to be created and deposited, so it's not like it's officially been deprecated everywhere.

Justin Traglia: Okay. If there's nothing else, I think we should end the call. Yep, I agree with what Jihoon just said. Let's bring this up, at ACDC this week. Does that sound fine in front of us?

Barnabas Busa: Yeah, just hope that we can finish up the Heze scoping quick, in the beginning of the ACDC, because I… I was hoping to bring up this, They were already in the previous call.

Justin Traglia: Yep. And Potuz, I agree. I mean, I would actually like to make this call an hour and a half. Yeah, if there's more stuff you guys want to talk about, I'm happy to moderate. I just thought. Afraid of going over too long. In case other people have other meetings. Would you like to continue talking about Barnabas' proposal now? Or… Anything else?

Barnabas Busa: I think people would need to read the, the PR change first, and then the Consensus back, Tria So there's the two… Two peers.

Nico Flaig: So, but it complicates the spec, right? And I think NC did the analysis of this, and there's, like, two people in the last half a year that did a 0x00. So I'm mostly wondering, can we solve this now at the tooling layer, and then maybe revisit if there's still people depositing this, or…

heluani: So the EIP as it stands now seems to absolutely trivia. We just dropped the deposit. And that's it. It seems to me that Barnabas change is strictly more complicated.

Nico Flaig: Yeah, yeah, that's what I'm arguing for. Like, do we need Barnabas' change or not? Because current EIP is really simple, so… The current JPM.

heluani: It's trivial, it's trivial, and Barnabas change sounds to me that it's going to be slightly more complicated, probably easy too. I'm not really sure if it's going to save any money at all.

Justin Traglia: I think I'd argue that it's about the same complexity. There was always a question about where to drop the deposit. as well, like… I believe the EIP drops it in the applyPendingDeposit function. So it allows the deposit to be in the pending queue, which I always find to be a bit weird, instead of dropping it before.

heluani: Yeah.

Justin Traglia: in the deposit request, but I know…

heluani: I haven't even looked at this. I haven't even looked at this, but yeah, I would just drop it when we process the deposit request.

NC: Yeah, so I opened a PR draft on the EIP's repo. That does exactly this, and it's also very simple to do.

Justin Traglia: I can't remember, but I thought there was a reason why we couldn't drop it at the deposit request level. Maybe that's… maybe I'm wrong.

Barnabas Busa: How would you do that? How would you drop it?

Justin Traglia: How would I drop it at the deposit request? You would update the request handler. And if… Oh, that's right, so for top-ups. So currently, we allow any validator Withdraw credential for top-ups, right?

NC: Yes.

Justin Traglia: And the… Going forward with top-ups, you would have to set it to a non-0x00. value, which is, like, not a big deal. I don't think anyone uses 0x00 for top-ups.

heluani: Wait, wait, wait, you…

Barnabas Busa: No, we didn't… when…

Justin Traglia: What are you saying, Curtis?

heluani: So, so we do, but still, you can just check that it's a top-up at the processing of the request. Still, it's… Probably.

Justin Traglia: No… no, I think that, You don't know if it's a top-up in the… deposit request handler. You know if it's a top-up in the pending deposit. Handler of sorts.

heluani: You can check on both, but you have the state anyways on both. But yeah, so I, I, I did myself a top-up with zero, withdrawal credential.

Justin Traglia: I guess, yeah, you could check to see if it's a top-up.

Barnabas Busa: And if you make a top-up on the launchpad, it's gonna be 0, credential 2. That's the default, basically.

Justin Traglia: I don't think so. I checked my top-ups recently, and I thought they were all 0x02.

Barnabas Busa: I did one last week, and it was 0x0.

heluani: Yeah, my top-ups on mainnet were always zero, on the zeroed-out everything.

Justin Traglia: I'm misremembering, though.

Barnabas Busa: a little bit.

heluani: I didn't… I cooked them myself. Anyways, I don't mind Barnabas' changes. It's just…

NC: I think there's also one point, is that because now… we have brought up this issue that, regarding 0x00. So, in the field… because Nico's point is to look back, to the past 6 months, maintenance stats, right? But since… We have already brought up this issue, so we can forecast in the future there will be also less, likely there will be 0x00 deposits. Been the past 6 months.

Barnabas Busa: Yeah, I'm not…

NC: So…

Barnabas Busa: But then, I'm not expecting a huge inflow of 0x00 editors, it's just… technically, when you create a 0x00 Validator today, and you don't make a deposit, that is a valid operation, and I just don't think that any valid operations should automatically become a burnt-Hit transaction after a fork. Like, if someone doesn't keep up with ACDC, or doesn't read the blog post, or doesn't… you know, something, like, the default behavior should never be that we just burn that ETH, especially if the change to not burn the ETH is… Pretty trivial. Like, we are, yeah, we of course are in the loop that, we deprecated 0x00, people should not make it happen, but…

Not everybody is following the process as closely as we do. Like, if you're just a random person, and you just want to make a 0x00 deposit for whatever reason. You should not be punished for it. Ever, in my opinion.

Justin Traglia: I think I agree as well.

NC: I mean, what if you submit a 0x04 or 0x05 deposit? Those walls of guests.

Barnabas Busa: August.

NC: Whoa.

Barnabas Busa: But, yeah, but… you, you, you wouldn't make that accidentally. Like, 0x00 is a valid operation today. 0x04 has never been, and never, like, it might never will be a valid operation, but 0x00 is a valid operation today. And just because we make a blog post that, hey, this is deprecated, hey, don't use this, hey, don't use that, it doesn't make it… Less likely that… There will be a single person. That might lose 32 ETH because of this. And that is a ton of money.

Justin Traglia: I'm sorry, what are the current… Sorry, doesn't this only apply for newly deposited? like, new deposits, not top-ups. Like, with the current spec, wouldn't… top-ups also… Be deposited like normal?

Barnabas Busa: No, this would be just 0. Deposits.

Justin Traglia: Yeah, new 0x00… sorry, new 0x00 deposits, yeah.

Barnabas Busa: Yep.

Justin Traglia: So, I mean, I was gonna say, like, if this negatively affects top-ups, I don't think it would. Like, it doesn't affect top-ups right now.

NC: Right.

Justin Traglia: Hmm. Yeah, we just need more inputs from clients, I guess. I don't have, like, a super strong opinion on this.

heluani: So…

Justin Traglia: version seems to work fine.

heluani: I'm sorry to interrupt you, Justin. It seems that the easiest, from an implementation perspective, the easiest change would be not something that Barnabas would like, but the easiest change would be to just change the function that creates a validator, and never create a Validator with a 0x00. Withdrawal credential.

Barnabas Busa: But then how do you get your ETH back?

heluani: You don't. That's why you wouldn't like this change. But that seems to me the easiest spec change.

Justin Traglia: Barnabas, I mean, even if you made a 0x00 Validator, like, in your proposal, I don't think you'd be able to get that ETH back either.

Barnabas Busa: Yeah, you can, because Barnabas allows… Barnabas allows you…

heluani: And that's changed.

Barnabas Busa: Yeah.

Justin Traglia: Oh. I guess, yeah, that's right. You could just do a BLS change.

Barnabas Busa: point that if we can… if this would just happen with a single person accidentally, I would already feel bad, because that's a… it's a shit ton of money for a single person. And there's just no good reason to burn the seat. Even though the 0x00 is deprecated, or killed, or whatever. Like, this is not about, you know, reducing the Validator state size, because obviously we don't have that many 0x00 validators anymore. It's just… Keeping that… One poor guy, that accidentally made a bad deposit still be able to get his money back.

NC: I think I mentioned this in your PR, is that essentially creating a path That, you're allowing an inactive 0x00 validators to be in the registry, and the only way to get them unstuck is to use this BLS execution change. So you're essentially creating more entities that rely on this mechanism that, That is destined to be deprecated. So,

Barnabas Busa: I didn't tell your office contract.

NC: Come.

Barnabas Busa: The entire deposit contract is just going to be deprecating, because we're gonna have a post-quantum keys registry. So it's… it's not like we can do anything with the current Validator state size, it's… It's… it's already too big, it's… it's gonna be X anyway, so… So there's no… no reason for us to… us to worry about this one or two extra 0x00 deposits. We just need to rethink the whole deposit Contract, and basically start from scratch for the post-quantum case.

NC: But as you say, like, this really impacts us so little. Like, I don't know, I just…

Barnabas Busa: It does, yeah. But it's not zero. That's my point.

NC: I don't… my opinion is, like, I don't want to spend too much time thinking about this, because it's quite trivial, and I would… Prefer to have a minimum spec, just get through the devnets. Easy attests, stuff like that.

Justin Traglia: Yeah. I mean, I can relate to NC's point about Not wanting to keep the, BLS execution, or execution, whatever the function is. VTEC. handler. It would be nice to remove that.

Barnabas Busa: Yeah, but, like, the BLS change… To execution, that's gonna be there, whether we want it or not, till at least one more fork, right? When we basically killed the 0x00 completely. Like, this is just the first step out of three. Like, I would actually propose in the follow-up work, we're just gonna… straight up exit all the 0x0 validators, that are still online, and then the one after, we disable the BLS change option also. And at that point, we're probably gonna have the new deposit contract anyway.

NC: thinking a little too far, just because, like, yeah, like, now the vision is to have, like, a three-step process, but it's not guaranteed that the next two EIPs will be CFI or SFI next fork, so… We need to try our best to do whatever we can, like, in this current fork. Without assuming that, the other EIPs will be getting in, in the next, future forks.

Barnabas Busa: I think it would be good to have a vision of what we actually want, and how we want to have the Validator deposit contract look like in the next 5 years.

Justin Traglia: it's not clear to me that the deposit contract will be completely replaced by something new. Like, I… if I recall. Kev was working on making it work with the existing deposit contract.

Barnabas Busa: Yeah, but in that case, we're gonna need to do something about the Validator states, right? And recycling the Validator indexes. Indices.

Justin Traglia: Sure. Sometime in the future, I guess.

heluani: No, not necessarily. The contract… you could reuse the same contract and apply different logic on the CL side to create a new slice. I mean, nothing prevents us from, like, treating 0x03… 0x04 validators as PQ validators. Just add them to a different slice. That's… we could have done that with builders as well.

Justin Traglia: Okay, let's discuss this more later. We're at… we're at time for the 90-minute… slot.

heluani: See you guys in a couple of days.

Justin Traglia: Yep. Talk to you guys at ACDC. Bye, everyone.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.