All Core Devs - Testing #75

Transcript

Mario Vega: Hi everyone, let's just give one more minute, to give an opportunity to people to join, and then we can start.

Alright, I think we can… we can start.

Akash | ECH: Yeah, we are live.

Mario Vega: Thank you. Good morning, everyone. Welcome to ACDT number 75. Today is March 23, 2026. Yeah, let's just get it started. We have an agenda comprising almost entirely of DevNets. And we are gonna decide whether we, split into two breakouts, further into the call. Because, it might, might or might not be necessary, but we can, we can start discussing, and then we can decide on two breakouts, like we did just, sorry, last week. Yeah, let's just get started. So yeah, first item in the agenda for today is the Blob DevNet Serum. I think, Barnabas, you might be the best person here to give us an update. Let me see if he's on the call…

Barnabas Busa: Yep. I can give a quick update. So, in Blob Design Zero, we are testing the partial cell. Tough, and Nethermind and Erigon is failing to stay at the head. This seems to be just an issue in their trunk branches, I'm curious if, Ericon or another mind has a… They will release that we could use instead of the, trunk branches, because the trunk branches seem to be just breaking way too often. So, ideally, we should have a stable release with the GitLab V3, and then we could pin it to that, and we could resync it. Unless people want to dive into… what the… the breaking databases are. I have… I've pinged, both… both client teams with the…

the log messages that I see, and I'm just, curious to hear, what's up. And on the other side, we're still waiting to onboard more, CL teams, once the partial cell implementations are done, for them.

Łukasz Rozmej: We will take a look into… into what you sent us. I… I wasn't aware yet, I haven't… Noticed it yet. Aww.

Barnabas Busa: Yeah, just let me know if you need anything else.

Mario Vega: Alright. Anything from Erigon, you guys, if you guys want to chime in, or…

Andrew Ashikhmin: Yeah, so we are, finishing our 8037 implementation. Most… we are mostly done, still, like, maybe a couple of bugs remaining, and I saw a message from… was it from Stefan, or… About the, the assertive, so I'll take a look into that, tomorrow, probably.

Mario Vega: Oh, I think, this is for blot definite0, is that correct?

Andrew Ashikhmin: No, no, no. It's about, about, about, 8037 testing.

Mario Vega: Alright, yep. But is there anything for Blob.0 from your side, or…

Andrew Ashikhmin: Nothing.

Mario Vega: from there.

Andrew Ashikhmin: No, I haven't looked at the block.0 issue. Okay, I'll take a look tomorrow at it, probably, as well.

Mario Vega: Sounds good. Barnabas, anything else? Notice. Or below them is zero. Okay. Alright. Yeah, let's… let's move on. I think, Last week, there were some issues regarding Ball DevNet 3, and there were problems on clients staying together, if I understood correctly, so let's just jump into Bald Devna 3 issues. Stefan, I think you're the best person here to give an update, or also Spencer? Oh.

Stefan Starflinger: Yeah, sure. So… We are getting some updates from clients in, I think Bezle pushed an update today that I will look into. Erigon has been pushing a lot of updates over the weekend. I'm just looking into those right now. It seems there has been a lot of progress. Nimos EL and Etherix are, very far ahead. I think they're pretty much done regarding the testing. But maybe Spencer can… say a little bit more regarding the testing. I think they're still with, Guess there might still be issues regarding, the contract size increase. I'm not sure if that's just some error mapping, or if there's still some… some tests. Bezo, I hope, will be passing all the tests with the recent push. I will… I'll fix that. Bezo had 19 failures.

Goethereum still has 49 failures, and Nethermind, still has… and Reth, they both have still over 400 failures, so they're still actively developing. But I think we're pretty close now, from the implementation, and I will be doing a lot of testing this week. Over to Spencer.

Spencer Taylor-Brown: Hey, yeah, I don't really have… have that much to add. I think, Yeah, a lot of the failures on Hive are, our exception mapping failures, so… Yeah, I guess… over the next day, day or two, I'm definitely gonna be pushing to get all the tests passing. Passing on Hive, yeah, there's been good progress over the past… the past week. It's worth mentioning that We might have another spec change, like, tiny, tiny change for, like, a single test. but we can, deal with that for any other future spec changes to 1837. And the next definet. So, I think, we're all in agreement to stick with what we have for now, get everything passing, and then… future updates can happen for DevNet 4.

The bug that Stefan found on Erigon. And we don't actually have a test for that, so… Yeah, I'll get working on that as well.

Mario Vega: Sounds good. Any comments on DevNet 3? Either specific to 837 or anything else, from clients. Who want to chime in? Any issues with the tests? Or, concerns?

Dragan Rakita: From Redside, I run… I have run all the… the state tests, the latest 5.5… 5.5.1, There are a few inconsistencies, but as Spencer said, we'll move that changes to the DevNet 4? So… Yeah, des… Text feature seems okay, no.

Mario Vega: Alright, anything else from other clients, basu, Nimbus, or… Geth? Are we doing okay with this, or… Do you guys need help? Great. Great. If not, I think the next point in the agenda is, block load access list definite number 4. I do feel like we're still, like, trying to iron out the issues on DevNet3, but I wanted to open the floor for discussion if we want to start, Yeah, discussing, WN4, changes. Stefan, how do you feel here? Do you feel it's too soon, or should we defer this to next week, or should we start… This week.

Stefan Starflinger: Yeah, I think it would make more sense to defer from my perspective. But I'm open to discussion if anyone wants.

Mario Vega: Yeah, I agree. As we…

Stefan Starflinger: can talk about, like, would we want to create a Glamsterdam Definite 1, instead of having a Battle Definite War?

Mario Vega: That's a great question. I think we should, wait until we discuss the EPVES Debit 1. Actually, we should jump into that. And then, and then discuss cards are done, definitely one. Alright. So, ePBS.Net1, do we have someone with the status on this?

Barnabas Busa: Epv.1 is currently on hold until we can make a decision on the PTC questions.

Mario Vega: Okay, great. Alright, so, at this point, we should decide whether we want to do the breakout into separate rooms, or we want to stick into this. I feel that the only decision that has to be made for ePBS is PTC. And… from the sound of it, this looks like it's gonna be a quick one. So, should we start just discussing that, and see if we can close it out, and then jump into block-level access to this right here? Is that fine by everyone? Yep. Alright, yeah, Botus, stick it away if you want.

Potuz: Oh, I… we said that we would have clients coming with options, so I started asking clients what they prefer.

Mario Vega: Sounds good. Alright, let's start with the client's, Either any of the SEAL clients would want to kickstart the conversation of BTC?

Stefan Bratanov: Yeah, I can talk for Teko. I think we… Well, from my perspective, I prefer the newest version. Which is basically, computing the PTC during process floods. Makes the state size a bit smaller. The only thing is the additional computation process loads. But I don't think it's that much of a big deal. It's a good, Alternative to the previous one.

Mario Vega: Yeah.

Stefan Bratanov: The one that, Justin showed.

Mario Vega: Alright, any other clients I want to chime in? And with, preferences?

Eitan Seri-Levi: For Lighthouse, I don't… we don't have a super strong preference, but we think it might be useful to have the PTC in the state. So that we don't have to, like, recalculate it for older states, if we ever need that for some reason. But I think it's maybe more useful to have the full, PTC, for the epoch instead of just the last one. But yeah, we don't have, like, a super strong opinion.

Mario Vega: Any other CL clients want to chime in on PTC?

Dustin Brody: My tendency is to prefer things in-state to not in-state for actually testing reasons, largely. I think we've seen Hmm. Well, many bugs over the years, that amount to the caching was wrong. And across every client, essentially. I don't… I'm not gonna single anyone out. And this both creates… by putting in a consensus, means that, like, that there will be… A consensus test… testable thing, and then that will be, sort of, the same across, sort of. Tests were tested in that regard. I don't have… yeah, that's sort of my bias, I guess, here.

Mario Vega: Yeah, but this?

Potuz: Yeah, for Prism, it's essentially the same thing that Dustin just said. We… we would prefer having this on the stage. It's not a strong preference. 50-20 works. 42.92, I think, is the other option, also works. It's fine for us. But our preferred method would be not only the previous version that has two epochs in the state, but actually add an extra one, and keep three epochs. One for the previous epoch, and the next two epochs, so that we have duties also. In a finalizing chain. we should, by just having the dependent route on the state, we can just pick any state, any head state, or any state on any branch, and it would just give us the PTC

It will solve completely what caching is, and yes, it adds, between 256 and 384 kilobytes to the state, but the state is several hundred megabytes anyways. So, it avoids hashing, it avoids rehashing, you only need to compute it once in epoch, it solves completely the caching, and, as Dustin said, it's testable on spec test. I think its pain on space on the state is not There's not much punishing.

Dustin Brody: Yeah, I would generally… I mean, to me, the size is not the argument, I mean, against that, if such an argument exists. I'm sure some people are bothered by that size, but given that, as you say, the size is several hundred megabytes and growing with every validator, Because, of course, that's… it's not only the active validators, then… This is… the relative… the ROI is pretty high on this for me, even for 256 or 384 kilobytes.

Mario Vega: Alright, mark, any other guys who wish to chime in on this? I feel like… It's a little bit contentious, so it seems to me.

Nico Flaig: Yeah, so from Lodestar's side, I guess we are also fine with all three options, to be honest. So I was working on, the 4992PR, And found it a bit wasteful that we have to process every slot, and in Lodestar, we didn't even use that data, because we have the cache already. So the… the modifying the state each slot was not ideal, in my opinion. That's why I brought up, 50-20, which is more minimal, and you only need to update the state once. But I'm also in favor of the other option that Paul just mentioned, where you cash restates. Because this also solves, a problem, or it's not a real problem, but basically, right now, if you, you need to get the duties for the whole epoch at slot zero, and if you're the validator and you have a slot zero duty.

You basically have very little time to prepare for that. I think for PTC, this would probably be fine. But yeah, still, I think on duties it's even good to have another epoch of look ahead. So I would also be fine with that change.

Mario Vega: Alright, it doesn't seem like we have a… A strong… Decision. Pathos, do you feel like, should we take this into the breakout room? And then make the decision, or…

Potuz: I think we could make a decision here. It seems that it was essentially everyone on the same PR except Techu. And even within Checo, I see Enrico being convinced otherwise.

Enrico Del Fante: Yeah, I think, like, the testability argument and this lot zero timing, that has been… Improved on the block proposal. and we get something like that for BTC as well, I mean, I think, like, at the end makes sense to have Some bias toward having more data on state, and having more stable things, and testable things. That's my personal…

Mario Vega: Alright, do… are we missing any sealed lines from commenting? Alright, should we make a decision, or what is how do you feel? Or what's the… What information are we missing to make a decision?

Potuz: I think we've already made a decision, it seems.

Mario Vega: Yes, and I see your comment on the chat, What, specs and tests are you talking about? Do you have a PR for that, or…

Justin Traglia: I don't know which PROTUS is gonna update, but he said he was gonna make the changes that we just agreed upon. Here. So updating it to be 3 epochs worth of PTC. Values. Yeah, to be clear, the PR's not ready yet, but it will be in a few minutes.

Mario Vega: Alright, so… steps forward should be… seems like, it's updating the PR, and then merging it, and then that's the path we're gonna follow for EPBS, Definite 1. Is that… is that correct?

Justin Traglia: Sounds correct to me. And then, we can produce a release with this, shortly. Maybe end of this week.

Mario Vega: Excellent. Alright. And one last question from Barnabas, do we do variable PTC deadline? Just to close it out.

Potuz: No, we were never going to test benchmarking, on DevNet 1. DevNet1 is supposed to be about testing the new structures. So, we don't even want to be dealing with the builder API, for example, so we're gonna be just only testing on DevNet1, external builders, Withdrawals from builders and these kind of things. But not even the Bilaterabia.

Barnabas Busa: No, I meant, do we do variable PTC deadline in.

Potuz: Yeah, so that's something that is just not a hard fork, it's not… it's more about benchmarking. I think those kind of things should be discussed, like, several weeks, months from now. At any rate, there's… Tony has pulled up a document. arguing against variable B2C. There doesn't seem to be even consensus on whether we want this or not. It's a trivial change that we can add at any time if it's deemed, important. I don't think we need to discuss this now. We can just focus on testing what We have a broken Definite Zero now. We can just focus on when we're gonna ship Definite 1, when we're gonna have this PR, and just continue testing the structures, not the bench.

Mario Vega: Okay, great. So, Justin, share the document in the chat, please take a look, regarding this. Cool. If nothing else, I think we can jump into block-level access lists now. Yeah, I think, Tony, you have a couple of topics that you wanted to raise, are you ready to… To go ahead with this?

Toni Wahrstätter: Yes, the agenda for, the Battle Breaker would be very… small today, so I think we can also just keep it in that call. basically asking clients how far they are with their optimizations, especially focusing on the parallel batch I.O. And we should also have some first numbers on the execution parallelization, and it would be great if we Yeah, if we get an update on that. Yeah, maybe if someone… if someone from Benchmarker is on the call, or… People test, involved in the testing. Would be great. We can't hear you, Johann.

Marius van der Wijden: Well, yeah. I hope that Johan comes back with his microphone, but until then, There has been only, I think, one or two test runs with the… With block access lists, yet. But the one that I looked at… Had a lot of failures, but… The tests that were not failing. And those are most likely, like, unrelated failures. The tests that were not failing, they were significantly faster. So, yeah, the parallel execution seems to have… Pretty good benefit on the worst cases in execution, just as we… assumed, so, that is very nice. I don't know if… Rafael is on the call, or Johann, has this microphone off.

Jochem Brouwer: Yeah, let me check. I see a green thing, so maybe you can now hear me?

Toni Wahrstätter: Yes, it works.

Jochem Brouwer: Okay, great, nice. Okay, so for the benchmarks, we already have a lot of benchmarks, but for the clients, we would need, we need these flags. Especially the prioritization is now the batch I.O, so we need a flag. for clients to enable slash disable the batch I.O. optimization. So this batch I.O. optimization is that you read the, well, your prefetch. the, block access list, reads, so that you can quickly get them when they are necessary, when you are executing the, the benchmark. I see on the Ball Definite 3 spec, and also on the ball definite 2 spec. That's, Nethermind, Eragon, and REF do not have the… sorry, Ball Definitely, Nethermind and Eragon do not have the batch I.O. flag, so, we cannot,

we cannot run the clients with this batch.io flag enabled slash disabled, so we cannot see the, well. the effect of this optimization, so the request for these clients, so NetherMind and Irigone, is to enable this flag to specifically allow this optimization. Then for, the… other part, this is about the filling of these tests. Camila's been working really hard on this, but just to check here, for these benchmarks, I think the only thing that we need here, or well, yeah. the numbers what we need here is the performance, and we don't need, like, a trace of this. I just want to check with everyone if this is correct, so we do not need, like, an opcode count, because this is necessary for the gas repricing, but we don't need this for the FX for the batch I.O, is this correct?

Toni Wahrstätter: Sorry if that was directed to me, could you quickly repeat that? I didn't get it.

Jochem Brouwer: Yeah, it was more like a general question, but for the batch I.O, the thing that we need here is, like, we need a comparison between the runs of clients where the batch I.O. is enabled and the batch I.O. is disabled, and we do not need, like, an opcode count, so, like, a debug trace. Is this correct?

Toni Wahrstätter: Yes, exactly. We're mostly interested in stateful tests between enabling it and disabling it.

Jochem Brouwer: Okay, great. Okay, then, Camille, the book here is that we need to fill these tests, and we are currently using Nethermine to do this, but there seems to be a bug or, like, a memory leak in the bug building part. So, I would suggest, or, well, yeah, let's use another client to fill these tests, so that we can start running these stateful tests also. Because we're currently running the complete test, but that's of course not… But it should not have a large effect on, on the batch I.O. numbers. So let's fill those on another client, like Peso or GEF, or whatever, supports filling these.

Kamil Chodoła: Yeah, I will… I will try to use any of those, and still will be debugging what's happening here, Nethermind.

Rafael Matias: Also from my side, so we do already… so, end of last week, I was able to run Geth. and BESU, like, with and without optimizations, and without balls at all, like, sequential. And, yeah, I shared, like, the results with, Tony and some other folks, and you could definitely see, like, the performance, gains using balls. And today, Stefan also told me that Wrath also has these flags now, so I'm just right now doing a run with Wrath. And yeah, once it's finished, I will also share the results.

Jochem Brouwer: Yeah, I want to quickly add, because we are really interested into this, specifically, this batch I.O. thing. So for the clients, we need this endpoint where this prefetch is enabled or not, and not that we disable, like, the entire block of access list optimizations without it, because then we are also Taking out, like, parallel execution, and other optimizations which clients might have. This batch I.O. is necessary because we have to decide, yeah, on the final spec of the dock access.

Rafael Matias: Yeah, so right now it's just Nethermind and Erigon missing. On that front.

Marius van der Wijden: But Raphael, you're running the test on Definite2 spec right now, right? Okay, when… And we expect to update that to .3 spec? once DevNet3 is launched, I guess?

Stefan Starflinger: Yeah, exactly. That's… once it's stable enough, then we switch over to Definite 3 specs for the benchmarks. The idea is to still have Definite 2, just… So we can do a first iteration of the benchmarks. But not necessarily for the numbers to make a decision on.

Jochem Brouwer: Okay, so just to iterate on the asks, so this is an ask for every GOM and for NetherMinds to add this flag to enable or disable this batch I.O. optimization.

Stefan Starflinger: Yeah, and if Nimbus EL or Aetherax also added, that would also be great.

Andrew Ashikhmin: Well, we don't have that optimization in Erigon yet, so we just have to implement it.

Dustin Brody: Sorry, could you clarify what the… Flaggers, exactly?

Stefan Starflinger: So if you look at the spec, there are 3 different flags, And the most important one is the no prefetch, or the prefetch one, where there is an optimization to read all of the state. In the beginning, and we want to enable and disable that to see the performance.

Dustin Brody: Right, and this is prefetched, like, in isolation, right? Not the other parts, right? That's…

Stefan Starflinger: Yeah, exactly, in isolation, yeah, to just trigger this one optimization.

Dustin Brody: Right, right. Okay, understood, thanks.

Rafael Matias: Yeah, I've seen some people have asked, like, for the link of some results, I'm just getting it, and I will share it here in the chat.

Mario Vega: Okay, Tony, is that the entirety of blog topics, or do you want to do… Discuss something else?

Toni Wahrstätter: No, I think that's it. If no one else has something to discuss, so those optimizations and the first benchmark results are basically it. So far, it looks like baseline GEF get a 3X out of the execution parallelization. And yeah, we are still working on the stateful tests to test the batch.io.

Rafael Matias: Also worth to note the 3X we're mentioning, that was, with a limit of 4 CPUs. Yep.

Mario Vega: Alright, sounds good. Any other topics regarding block.access lists? There's one last topic that my client brought up in one comment on the agenda, so I wanted to jump into that. But I want to check first if there's nothing else from blog.access list that we want to discuss today. All right, cool. Like client, are you there? I think there's one comment from you. Yeah.

Lightclient (Matt Garnett): Yeah. Hey guys. Just wanted to quickly bring up the documents that we came up with last week on the frame transaction then pool. I know this was one point of contention on the last All Chords at its execution. call. So… The way that we've written it is we first wrote a… kind of, like, broader HackMD. That goes over… the general ideas of, like, how do you build a mempool for native accounts abstraction? We talked about the… some of these, like, shared validation rules. These are things that were really pioneered by the ERC7562. And then we lay out a handful of strategies that we feel are, like, reasonable approaches to building the actual transaction pool. Then the second link is a PR to the EAP, which is a more detailed implementation of the spec for Strategy 2.

I can quickly just, like, say the strategies. The first strategy is self-relaying. Oh, so that means that the… Consensus nodes all support users who submit AA transactions and are going to pay for their own gas. This is, like, you know, the standard use case. Today, for transactions? But they don't support in the transaction pool gas abstraction. So it's still supported by the actual changes in the protocol, but we aren't going to, in that strategy, allow people to propagate, like, bundled transactions. And I think that if we did strategy one, it's quite simple. The risks are fairly minimal, in many ways, similar to what we already have with 7702 transaction pool. But the sponsored transactions will have to go through some, like, private flows, rather than being in the public mempool.

We feel that this is… this is a really good strategy, and it's a good fallback if we decide to do strategy two, which is kind of what our preferred, you know, preferred strategy is right now. It's a good fallback if, like, through that process, we have too much contention that we can't move forward, or we find problems that, Are not easily solvable. So, that's, you know, that gives us native cano structure of the protocol. It's, you know, pretty easy and straightforward. The second strategy is the canonical paymaster strategy, which is, you know, still supporting self-relaying, but we also support bundlers who use a canonical paymaster. And the reason we can use this canonical paymaster in this instance is because we canonicalize a paymaster that has certain properties which the protocol can reason about.

And that we know that it's safe to allow many transactions to depend on for validity. And, you know, without, like, explaining, like, in detail, essentially what we need the canonical Paymaster to do is to once it receives Ether, once it has balance, that balance doesn't leave the contract unless it's paying for the execution of another contract. Or, sorry, another user. And so, like, with some of these building blocks, we can construct a pretty safe transaction pool from that. So that's our preferred approach right now. If we were to go the full route and do totally permissionless deployment of any kind of Paymaster logic, then we would need to do the full ERC7562. And, you know, that is pretty well specced out, but it does bring into…

the protocol, like, you know, staking for paymasters, and a reputation system for paymasters. And so, we felt that while there are nice benefits of it, because you have high flexibility. The complexity is just too high for us to recommend going for this approach from the beginning. So that's… What we're thinking on the transaction pool. Happy to answer any questions if they exist right now. Also happy to have a breakout room sometime this week if people would rather have, like, a longer-form venue to discuss this stuff before we jump to ACDE on Thursday. Just, like, let me know what people are feeling.

Mario Vega: Thank you. Yeah, I agree that a breakout… even for personally, myself, I would love to join and discuss this, and… Okay.

Lightclient (Matt Garnett): I'll try and schedule it for Wednesday. That's good with people.

Mario Vega: Awesome. Yeah, thank you so much. Yeah, thanks for sharing this. This was the last topic in the end of today. Is there any other topics that we have to discuss today anyone wants to bring up before we close the call today? Alright, if not, I think it's, time to close it. Thanks, everyone, for joining, and yeah, have a nice week.

Potuz: Hi, everyone!

Jochem Brouwer: Thank you, bye.

Marius van der Wijden: What do I think you…

Łukasz Rozmej: Bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.