All Core Devs - Testing #49

Transcript

Mario Vega: Hello, hello?

Pooja Ranjan: Hello.

Mario Vega: Let me give, a couple more minutes before we start to give… People a chance to… to join.

I think we can start. Yep. Awesome, thank you so much, Priya. Yeah, welcome everybody, this is ACDT number 49, today's August 18th. 2025, yeah, let's just get right into it. So the first item on the agenda is, Osaka.net. As I understand now, Fusaka.net 4 is down, what are the other updates, Barnos, if you… if you want to share?

Barnabas Busa: Right, so there are a few, Data 3 is still online, and we plan to keep it around for the foreseeable future. We are slowly rolling out the images of the trend branches of different client teams. I think most of the ERs already, have everything in their master or main. And, we are waiting for the CL teams also to, start merging everything into their, developer branches. And then once, that's done, then ideally we're gonna be, running .NET 5 as well, hopefully with, no more huge, changes. No more spec changes needed. No more spec, clarifications are needed. So we are aiming to launch that, maybe next week, Tuesday. Ideally, … Most client teams will have something by them in their master branches.

Mario Vega: Sounds great. Any oppositions on the timing? … I don't think there's major updates that need to be done by clients, if I understand correctly. But is there anything that needs to be… Done other than merging into main. From clients? Any concerns, basically? One message, DevNet 5 should also be pretty weak, similar to DevNet 4. Okay. Same BPO schedules, I would assume? Or it's gonna be different.

Barnabas Busa: We're gonna try to do the, nicer numbers with, 2 to 3, 2 to 3, so target, 2-2, to max value 3. to have, … We have a better calculation for the, … what's the value called?

Mario Vega: The fraction?

Barnabas Busa: construction area.

Mario Vega: Yeah, alright. Cool. Alright. Good. This is looking good. Okay, great. … Yeah. Alright, on the testing side, I can give an update. The next item is basically VPO site tests. We are almost ready. we have tagged a new release of East that we can use to test, the clients. So the approach is gonna be that we have now, tests with the exact same, scenario that happened on the W4 fork. and other similar test cases and scenarios for that. So, we have tagged the version 1 for Devon 5, and it's currently building, and as soon as it's done. We're gonna, make the release, and then update Hive. With this, we're gonna run the tests against all clients, and just verify that the BPO issue that was present in demo 4 is already fixed by everyone.

… Yeah, that's basically the update on that. We'll keep you updated, and we'll try to update Hive as soon as possible to get the information out. Alright, any comments on this, or either DevNets or static tests? From any… anyone? Yeah, right, I guess it's just waiting on… until Demet 5 for next week. Alright, good. … Moving on, on gas limit testing updates, there was a point brought up by Ben, that we should discuss 60 million gas pre-Fusaka. Ben, do you want to, expand this?

Ben Adams: Yeah, so there's been a lot of work going on to improve the worst-case scenarios, … But… will be fixed. We, post Osaka through repricing. But… sort of… I'm trying to gauge people's thoughts on going to 60 million, as was originally planned, briefly, so… Anybody has any thoughts?

Justin Florentine: Hey Ben, this is, Justin from Bayesu Team. I don't have all the details, happy to engage offline with them. We had some concerns with EC Recover, between 45 and 60 million gas.

Kamil Chodoła: Do you mean some particular scenario for EC Recover? Because from opcode testing, it was quite fast… fast enough to consider 60.

Justin Florentine: I don't wanna… I don't wanna derail this. I know that there's a separate conversation happening on Telegram. I just wanted to raise the… issue that Bayesu has some hesitancy about going to 60 million gas.

Kamil Chodoła: So, from my perspective only, if I can chime in now, is that there are just a few things which we still need to consider. It's like, Modex performance before Fusaka repricings, is that GEF still is working on improvements. There was one attempt, … Which was showing quite significant improvements, but was causing some regressions in other places, so there still need to be some work being done here. And yeah, I guess we'll discuss that a little bit this week, to see what else can be done, and how we can help them testing that out. Other thing is that we integrated EST tests and benchmark tests. On version 3 of those benchmarks, we noticed quite a lot of slow scenarios for Bezu.

Which already we are in touch with by the team here, so… it was more specifically about, Mot Aris. scenarios, but yeah, this needs to be analyzed. Maybe this is some other inputs in tests, which we maybe should analyze. But anyways, this is done, and also there is a new version of benchmarks, which we integrated, but it's having one quite significant regression, which is causing tests to basically kill the nodes. So… yeah, but this is not, like, it is slow on some particular example, on some small gas limits, but by mistake, EST tests are enabling some very, very huge blocks in giga gas. So yeah, so this might actually crash. And yeah, last thing, which would be stateful testing.

Which I'm working on exactly right now. is to enable a way on how to write and execute stateful tests on top of mainnet and bloatnet, and reproduce, for example, Xen scenarios or other heavy state access scenarios. And see how impactful those will be, and, like, at least open this branch of scenarios for reproducible tension testing. But yeah, hopefully this week I will have something to share with Carlos Perez to work on new scenarios out there. But yeah, most specifically, the most important issues right now with 60, which I see, is ModeX with GEF. And, some new scenarios found for Bezos.

Mario Vega: Excellent. So, a question from Tim on the chat is that, given EC recovery prices aren't going to be changing in Fusaka, is this a blocker for you post-fork? It sounds from Ben that it isn't, but, … Maybe we'll want to dig a little bit deeper onto that? Repricing is your programming?

Tim Beiko: Yeah, that question was for… for Bay2, yeah, not for, Nethermine, like, if, … Like, yeah, … Is there a… plan. Like, basically, like, yeah, if… Is there something that, like, at the client level, you think you can do to, to, improve this, or, … Do we, like, actually need a repricing to… To, fix these issues.

Justin Florentine: Right, so I'm not running point on the gas refricing issue at the moment, so I can't really speak to that, but I do know that we have multiple opcodes, as well as the modics pre-compile, that we worry about transitioning from 45 million gas to 60 million gas. That is, … a performance problem for us that, you know, we're currently working on it, and we're trying to optimize it, but, you know, it's a known issue.

Mario Vega: Yeah. Yeah, Camille?

Kamil Chodoła: So yeah, so looking at the data from EC, there is a EC ad scenarios which are particularly slow for Bezel. … Yeah, looking at the other clients, those scenarios are way much higher, and the slowest one for best-looking right now is, like, 23 megabits per second. So, yeah, up for discussion, but for 60, it will be slower, but it won't reach, like, even 3 seconds blocks. What would be good, yeah, as Marius is mentioning, to get, like, a… There's a team statement to get exactly what… maybe there is something which we are missing here, and we should test more deeply.

Mario Vega: I think, Camille, if you can share the results that you're looking at with the best of team and with everyone, I think that that would be very helpful.

Kamil Chodoła: Yeah, I will bump it up on our Discord channel with Caslimid testing.

Mario Vega: Awesome. Alright, so it seems like we have to do more analysis based on the results. Is there any other concerns from other client teams, regarding gas limits? Do we need more benchmarking? Do we need more information, or what's the current stance from everybody else? I see Rev on the call, … Does anybody have any opinion? Alright, Red says no concern. Anybody from Ericon? No concern, okay. And Marius mentions that we need more benchmarking with regards to state, and… Yep.

Maurice, do you want to expand on that, or…? On SnapSync?

Marius van der Wijden: After yours? Yeah, so… I don't know that much about it, but, apparently we're running, periodical snap syncs on these networks, and for Perth DevNet 2. All clients failed. To SnapSync within 70 hours, and, that is with it. Network that is twice mainnet size, so it's not that much bigger. So that is something that all clients should be aware of, and should look into what's happening on their client, and I can send some… more… Info in the chat.

Mario Vega: So through… Respond to answers… con… comment, we… I think this is a work in progress. So right now, we're trying to unify. Most of the tests into execution spec tests. And we're trying to make releases to make consistent testing across all the clients, but the framework is not perfect yet. mainly we don't have stateful tests in East, because it's hard for them to produce. So we're still working on a way to have this unified. But I think at the end game, or the final… … the final, like, benchmark testing that we have is all unified in East, and we can, like, take the same measurements per team… on each client, I'm sorry, on all clients, using the same tests, yeah.

Ansgar Dietrichs: Yeah, that sounds good, and I wasn't trying to imply that we should already be there, right? Like, I think that's… that is a big project, so I was just curious, kind of, what the current state is. I do think a nice goal eventually would be to basically get to a place where these are reliable enough that they actually give us confident visibility into blockers and bottlenecks, right? Because I do think right now we're still in this position where sometimes it's very hard to say, okay. this looks good on all clients, but then maybe the way these benchmarks were done didn't account for all the client-specific problems, and so then, out of quote-unquote nowhere, at least from the kind of top-level view, new problems emerge, and so I think really trying to get to

the highest visibility possible, I think would be really, really valuable.

Mario Vega: Yeah, completely agree. So, and just to highlight, So the latest version that Camille mentions is… it added a lot of tests from Eastside. So I think that's one of the reasons why we spotted new things. But I think, the longer that we go on East testing, the less new tests that we're gonna see, and it's gonna be more stable. So yeah, just to summarize, we have many new tests now, and I think that's the reason why we catch some things that we didn't catch before, in my opinion. Yeah, Camille?

Kamil Chodoła: Yeah, and just exactly to that, the tests which we run right now, not yet on the desired machines, but on the side, on weaker machines that actually we use, EC Recover is, from EST test suit, is performing at 60 megabits per second for all of the clients. At least 60 megabz. So, not sure why I need to understand a little bit more what exactly is the thing about EC Recover. Here.

Mario Vega: Also, if we can take a look at your… your specific tests that you are running for easy recover on Bisu, I think it will be very good. that we can import them into East to have them in this benchmark test set, so we can run them against all clients. So maybe there's a blind spot in our tests, specifically for East Recovery that we're not seeing. … Yep.

Justin Florentine: So I don't want to, bias this too much towards EC Recover. I think it's… it is an issue. I, again, I don't have the details on that. I think it's the, ModiXP stuff, though, that is more of a concern. So we have specific, just to elaborate here, we have 9 opcodes and the mod EXP precompile that are below 20 million gas, 20 mega gas. So that's… that's… more the area of focus that we need to address if we want to consider raising gas limit before Fusaka.

Mario Vega: Of course, yeah. In any way, I think if you can share the benchmark tests, just so we can also write a similar test for East, I think that would be great, just to make sure that we're not missing anything. Yep. Who can we reach out from Besu's side? sorry, who can we reach from Beso's side to… that has the details on this specific.

Justin Florentine: So, Mezian Hamlot, who's currently out for August, is our performance engineer, but Simon Dudley has been doing a lot of this testing as well. He's in the Australian time zone, though, so he doesn't usually make these calls.

Mario Vega: Excellent. We try to reach him async, just to see if we can implement the same test in East, yeah. Camille, yeah?

Kamil Chodoła: So, yeah, just last question, by 20 megas, you mean 20MGAS per second performance?

Justin Florentine: Yes.

Kamil Chodoła: Okay, yeah, so here is, yeah, all for the decision if 20 megah is something which we want to treat right now as a security concern, or we should fit into, like, 4-second execution time. So, based on that, the decision should be made. Like, best would be, of course, to be below, even 2. But, yeah. ModeXP, as I can see for now, for sure, there is some work needs to be done for GEF. Where we have 15, or something like that, while Bezu's slowest scenario, which we have right now for Bezu on our end, is 19. And whenever I was thinking about any potential… increase. It would be that it would be good to be… But worst scenario is not above 3 seconds.

And surely not above, or close to 4. With GAF, it is for exactly. With Bezos, it will be close to free.

Mario Vega: Alright, thank you. … And yeah, Luis from East, team, he shared a… the links to the exact test that we're running on Eastside. So yeah, just for reference. If anybody wants to take a look. All right, cool. Anything else on benchmarking, or we can move on to the next, … Next topic.

Kamil Chodoła: One thing only for me, that whenever any client team or anyone here listening is building any new scenarios or any new tools, please maybe consider at first. to work on EST benchmarking approach. So, this way, we'll be able to run it on our existing infrastructure and verify it against all of the clients. Even recently, if Efrex from Lambda team, Lambda Class team is joined, so they are also being verified right now. So it would be really cool to follow that approach moving forward, rather than creating separate tools, because it will add additional level of complexity to integrate everything.

Mario Vega: Makes sense. Perfect. Yeah, thank you so much. Cool. Alright, before jumping into… The next topic is Sunnyside Lives, but I think there's a quick topic that Barnabas just brought up about the PR that has to be merged. When I shared the link? Here, Barnabas, anything that you want to add to this, this PR? Any comments?

Barnabas Busa: Yeah, so my general comments would be that ideally, we want to be able to locally test blood numbers over 32 as well, and this limitation, is, present in the minimal preset only, so this is a CL-only change. I don't even want to pump this to the team that uses mainnet, so we can test a larger number of, Blob's throughput as well. Please take a look and, … Approve it if you want, and argue if you don't. And ideally, they should make it into the full releases. We're gonna… if everybody agrees, then we can include it in Alpha 5 release with Justin.

Justin Florentine: Sorry, Barnabas, was that for me?

Barnabas Busa: This is for CLW.

Mario Vega: CL specs, just in… just in trouble.

Alex Stokes: Is there any reason to not do this, Barnabas?

Barnabas Busa: Not that I can see.

Alex Stokes: Okay. Yeah, it seems fine, just from a first pass. For me.

Barnabas Busa: The only reason not to do it is, it's so late. But that's about it.

Mario Vega: Alright. Yeah, please comment and approve if we are moving forward, or was just, … leave the comments. Alright, thank you. The next topic is Sunnyside Labs. … Yeah, I just wanted to put this point here, if there's anything to share, but I'm not sure if anyone from Sunnyside has joined the call. Or if there is any updates.

Minhyuk Kim: Oh, yeah, I, ….

Mario Vega: Thank you.

Minhyuk Kim: I'm Mina from SciSight Labs. We don't have anything much new since last week, except for this report that we put out on Discord last week, so please check it out if you haven't, and… Thank you.

Mario Vega: Excellent. Thanks. All right. Yeah, and, the last topic… … is the sephe discussion. I think Mikhail raised this on… ACDE last week. But I'm really not sure if there's any updates on that, or if Mikhail can comment on this topic. So, I just added it to have a follow-up discussion today, but if there's not news on this, or if there's no one here that can comment on this. … Let me share the link to the comment in the chat. this discussion I'm referring to. Basically, the safe head, … on the… on the EL clients, there was a discussion at the last ACDE, about changing the semantics or the behavior of this endpoint.

… Is there anybody here that has more, context on this, or any updates on this topic? Seems like not. Yeah, we can follow up this topic on the ACDE, I assume. Alright, that was the last topic in the agenda for today. Anything else that anybody wants to bring up, or any concerns? About any other topic?

Going once… Going twice.

Marius van der Wijden: I have something really quick. I just looked through the benchmarks, and I noticed that Keth and Aragon are at 30… 30, 33 million gas for the point evaluation precompile. So… I think it's probably… would be nice for those teams, us and Aragon, to look into this. Because those are the only cases Significantly below 60 megabytes per second, for those clients. So, point evaluation precompile.

Kamil Chodoła: Isn't it that it is more or less the same performance for all of the clients, because of how it works right now? And the only way to improve that would be to just reprice it.

Marius van der Wijden: Is it? I don't know, I didn't look at the other clients.

Kamil Chodoła: just opening that right now, but as far as I recall, this performs almost identically for all of the clients.

Marius van der Wijden: That's probably because they use the KZG library for it, right?

Marcin Sobczak: Yeah, I recall the sign. It was about 50 to 52 megabits per second for all the clients.

Marius van der Wijden: But I… I don't see it in the Netherland chart, so… I don't know.

Kamil Chodoła: Nethermind is there, but … for some reason, we have warming enabled for that scenario. Yeah, and this throws Netherland quite high on the chart.

Marius van der Wijden: Okay.

Kamil Chodoła: But without it, it will be also on the same level, as I recall from the past results. Just some recent merge, … Make it, again, fast.

Marius van der Wijden: Yeah, I think that this is probably a case for optimizing CKZG. And or repricing the party valuation pre-compan.

Mario Vega: Do we need.

Kamil Chodoła: I mean,

Mario Vega: Oh, sorry, I just want to mention if we are missing any cases on this, on the east.

Kamil Chodoła: The scenario is performing exactly the same.

Mario Vega: So far.

Kamil Chodoła: identical, ours and East, exactly the same. Both with same performance, … Just that, at least with my assumption that it is more or less around 32, 33 megacity per second. It does not much violate the rule of 3-4 second blocks. It's on 3 seconds, actually. So, yet to be, maybe, considered, how much it affects the 100 goal, or something. We can bring it to the discussion. To consider it moving forward. But definitely, if we want to go higher, we need to consider either some optimizations in the library. As we all use the same one? Or repricing for Glamsterdam?

Barnabas Busa: One more note, could every, EL try to think to proof that too, and see locally what kind of problems you're facing? Because none of them… none of the clients being able to sync on our sync test seems to be… There might be some other underlying issue.

Kamil Chodoła: We're looking at mine for us.

Mario Vega: Thank you. Alright. Anything else? It seems like we have to take a look at these, benchmark results, unify, and … take a follow-up on next week's ACDT, if possible. … Anyway, okay, any other topics? Anybody wants to bring up? Before we… we wrap up? Okay, if not, I think that's it for today. Thanks, everyone, for joining. And see you next, well, next Thursday, on ECDC. Thank you, everyone.

Barnabas Busa: Curt.

Spencer Taylor-Brown: Thank you, myself.

Kamil Chodoła: Right?

Ansgar Dietrichs: Everyone.

Alex Stokes: Thank you.

Marius van der Wijden: Let's put it.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.