All Core Devs - Testing #96

Transcript

Justin Traglia: Good morning, everyone. Just a couple minutes, then we'll get started. PK, will you be able to give the DevNet updates today?

pk910: Yep, can do.

Justin Traglia: Okay, I'll ask you on that. Couple minutes. All right, let's jump in. Good morning… good morning, everyone. Welcome to ACDT 96. Today, September 14th, 2026. Let's get started with DevNet updates. PK, are you here, and could you maybe provide these for us?

pk910: Yep, I'm here. So, we basically have two definites to talk about. The first is Definite A. We left Jochem's attack running over the whole week, till this morning. And, yeah, we didn't really, see any further degradation, than we already have. Mainly Besu and Erigon were still impacted, and, were not producing blocks. Yeah, we basically stopped the attack this morning, so we gave clients some time to recover, and it already looks good. Participation, rised a bit again. And we throwed Erigon to the latest image, and wiped, and let Bezo resync. So they are basically on the way to get back to head to… Yeah, one small operational issue on Nimbus nodes, because we've restarted them without boot nodes.

So they are basically, syncing up now, too. Should be ready soonish, too. that's basically it for Definite Aid. The other definite we want to talk about is, DEFNet 11. We started the Defnet last Wednesday. Running the unstable main and master branches from all clients. And, Glamsterdam activated today. It went through fluently. All clients, are ahead, we had no participation drop or anything. And… yeah. And the network was basically for Lido and Optimism. They've both deployed the stack. And also gave an early feedback that everything went well on their side. So yeah, I think that's it from the dashboard side.

Justin Traglia: Awesome, thank you. Is there anything… Clients would like to mention? I'll give you an opportunity to speak if you want to.

Saulius Grigaitis: Are we going to have these, like, spamings and other malicious actors in this network?

pk910: Not on DevNet 11. We wanted to give, especially Layer 2s and Optimism.

Saulius Grigaitis: So we should keep, trying to… to get the net 8,

pk910: Where they can, test the flu and path nicely. Again, But yeah, we should try to get, definite 8, up again, obviously. Because that's technically the same, version, there's no difference. spec version or anything.

Justin Traglia: CFI. Which network should we, Perform tests, which might break everything on.

pk910: Definite aid.

Justin Traglia: Definitely, okay. Okay, any other questions or comments from clients regarding DevNets?

pk910: I might have one more point, but we might delay it till after the, testnet, launched that agreement. Because, we wanted to do Sepoliaa and manage ShadowFox at some point. Just a question, what do people want to see? And… Yeah. If you should do them.

Justin Traglia: Okay, and to be clear, do you want to discuss that now, or after the next discussion item? Okay, moving on to the next point, I'd like to confirm with clients, that we're still on track for the Sepolia upgrade on October 6th. Are there any clients that are against this, or is this still fine for everyone? And an explicit confirmation would be appreciated as well. Thumbs up from Nethermind. Thumbs up from Nico, from… Lodestar Lighthouse. Breath.

Eric on. Guess. Tycoon. Granty. Perfect.

Enrico Del Fante: My only concern would be to… See how this progressive list size The discussion will go through the CL breakout later. That… maybe we could have better… Signal after that.

Justin Traglia: Okay. Yeah, I agree. I mean, that is… a concern of mine as well. But I think most clients are handling it fine, like… Micah, what you're doing right now. I don't care. like a converting or changing Tech here to perform the LUT checks during deserialization.

Enrico Del Fante: Haven't retested, but I have another approach to that PR, which then has some additional implications, so… Depends where we wanna go there.

Justin Traglia: Saulius.

Saulius Grigaitis: Yeah, my… my suggestion would be then to try to get DevNet 8, to healthy state as soon as possible, because, I mean, it kind of feels that this data is… is… somewhat abandoned. And, you know, there are multiple nodes, multiple clients that are struggling, and I don't see that much of progress here. maybe someone could coordinate, like, ping the clients that are struggling and figure out which side of… is it EL or CL of those nodes are struggling, because you can see that a lot of blocks are missing. And so on. So… Yeah, somehow this needs to get… To a much better state than it is now.

pk910: Yep, I'm working on that. I've basically already notified some clients this morning that are the majority of struggling nodes. I think they are on a good way. Bezo is completely wiped and is re-syncing now, after several incidents, and Erigon was basically on an older head and is syncing up now, too. So… Yeah, and Nimbus is also thinking, so there is some progress in all these clients.

Saulius Grigaitis: Yeah, and at least for us, it was a bit… and I think for other clients also, it was a bit tricky to… to progress, because often we just see that, most likely the cause is on, on the EL side, and the EL side, was not updating for a while, you know, so it's, so we kind of see that our nodes are not proposing, but, But in most of the cases, it's an issue on the EL side. So, in such cases, I think it would be really great if the coordination for the… for this particular case, then, someone pings EL guys and, you know, and keeps pushing them. Because this is… it would be much, much easier for the CL people to actually see the real problems, instead of just thinking that EL is, you know.

Doing some… some bad stuff. So yeah, maybe this would help in future Nimbus.

pk910: Yep, that's a fair point. Just one thing to add is that last week we were running heavy attack transactions, so many ELs were basically struggling a bit. But yeah, good points.

Saulius Grigaitis: Okay, thanks.

Justin Traglia: From the chat, there's a question about why… why does it take EL so long to sync? Do we know the answer to that?

pk910: I think lots of ELs are currently configured to do a full sync, so they basically run through the wall trans, attack transactions from the last two weeks, too. We might change some of these to do snap syncing to speed that up a bit.

Justin Traglia: I see. Okay. Any other comments on this? I prefer moving on. Okay, the next agenda… agenda item is, from me. In the Consensus specs repo, I've made a proposal to restrict which EIPs can add tests to the repo. I'm proposing that we limit it to CFI and SFI EIPs. This is just to make, It easier for maintainers, because anyone can propose a, an EIP and modifying and adding tests to the specs is a bit more invasive than just adding a self-contained. spec directory, I'd like to discuss this with clients and the community, and

Are any… is anyone for or against this, or… And you're reason not to do this. Curious to hear what you all think. And we can also discuss this offline on Discord, so if you have any thoughts, I made a thread on Discord. Alright. Yeah, PK, we can jump back to that. How do clients feel about Shadow Forks?

pk910: Just to recap, we briefly planned to do a Sepoliaa shadow fork and Shadow Fork. Yeah, only if clients feel ready for it, and… Because it's a lot of work involved with launching them.

Justin Traglia: What would you… would you need anything from clients for this?

pk910: Basically, it's just an okay that there are, unstable, master, whatever branch are ready for testing it. Because we likely don't want to repeat it. So immediately it goes right. I think if no one is against, then I would, suggest to start up the Sepoliaas at Shadow Fork sometime this week.

Justin Traglia: Pawan from Lighthouse is suggesting that we would wait one more week. Given that… you said, given we need to handle a few bugs on Lighthouse. Would that be fine?

Pawan Dhananjay: Yeah, just, like, just bringing up, like, some of the issues that we had, the progressive containers thing, like, we still haven't completely audited our dependency around that, so just wanted to make sure we have, like, a Fixed version before doing that, maybe?

Justin Traglia: I think that sounds reasonable.

pk910: Yep, then let's talk about on the next testing call again, if everyone's ready. I think that's a good plan.

Justin Traglia: There are some more comments regarding the… defining which EIPC NAT test thing. Jochem, would you like to talk about this? Or I can read your comment?

Jochem Brouwer: Yeah, I mean, I can just explain. So, I think the idea that we only add tests for EIPs which are in CFI in the trunk range of the Consensus-ex repository itself is fine, but this should not give, like, the signal that people should not work on tests. So, if the EIP moves from PFI to CFI, then ideally, we already have the tests in some PR, or in some branch, or whatever that we can directly merge them into the junk branch. Because it will not be process efficient if people only start working on tests for this EIP once it arrives in CFI.

Justin Traglia: Yep, 100%. That's also kind of what I had in mind as well. EIP champions would work on tests outside of The Consensus spec repo. In their own fork, or whatever. And they can also generate tests. And provide it to clients from… from there. Thank you for that. Any other comments on that, or any other comments on anything else before we jump into the breakout calls? June?

Jihoon Song: Yeah, I just…

Justin Traglia: Oh, you're muted now.

Jihoon Song: Oh, sorry, fat finger. Yeah, I just wanted to add, to… which EIPs can add… I mean, may add tests. The intention behind this proposal is because usually Tests, touch multiple files, and… Attempt to modify existing testing infrastructure, so… As Justin said, it may make maintaining tests for existing hard forks quite difficult. I mean, if you… if you have to remove the tests out, it will go to be messy, and I just wanted to say that this is not to discourage people, from working their tests.

Justin Traglia: Yep, totally agreed. Okay. Anything else? Otherwise, we'll jump into the… breakout calls. Alright, bye everyone. I'm going to go to the CL breakout.

iPhone: See you.

Maria Silva: Do we have someone to leave this session, or not?

Spencer Taylor-Brown: I actually don't know.

Maria Silva: That's perfect.

Alexey Osipov: Yeah, hello.

Maria Silva: So I don't mind doing it, so I think… let me just open the agenda again. So I think we have three topics. The first one is, let's agree on blob elided length versus payload length. Can we get some, context here?

Alexey Osipov: So… Hh slash 72, standard for, spar… or so-called sparse blob pools. Well… It, introduced new, format for, Block Exchange, which, Also in place, empty blobs in transactions, pulled from a period to another, and by… Compared to another. So, when we announce the length of, Transaction, we… in the other mind, we use payload form according to old specification, but, then we pass, network of… blobs in network form, with, emitted blobs, and, seems like,

It might be good to update, and agree on the spec, that, proper lens, should be used. I mean, network, form, one, updated, According to new… Standard, I mean, ACH72. And, if we, could agree on that, and match this, and, input, further it, In the minutes. Yeah, if no one,

Maria Silva: Okay, do we have any opinions from the other clients? So, just to clarify, the ask is essentially, there was an issue found, and we kind of need to figure out what is the right way to fix it. I don't know, do clients have any time to look into this or not? Can we make a decision in this call, or should we defer it?

Okay, so should I…

Alexey Osipov: already implemented like that, by other clients, except NetherMind, so… If you, guys have, If you want to still use payload form links, let us know. Otherwise, we just… like, change it, have it, in other clients, and we can… yeah. And we just, will need to merge this, PR mentioned in the… Gentlemen.

Maria Silva: Okay, so just, just to make sure I understand. So you are saying that all the other clients agree on one way, except never mind, they had implemented it in a different way?

Alexey Osipov: Yeah, and we agreed to change, I guess it makes sense, and we just implemented it according to the specification, but we… But a more realistic way of exchanging this information is by using network… updated network form lens instead of payload form. Lance.

Maria Silva: Okay, so you are saying that you think you should… So Netherlands should follow the implementation from all the other clients.

Alexey Osipov: Yep, and we need to update and, I mean, merge this pair. Okay. Let's…

Maria Silva: In that case, I… I… Yeah, in that case, I would expect that's mostly a decision for you guys, like, I'm assuming that the clients implemented it in a way that they feel is the right way, or… I don't know, do you guys want to take some time to look into these and… and do your implementations, or should we just say, okay, never mind, follows what the other people are doing?

Alexey Osipov: Yeah, maybe it's a call for, that PR review, so it… the implementation, like, numbers are, exactly the same on all clients.

Maria Silva: Okay, so… I'm… I'm guessing then we can do this async, so we urge people to… check the PR, which is this one, and then please, comment and, Yeah, let's try to… so what would be the next step here? Is it just clients looking to it? And report back.

Alexey Osipov: Can you please review the PR?

Maria Silva: Okay. Cool. Anything else on this PR, or should we move to the next one? Okay. I think… It's the same, person that raised it, so it's, this PR… do you also want to give some… some context on this one?

Alexey Osipov: Yeah, regarding, collisions, Well, we have, a kind of, Testing issue, we need to agree. How we update, existing tests that, that are now failing, Like, a lot of them, in high runs, related to collisions. And, it's… so, I'm just asking to maybe agree that we roll back to the old EIP that implies that if we have such collision, we just rewrite storage. And we update the tests, and like, align the client's, In such ways that they just rewrite the storage. But I guess it's a small issue, it's a testing issue. But another is, when we…

Try to verify client's code. This behavior, Like, it's, of course, theoretical, but when we rephrase the code, it's kind of not fully defined. It means that we have block access list now, and I'm not sure how it should behave, and if, You guys have any opinion on how to handle this? It would be a… Great to share, for now, I would suggest to, like, include in BAL storage slot resets to zero, and, updates for such slots. In case of collision, but maybe it's too much work, and, people have, better ideas.

To handle BAL, in such theoretical case.

Milen Filatov: Yeah, hi, so I… I opened one of the PRs to… basically update the legacy tests to their version before EIP7610 got introduced, and I think we should go with that, because Those tests were pre-existing, before… EIP7610 got introduced, and… it basically covers the yellow paper plus EIP684, so I check… so it basically does the… It basically, like, nullifies the storage slots, if there ever is a collision, which is kind of pre-existing behavior before EIP 7610. And, actually restoring these tests quota regression, which we had in Erigon, so, I think there is some value to keep them around.

We just need to… like, my suggestion would be to just review… review the changes and go back to how the tests were. Before EIP7610. But, yeah, depends on what other clients think as well.

Jochem Brouwer: Yeah, so…

Alexey Osipov: Good to me. What about, BAL? Do you have any, idea how to, Who handles this part.

Milen Filatov: Yeah, so that thing is new, and, goes back to that conversation that we had in the chat, and still… About keeping an undefined behavior about BAL, but, yeah, it's, Because it's only those 28 accounts on mainnet that can trigger this. And either we keep it… like, my thoughts are that either we keep it undefined behavior, that's option one. Option two is, we decide to explicitly say if this ever happens, we don't include the BAWs, just for simplicity. Or option 3 is maybe we do EIP8253, and kind of get rid of this issue. But yeah, these are kind of my thoughts in my head, because I was looking into this last week, But, yeah, again, it's up to what majority of the clients prefer here. I don't really have strong opinions.

On this one.

Jochem Brouwer: Okay, I want to quickly add something here, because, it indeed depends on what behavior we are going to define, or what we explicitly are not going to define, because the, so we have to realize that in the book access list, there's also this opportunity To implicitly use the block access list as a state div. So, you don't have to execute transactions anymore, you can just apply the stative of the block access list. The problem is that if we run into this theoretical situation that we start to create a contract on an account where the storage is not empty. Then we technically, in order to produce the right state div, we have to produce a block access list.

Which clears all these accounts, and this itself does not sound super hard, but actually, this is a problem, because we need the pre-images for these storage lots. So, if we would iterate over the Marco tree, sorry, over the storage tree of these accounts, then, if we have, just… if we're just storing the, The, the, the hashed keys into the database. then it's of course not possible to retrieve the pre-images. And for clients which actually do store these PM images, this is somewhat easier, but, well, I guess you can already see that this is a somewhat complex problem. And the other thing is, is the question if you want to test this, because if we want to test this, then we are also going to explicitly define its behavior. So, this behavior, the… I think also one thing what

contributed to the conclusion about EIP 7610 is that this behavior has somewhat flip-flopped in the past between either reusing the storage of a contract, or explicitly marking it as an empty storage. So… My personal feeling is that we should indeed, and this might, of course, sound somewhat dangerous and somewhat weird to suggest it, but to explicitly not test this behavior, because it is indeed these 28 accounts on mainnet. And we already have EIP8253… well, PFI for AICOTAR. I wanted to CFI it in the last ACDs. Sadly, I think that this did not happen, but that is the plan. And I think also maybe Spencer, or Felipe, or someone from your specs team. But I think if you want to start to test this, this will make

well, I guess the testing part's incredibly complex for a situation like practically cannot happen on the mainnet.

Alexey Osipov: Yep, I'm mentioning this because undefined behavior leads to, some… troubles, on, the side of, verification and, bug hunting. So, client, may implement this part, differently, and, it, kind of, kills, like, entropy, and, like… Less undefined behaviors is better. If it could be done in the simplest way, agreed by everyone, and covered by a single test, it would be, I guess, enough. If you want to keep it and define it, I will not, argue a lot. Still seems like, less, not define it, Cases is, better.

Spencer Taylor-Brown: I think, my plan, at least, was to merge… milen PR that basically updates the legacy tests instead of removing them, and just using that as the, kind of, stepping stone before we get to 8253 in Hagata. Some could argue it's redundant, but if, you guys feel there's value in that, and yeah, I'll definitely, definitely do that. And I guess the… redundancy aspect comes from 8253 in Hegota, and then also At least on our side, we're planning to archive legacy tests, hopefully by the end of the year. But we still have… to do, I guess, a lot more verification, and so we're really confident before we do that. But if… yeah, if that works for everyone, I would be happy to do that.

And then, yeah, aligning to… make the undefined behavior, kind of what milen says here, to align with the yellow paper plus EIP684. M.

Milen Filatov: Yeah, the only thing to bear in mind is that the… reverted legacy tests, they wouldn't exercise this second case about the BAL undefined behavior, but if we all are happy with leaving that undefined, then we don't need to do anything about it. Just because those legacy tests obviously don't cover anything BOW-related.

Jochem Brouwer: Yeah, this is actually a good point, because if you start to fill these legacy test scenarios for Amsterdam. Then we also have to define its behavior,

Milen Filatov: Yeah, I don't know if, I think maybe Spencer would know this better, but, I don't know if, EOS actually has any logic for this, like, for the storage deletion, I think. So yeah, we… I don't know, do you have any… any… any thoughts on that, Spencer, or…

Spencer Taylor-Brown: At the moment, not… not right now, sorry. I think I'd need to look into… into the BAL aspect in a bit more detail. But in any case, why don't we merge your PR milen and legacy tests, and then maybe for… We can chat async, and we can, align on adding a case for about an ELs. Or we can try, yeah.

Milen Filatov: Okay, cool, sounds good. Because it also might turn out that all clients actually would do the same, like, they might actually have the same behavior in terms of BALels as well. But, like, I guess we haven't really checked, But in order to check, we might need an ELS test. But yeah, anyway, that sounds…

Alexey Osipov: Sounds good. You mean, that we just, pretend it's, it was empty storage, and, place, the, storage from scratch, and, log just this in, BAL, right? Oh.

Milen Filatov: It… I meant that, for example, I actually haven't checked what our behavior will be in terms of BAL, if we do end up having to delete. these storage slots. I'll need to check the code, but, it might… my point was that, it might turn out that all clients are consistent on this. If you see what I mean. I just haven't checked, for example, what would happen with the BAL,

Jochem Brouwer: Yeah, maybe to quickly also add on here, so the, Some confusion might also come from If we assume that the block access list is the state div of a block. So currently, that indeed is a state of off-block, except, if we have this kind of situation. Because then it depends, should a client insert these, the deletion of these storage slots, or not, because if we insert the deletion of these storage slots, then this indeed The, the state diff of this book. But if you don't include this, then this is also fine, but that means that we can, in that specific block, not use the block access disk for the state div. If that's, somewhat clear.

Alexey Osipov: Wait, deletion is where… Sorry, here, okay.

Karim Taam: I just wanted to add, we are talking to use the Block Access List for healing, so if we are missing some state modification in block access list, it can be an issue for the healing. We can have a, yeah, an invalid state at the end, so… On my side, I will prefer to have everything.

Jochem Brouwer: there's this one idea what I have, and this is, to… define it, but in an extremely ugly way. This would be to, like, to hard-code these situations. So, we already know these 28 accounts. We also know that if we were to create, like, this contract, then we have this non-pump, so in the execution part. Then we can hard code that, if such things happen, if such things happen, that we can hard code in clients how to block access to this work, because, again, the issue is that we need the pre-images for these storage slots, because we have to iterate over the storage tree. To figure out all the key spots we are going to delete.

And then to put that in block access list, and for some clients, this is not possible, because we don't have the P images by default. The problem with this is that if we start to define these tests, then, yeah, these are, like, very specific tests. I'm all for adding more tests, but this then also defines this behavior, I would say. So I'm not really sure about this thing yet. I hope my explanation is clear, by the way.

Maria Silva: Okay, so, what are the decisions here? Anything else? Okay, so… I see we have this idea of making a single feature that people can test and compare, and then close the topic. Anything else that… We need to do.

Jochem Brouwer: Yes, then I want to quickly ask, because we are going to create this fixture for this very specific situation. But then also from the comments from, Karim. then I would say that we should really move to a block access list, which also really generates this data, so that would mean that in this kind of situation, we also explicitly include these block access lists storage locations into, yeah, into the block access list, such that we can apply the block access list to get the right, storage route. Yeah, so I'm just asking, like, if someone thinks something else, that they should not include these deletions? that I would like to know, but I think this is the way forward, to actually include these deletions into the picture and the,

The scenario, yeah.

Maria Silva: Do we have any voices against? Okay, anything else that, we need to discuss on this topic, or can we move to the next one? Okay, so I think the next one is a clarification on, EIP8038. So this is the state access repricing Does anyone want to add any context here?

Alexey Osipov: Yeah, just, a small, comment that it's already like that in every client, I guess. It's just a bit outdated EIP text that, Like, assumes we, charge this, 100 wolgram access of… For former, beneficiary in case of self-distract, but we don't do so. If… I get it, goodly. And it's just a clarification indeed, that's it.

Maria Silva: Right, so, yeah, I read that, I think it makes sense, so, I can simply approve it, and we can… we can close it. Cool. Anything else we should discuss? Yeah, Johan, my understanding, this is only EIP change, not spec change.

Jochem Brouwer: Cool, maybe, Because I think this is not on the agenda, and then now I can't find it, but I know that Peville has opened up ER to limit the total gas for EIP 8037. I think it is good just to mention that at least here.

Maria Silva: Yeah, good point. So, I actually, I just merged it today, so the idea was there was no formal limit on the state gas, of the transaction, and it was kind of an implicit limit, where the limit is a block limit. But I think it's, like, adding, like, a big limit, that, But yeah, that's kind of… yeah, it doesn't change anything in the behavior, just makes it a bit cleaner, I guess. And so, yeah, I just merged it, And I would expect we need to update the specs, right, Spencer? But it shouldn't change. Any test behavior, or any… Client behavior.

Spencer Taylor-Brown: Yeah, it should be a small… Like, small spec edition, but yeah, it doesn't really change any behavior, that's all. And yeah, I think, we'll add a test for this, but I think, I guess it's also worth mentioning that we're gonna try and… Release our, like, test, official testnet release fairly soon. I wanna say… by next week, or at some point next week. So it will be in that release, if it is, yep.

Maria Silva: Anything else we should… Which of these cuts? Okay, if not, I guess we can… Finish a bit before time. Yeah, thank you all for joining, and yeah, see you soon.

Jochem Brouwer: Thank you. Bye-bye.

Spencer Taylor-Brown: Sure, bye.

Jochem Brouwer: And thanks, Maria, for leading this call. Thank you.

Maria Silva: Sure. Bye-bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.