All Core Devs - Testing #40

Transcript

ef berlin office: Okay, I guess we can start the call now, and people will slowly trickle in. I'd like to start off with interop updates. So we had an interrupt event. Last week a large amount of time was spent on scaling the gas limit as well as Fusaka. But we did have multiple other tasks, tracks with history. Expiry peer to peer changes, Cl. Hardening. Zk, avms L. 2 s. And quite a lot of other tracks, I think over the next week or 2 weeks, actually, sometime this week before Acd, we should have a summary of everything that happened last week. But for now I think we'd go into a bit more detail on 2 of the tracks with Fusaka being the 1st one.

We had all the specs and open Prs from last week merged into Belinterop devnet 2 this is a devnet that's gonna be up until next Monday or Tuesday, and the reason we use the bull interrupt name is such that so that we could make faster decisions without needing to go through the Acd process while we were in person. However, those changes will now go through the Acd process and will eventually end up on the Fussaka Devnet 2. Once they've been cleared by Acd. That being said, Fusaka Devnet, one is still live and will go down over the next day or 2 unless anyone specifically needs it. So if you do need it, then please speak up, if not, please, move all of your testing to Berlinterop. Devnet 2 and Fusaka Devnet 2 should come up sometime around next week.

Any questions regarding Fusaka in itself. Okay. The second track that we spent a lot of time on was gas limit testing. I think most of the time was spent on benchmarking clients as well as optimizing performance. I think the outcome was great. We should have a deeper write up. Come out later today. However, one of the things we soft agreed to on Friday was to raise the glass limit to 45 million. However, client teams would need to push in the fixes that they have in the performance branch as well as make releases. For this is this something that we can have clients agree to do over the next week? Could we maybe get someone from each client team to speak up on this

in the same room as Harry. So the no. Yeah, yes. maybe it's better this way. Yeah. So forget we need at least one fix to get into master until we feel like We don't feel feel super vulnerable. There's a bunch of things where we think like going above 45 is not possible right now without changes to without repricings. But I think that, like the biggest point, is like the modex stuff that we are already addressing in in Fusaka. So like, from my personal point of view, 45 would be would be okay. But I'm I'm not speaking for the team.

Ben Adams: Now that mine's okay. With 45 we would like to put out a new release. But I think we're comfortable with the the current release and signaling. I assume it won't be immediate, anyway. and we have a release similar.

ef berlin office: Thank you. Could we have maybe Roman? And then Milan from Aragon.

Roman: Yup or comfortable doing the 45 visits.

Milen Filatov: Yeah. Aragon is ready for 45 as well.

ef berlin office: Awesome. Thank you. Then. I guess we'll share some more info on the benchmarks. We did as well as what bottlenecks we found over the next week, and once the client teams are have all changed their defaults. We should the network signal to a higher gas limit. Awesome. Marcin. I know you've been working a bit more on benchmarks as well as yeah. Exploring gas limits a bit more. Do you wanna talk about it?

Marcin Sobczak: Yes, sure. So like, we know like for for the clients, the the worst case is the modex, and it will be repriced in Fussaka. So before Fussaka we have, like the we know what is our bottleneck and if when modex will be addressed, then probably the next bottleneck is is yet so it's it's performing good enough to go for 60 million. But if we aim to, if you want to aim in 100 million between Posaka and the next Harfolk. and then we probably need to reprice in in Fussaka, too. And right now I'm crafting more test cases for easy pairing and easy mall to check if it is performing good enough to to keep the pricing as it is, or we need to

to replace it, too. So far, I I didn't find the test case, which is performing like badly. So so so it doesn't look like there will be a need to to touch it in Fussaka. Oh, and yes. Do you want Barry to talk about some specific topic about benchmarking or.

ef berlin office: No, I think that I think that's a good update for now. So is, I think, for easy add, we should also have an eip up for that later this week. There is a proposal for what we want to reprice it to. And yeah, today and tomorrow, I guess Marcin and a few others would be figuring out. If we do actually need to reprice Ec. Mole or Ec. Pairing going forward. This should help us get to 60 million and further and prepare for post architis. The other topic that came up was log logs in themselves, either the pricing or changing the Dev. P. 2, p. Limit. I'm not sure if anyone has thoughts on that, or has time to look into it. I know that there's a proposal from Julio.

but I don't know if there's any more work being put into it.

Ben Adams: I mean, this sort of kicks in more around 85 million. So we'd still be fine at 60. There's something we potentially wanna well, probably definitely want to fix in some way for Usaka. But I I don't think we've come to a conclusion on what the right way is. Yeah.

ef berlin office: Is there some specific data point or something we should look at that would help us get to that conclusion, or is it just? We need to discuss this on Acd and come to a solution. There.

Ben Adams: Well, I I think we we need to discuss on Acd. Probably a helpful data point would be to actually emit an entire block of logs and then see if there are any performance implications, and then maybe that would help us decide whether we want to reprice or increase the peer to peer.

ef berlin office: Okay, got it. I think Pk has a reproducible test for this already. He was able to break snap sync locally, and I'm not sure if he looked at it from a performance perspective. I think he purely looked at it from does this break snap sync perspective? But we should have a test for this that we can reuse.

Ben Adams: Yeah, I I mean, I think there's 2. At the moment there's 2 obvious ways. We can go, and having some benchmarks, would probably inform which direction would be better.

ef berlin office: Okay, we can clean that up and get it up to people. I think the other topic was Estor with large tree depths. I think this is something we noticed on blocks with Zen contract transactions. It, again doesn't seem to be a major issue, at least for 45 million. However, as we move to 60 and forward, we'll definitely be an issue. I know. Last week Joachim was working on this a bit. Do you maybe want to go more into what the current status is. Do we have a reproducible test? Were clients able to trace what was going on on? Their clients.

Jochem Brouwer: Yes. So this is about the send contracts right?

ef berlin office: Exactly. Yeah.

Jochem Brouwer: Yes, yes, So what it looks like is that it's a. It creates a lot of small contracts. These contracts which are being created are actually proxy contracts, so very small ones which delegates call into something else. And what happens is that during this transaction it's a transaction with a gas limit of 28 million. and it creates 167 small contracts, and it also does a lot of storage. It stores a lot of values. There. I'm looking a little bit in. If there is something else, something else. Very fancy going on there. It doesn't really look like it. It really looks like it's a lot of create 2 or create fact, create small great contracts. and that there's a lot of storage being dumped in there.

and my hunch will be that the caching mechanism of clients that yeah, these, these key values, these are these are like a code. because you can't really predict before, and that that yeah, in what what addresses and what key values, we are going to read or write. And I think that might be one of the problems where we have seen like very slow blocks. There is a status, or I'm not sure if it was a status or a blockchain test. I think, a state test which does exactly that. It creates a lot of small contacts. and it dumps random storage of not random, but it creates a random storage fee there. and I think what we have seen in clients is that the execution time of this

state test is actually not. Well, it's it's not very alarming. but on the definite no, no, not the definite. On Mainnet. We saw that it was very slow for some clients. and this might be because it's like a very huge well, of course, the state database is very big, much bigger than if you would just run a state test like an empty, or an almost empty database. and therefore likely on Mainnet, because the storage slots not the storage slots, but the database reads they are. They are cold. This, then likely takes much longer, and this might be the bottleneck for mainnet. So what we should likely do is we should run this state test on not a small state.

but actually on a very big state, and then see if we can reproduce the well. Well, how slow this is on on Mainnet. and what I'm doing in the meantime is figuring out if there is something else going on. For instance, like we have some locks, maybe there's some extra lock data which might also be a bottleneck. But in the end I want to create, like a status that we can either completely rerun this entire transaction isolated or to figure out, okay, if there's something else, something very weird going on, something very exotic in the Evm that I create an even more compact version of this test that you can really squeeze out well, the worst case behavior of the of the Evm.

ef berlin office: That's great. Thank you for that. But okay, that sounds like we are still in the reproducing the issue phase, and not necessarily in a proposal of solution phase, right? Yeah, Ben, do you wanna go.

Ben Adams: Yeah. So I I lost Internet at home while I was at the other conference. And so when I came back, I I restarted it, and I did run this block. Unfortunately, I didn't trace it. But I I process 23 blocks. which included the problematic one in 600 ms. So I'm not sure exactly what the problem is. I mean. Obviously, we we did see a lot of valid. They just struggle with it, but might need to look into it a little bit more to, you know fully fully drill down on what's happening.

ef berlin office: So I can. I can talk to what the issue was on. Gas. Basically, we have a we have a buffer that. We flash to the disk whenever it fills up. After 256 MB and this was done synchronously with the the block processing. and so often on these blocks the buffer would just overflow, because we put a bunch of like the the likelihood to to overflow this buffer to go over the 256 MB is very high. And so what what happened? Geth is, we would just spend 2 and a half seconds, flushing this flashing this block to the disk and our fix, for it is to only flush the basically use a double buffering mechanism. Only flush the inactive buffer to the disk. And

yeah, and and and switch to the to the to the active buffer. Yeah, so that should also take care of this. Regardless of what we're doing, like, of how clients actually behave on this block, we should be making sure that yeah, this is that we that we have good benchmarks for these scenarios where we touch a bunch of stuff in the try all over the try and need to recalculate the state root and cannot necessarily parallelize it, because it's all in the same storage trap. But yeah, we will. We will be working on this and and try it out on the perfnet to see if there. Sorry not, maybe also on the perfnet, but mainly on the Shadow fork to see to see how the cans behave.

Ben Adams: So was it bad timing, or would it have always flush the by cash.

ef berlin office: It's it's bad timing, but because it it like normally blocks, don't add that much to the buffer, but these blocks they do so. The the chance of them flushing. The buffer is much bigger than like a random, normal block. but that also means higher the gas limit the worse this gets right unless you make your buffer significantly larger. No, we we kind of well, when the the way we change it is, we will just flush this buffer asynchronously. And so we don't stop the main processing threat on it anymore.

Jochem Brouwer: Yes. So this this birth of fish. That's the state tree. Both are right.

ef berlin office: Yes, yes, that is.

Jochem Brouwer: Yeah. Okay, cool.

ef berlin office: Okay, I guess we should have some more insights on this by Acd on Thursday. So I guess we continue the conversation there.

Jochem Brouwer: Yes, I will make sure that I will check if there are like other weird things happening. But I think also from Mari's point just now that it is indeed the well, a lot of State rights. And in these random contracts or not, random contracts. But like, yeah, there's like a very big state growth here. And so, yeah.

ef berlin office: Yeah. Thank you. Cool anything else on the gas limit. Topic. Cool? Okay. If not, then let's move on to peerless testing. I guess we did fusap earlier. That's mainly Prs testing. I actually think that was the entire agenda. Do we have any other topics we want to bring up today. Yeah. I think we might have an update on history expiry one second on that. Yeah, here's Matt with latest on history expiry. Hello, I just want to share some of the things that we talked about last week at interop, just in case anyone who wasn't there was hoping to get caught up.

Basically, we've talked about what the path forward for the pre merge history expiry is essentially all the clients have implemented what's necessary to do pre-merge history. There's like various things that each client is implemented, and maybe there's not like perfect alignment on every feature that every client has. But ultimately, like every client, can run without the pre merge history in one way or another. So we. you know, have a path to having all the clients choose to run without that history. So basically, now, what we've said as of last week is that clients are free to begin making it the default mode to run without the prune history. They can release this at their convenience. In the next week or 2 we'll have a blog post for on the ethereum foundation blog, just sharing that it's possible to run clients without the pre merge history and sharing how to run on the clients that we typically release information with for the the Fork blog posts.

So we'll publish that in the next couple of weeks. I think most clients have implemented E. 69. This is a kind of nice thing to have with respect to starting to drop the history, because now you're not potentially asking a lot of your peers who just don't have the history. You can know ahead of time whether or not your queries are going to succeed. But this is not a requirement for us, you know, having everyone drop the pre merge history. So that's kind of all that we agreed on for how to sort of close a pre-merge. Then post merge. We talked a little bit about what we wanted to happen there, and we kind of agreed that

we wouldn't modify the way the approach with the error file file slightly. There was a thought that with error one, this would be the pre merge, and then we would use the error files as the post merge, and after talking about it a bit more, we kind of realized that it might just make sense to have separate error files for each layer. So the execution layer will have a file that's storing all of the historical data related to it, and the consensus layer will have a file, an error file with all of its historical data necessary. So we're kind of recalling these, the Era E and Rsc files. We have a rough idea of what this will look like on the execution layer. Yasik is kind of the person who's been looking at this for the consistently, and I think he has a rough idea, too. So over the next, like one and a half months, we really want to settle on what that specification is, because that's going to unlock the ability of clients to start pruning more history. Basically, we want to move to this place where

we have a backstop to access the history for whatever reason, and it's been sort of in. It's sort of up to clients to decide. How do they individually want to handle history? Do they want to do rolling history expiry first, st or do they want to just support pruning up to suppose cancun? Because, you know, before Cancun, we were using call data for L twos. And there's kind of a lot of call data in that period of time. So we just want to give it to clients to have the ability to make those decisions. And the thing that's on the critical path is just agreeing on what is this like shared format? We're going to have

to import and export that data so hopefully, that is something that will be settled on in the next month and a half by the end of July. And then clients can really start to do whatever they want in terms of history. Expiry era itself is not like the only long term solution that we want to have. We do want to have some more decentralized and robust way of accessing the history. And this is kind of like what the portal network has been designed to provide us for a long time, and talking with clients throughout the week, and trying to understand where their portal implementations are, and what some sort needs to be done to get it into a place where it can be relied on by execution. Layers has sort of made us feel like we need to

from 1st principles. Think a little bit about what exactly is Portal trying to provide, because Portal has a long list of features that's very useful for individual like independent portal clients. But there's a very small subset that's useful for execution clients, and some things like serving range queries was not, even, you know, supported by like out of the box given just like how the distribution of blocks are put on the network. So we want to make some changes to exactly these, like very this, like small subset of portal features that are extremely useful for the execution clients. and then integrate this more tightly into the execution layer. So we want to try and modify portal so that it doesn't rely on an external like client system, and it doesn't share the header proofs for every block. Just to focus on giving value to the execution layer. And then, at a later point, we'll start to like slowly add back these features in as they become necessary.

So we're having planning to have a public community call about Portal either later this week or early next week to discuss these types of changes. But this is kind of like the types of conversations that we had throughout the week last week. And yeah, I think that's pretty much where the history expiry project is right now. Thank you. Does anyone have any questions for Matt on the history expiry topic? Okay? Then I guess we'd wait for the community call, and in case you had any questions until then, please reach out to Matt. If not, bring them up at the community, call that being said, I think that's all the topics we had for today. So if there's nothing else people wanted to talk about, then we can end the call a bit early this week.

Thank you. Everyone. Oh, Yup, yeah, okay, that's a buy perfect.

Ansgar Dietrichs: See you bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.