All Core Devs - Testing #77 (BAL Breakout)

Transcript

Awesome. Thank you. Yeah, then let's get into it. So, I posted the agenda into the um GitHub issue today. There are only like three agenda items, and the first one would be talking about the prefetch optimizations. So far, I think Rev, Gef, and Besu have already um delivered on those. So, I wanted to ask if there is an update from Nethermind and Erigon regarding the prefetch optimizations that we would need for our benchmarking.

Uh yeah, still doing work done. Uh should have it this week. Awesome. Thank you. Is there also someone from Erigon who could give a quick update? Yeah, yeah, I'm here, but I'm not the it's not me who's working on that. It's Mark. Uh Mark, are you on the call? Hm, maybe not. I'll check with him. Okay, awesome. Thank you. We can also follow um follow up async in the BlockLab access list Discord.

Perfect. So, we are we have Gef, Besu, and Rev and Nethermind getting there soon. And from Erigon, we hear back. So, this is great because then we are not locked by clients regarding the benchmarking and can can get into it, basically figuring out if we want to keep the state locations in the BlockLab access list or not. And this brings me directly to the second agenda item, namely updates on the benchmark. And I would pass the mic to Jochem, who can give us a quick update there, I think.

Yes, can everyone hear me? Yes. Okay, cool, cool. All right, so about the benchmarks. So, one big question here about the benchmarks is that we created benchmarks not taking into account the current pre-fetching logic of the client. And for this, we are creating new benchmarks and I will directly link that. This is a PR from Tony. And what we do here is we also try to trick the pre-fetcher into adding more work to the state IO. So, what we want to do here is we want to also overload or to add extra work for the pre-fetcher itself.

This is of course dependent on the how the pre-fetcher works without the block access list. So, this is based on the pre-fetcher how it works in ref and in dev. And we also have a variant for the Nethermind variant. And this will be used to also see the optimizations we which the block level access list actually get here. Because what we want to do here is we want to create the worst case for the as benchmark.

Which is the worst case without the block access list and then compare that if you if you activate the block access list and also the optimizations there to see what the benefit is for those. Yeah, I think because I think we should also keep this so much short to get all the agenda items. So, that's the update on the benchmarks. Awesome, thank you. And maybe just for client teams, it would be nice if you could maybe look into those or at least in this deal channel in the in the benchmarking channel there.

We have been talking and describing the benchmarks um for a while. Just to make sure that um we're actually doing the right thing. But basically what we're doing is we have a very small first transaction that writes into a contract um some offset number. The next transaction will read that offset and then start loading from storage slots um using that offset. So, transactions will be dependent on each other because each transaction writes into the same storage slot. And the next transaction will then read from there.

And at the same time, the first transaction sets the offset such that if you do prefetching against the parent state, you will just end up prefetching the wrong storage slots. And for Nethermind, the difference there is we use different transaction senders because I think you guys are not only running it against the parent state, but also per sender against the parent state, which means that by using different senders, we also make sure that um that even Nethermind is facing the worst case. But it would be nice if clients could have a look and yeah, confirm that.

Is there any comment regarding regarding this? Yeah, I have maybe one the comment here that um what we really want to test here is the worst-case situation for all the clients. Um and this is dependent on what optimizations the clients actually use. And since Tony and I are not a uh client developer, um we don't actually know exactly how these prefetches work.

Uh so, this is also dependent on well, the optimizations which all the clients make. And it will definitely be very, uh, important also for each client that to take a look into this test if this actually hits the worst case for your specific client. And this is like not um, uh, doesn't want to uh, um to to put like the client in a in a in a bad daylight, but what we really need to do here is to find out like the worst case for every client. Um, so, yeah, we will need eyes from all the clients teams to take a look if this actually hits the worst case in your client also.

Thank you. Awesome. Thank you very much, Yochem. Um, maybe in re- with regards to timelines, what do you think how long will it take us to to get the first results on those at least already for Geth, Reth, and Besu who are already um, ready with regards to batch prefetching. And then we can also soon add Nethermind when as soon as they are ready. But how long do you think will will this take us?

Um, this would depend, okay, so we have to fill these tests on top of, uh, Bloat net. And we have to run them. Um, and let's see, is Reth on the call? I think, oh yes, Reth was also on the call. Um, so um, when I take the benchmarks, I think that the re-pricing benchmarks which we added this morning are not yet running. Um, and this will then also require like maybe in parallel also these specific ball benchmarks to run. Uh, so this is also dependent on, yeah, like like uh, the scheduling of the benchmark machine, I would say.

Yeah, I can, uh, on that note, I can prioritize, uh, specific runs if we need to. Okay. >> we get results earlier, yes. Yes. Cool, cool. Um then Uh yeah, one one moment, Maria. Um yes, so for the filling like I would suggest that uh for these block access lists we would only fill the tests specifically uh aimed at these uh well, the block access list tests. So, the worst case for the block access list these are like well, on the top of my head, maybe like at most one in the test, so not like 5,000. So, these ones should be very small and they should also be somewhat fast, like not all taking 1 hour. Um so yeah, we we could also run these fast

and also get results uh fast. Yeah. Awesome, great. Yeah, go ahead. I was just going to ask that if we need to prioritize some ones to free up machine space, um I can write down a sync list of things that are like the key things for re-pricing so that we can free up as much machines as possible. Um but yeah, maybe it's better to just just do that, I think.

I'll I'll send that to you, Rafael. Great, thank you very much, Maria. Any further comments regarding the ball benchmarking? Just one question, are we talking about the test I shared on the chat or it's another test? Yes, so this suite is actually uh like a very big suite, so like a lot of tests.

So, what for the ball benchmark what we want to do is to we will only fill the um the specific tests aimed at the block access list and then run those. So, uh the suite which you mean is um a very large Oh well, it is a somewhat large 1,000 1,008 tests. And this will be uh less than that. Any further questions? Otherwise, we can move on to the

next and last agenda item, which is Filipe and Rahul have been working on new test cases. Do you guys want to give a quick intro into what you have been doing there? Uh sure. I Yeah, I'm not sure how that uh who added the the bullet point here, but yeah, I have a PR open for a little bit more testing around invalid cases. Um I brainstormed uh some some missing coverage there.

And then um I thought I believe Rahul had something to he wanted to add on top of of this. Um and so I'll yeah, I'll give it a I'll back to Rahul. Yeah, so um we've been trying to like add more coverage to the invalid test cases. Um and we think invalid test cases are like quite a lot import important than the happy paths that we have. As so far, we've been adding the invalid test cases manually.

Um so we're kind of exploring a path where we can probably generate invalid test test case automatically for all the valid test case that we have. So that should increase and cover a lot of edge cases that uh that's been uh lately. So, I think Ignacio reported a edge case where some of like the system transaction around withdrawal were causing a bug. So, the approach that we're taking should should increase our coverage by a lot to cover these cases.

Awesome. Thank you very much. I think you also worked on a table or something. I I just remember that you posted something somewhere where you had a table which clients are actually comparing the ball hash with the hash in the header. It would be nice if you could post that into the ball discord actually just to get a a sense of because if you compare the hashes, then basically nothing can happen and you should always be arriving at the right ball. And yeah, if you don't compare hashes, then things like spurious entries, for example, having too many withdrawals or yeah, local access list entries from non-existent transactions essentially,

they could be could end up being not caught. So, maybe Rahul, it would be very cool if you could just post it somewhere for clients to have a look. Sure. The TLDR is that we recommend clients before execution you compare the provided ball from the block body and compare it with uh the block header to ensure that you don't start loading state which is coming from an incoming like bad corrupted ball.

And then post execution, you try uh hashing the generated ball and compare it against the header. But, this is this is not like a must. So the idea is that the test we now start doing will ensure you know you know no no spurious entries or incorrect values are in the in the in the bell. So even if you don't do hash and compute like a virtual balance at the transaction level these tests will ensure you know clients don't accept the correct things.

But I'll I'll drop a message in the Discord. Yeah, and just to add to that because I think you um you just um phrased that um wrongly. Um clients cannot really do anything um pre-execution because over the engine API there is only the block live access list. So you would take the block live access list, create a hash from it, and put that into the header. But I think you you would just I think you you meant it right that you would then compare post execution the derived balance that you built during execution with the value in the header.

I guess that's what you meant. Yeah, yeah. I'm I'm talking about the validation part. So if you if you're the validator you'll ensure that the block body balance matches what's supplied in the header. Exactly, yeah. Awesome. Any further questions on that? Otherwise, I would say we can move over to discussing open points on 847. Okay, it seems like that's not the case.

Yeah, how should we continue from here on Maria to you wanna um do you have like a a list of open questions that we want wanted to discuss or do clients want to just take over from here? Um right, so we have um two uh documents. So we have Spencer sent. I can maybe share he had these um summary with this proposal. Um and then I have these documents that does more of like a deep dive into each question.

Um so with 10 minutes I don't think we'll solve all of these. Um so maybe um what we can do is we can I can just give like an overview of the general um open questions so that we are all in sync. Uh and then we can start the discussion one by one. Does that sound good? Okay, perfect. so maybe just to help I can maybe share my screen.

let me see. Okay, um can you all see this? Yes, yes, yes. Okay, perfect. Um so the first one I think this is something we discussed before. So this is this issue where we are when we are in a call frame so and we we have um a a revert um we are refunding the state gas um back to the to the to the parent, right? Um because no state is created so we shouldn't charge for that state. Um but at the top

level, so if we are just during normal execution, so we haven't entered any call frame, um at the moment the design is we don't refund. So, if there's uh um some some error in the middle, we we don't give any refund. so I think this is one of the examples where we are kind of like we are into these two options. So, we have on one hand the general principle of ideally we should only charge the things that the users use, right? Like if if they are not creating state, we shouldn't be charging for state. And um the other um issue is the precedent. So, by not charging state

in this specific scenario, we would sort of be opening a new precedent because how the EVM worked before is that um this refund wouldn't wouldn't be given. Um and so I think a lot of these options, the uh sorry, a lot of these issues, the the question is like the these uh tension between the general principle of only charging for things that are used versus opening a new precedent um in some cases. Um Uh Ben, do you want to to say something?

Um Yeah, there's this essentially because if you have an element with a S store, you charge the gas and if you revert, you're not you're not refunding the fact state. You you spend the S store gas before the revert, you're still charged for it. Whereas in this Right. now we're making a distinction, did you write state? Right. Yeah, okay. Got you. Yeah. and I think a lot sorry, a lot of these also comes from the fact that previously some of the costs were separated. So, an example is and maybe I'm I'm I'm going a bit too

fast, but on issue number three, this is the same thing. It's like you are doing a contract deployment, something fails, and then what do you do? previously you would charge the call the create because the cost of the create would be charged before starting the deployment, right? Um but now we have a decision like we can say we still charge the cost of the create just for the execution cost, right? Not the state growth cost, and then we only charge the state growth cost if the account is actually created. So, this is a a decision we have now, but again, it's this tension between

keeping with the precedent of the EVM where you would just charge it. Um and um now you can decide to not charge it. Um so, issue number two is very similar, but it's with S-store. So, it's when you kind of create it during the same transaction and then um revert back to zero. Um issue number four is um is is very similar. Um is when you essentially create and you have it's successful, but then you do a self-destruct. Um and so, in this case, um do you do the the the refund or not?

Um then issue five is kind of like very similar to to the other two as well, but now is it is uh a call. Um I think this one requires mostly um like um how do I say like uh uh to be more specified is not an issue per se. Um and then I think issue six is the one that is a bit more different, so issue six um is happens with um authorization, so um 7702 authorizations, and um the issue here is when So, in set

delegation, essentially we have these costs um if an account exists or not. Um and the way this works is um at the moment you have these intrinsic um costs. And then if the account already existed, um you essentially give a refund. Um and this is essentially how the EVM works today. The current specifications, my understanding is that it's going away from these and actually um changing the intrinsic costs. So, what the current spec is doing is

refunding back um what was the intrinsic cost. And this is also another deviation from the like it's a new precedent, right? Like we have this thing that the intrinsic costs of a transaction should be fixed and shouldn't be something that changed during the execution. And so, this is another another question, like what do we do here? do we uh mutate these intrinsic um gas, or do we do a refund um at the end? So, this is kind of like the question. And then issue number seven was um

that uh issue that was raised by Sorry, I'm having sort of blank. Um Let me see if raise it. I think it was Jared as such as Jared um raised this um in the still this call. And is um about the the validity checks when you are seeing if you're going to start executing a transaction. Um and at the moment I think we don't have anything specified in the EIP, but is actually my understanding is like most um clients and the current specs are aligned with this formula that

I'm showing where you just check um if you have enough regular gas available um for the entire transaction limit. and what I'm saying here is like um we could be a bit stricter than this because we can also check if we have enough state gas available. So, this could make sense like if we don't have enough state gas um and we start executing the transaction and then we figure out at the end that we don't have enough um state gas available. So, I don't I don't have like a a big um a strong opinion here. I think just doing this already protect us in a way

against DDoS. Um so, it's just um I think something we need to discuss and I think a bit more carefully. And also, I think again we maybe we won't have time to go over all of these. So, maybe the key thing is like I ask everyone here to read these and try to come up with like their opinions on what we should do here if there are any and then we can discuss these more carefully on Wednesday. And in the meantime, if there are any um opinions or any questions, um happy to take them in the final minutes to have.

Okay, now are there no questions or comments already on these? Yeah, I I think the main action point here is that also all the clients here take a look at this document and then before Wednesday well get at least like a total of insight into this because well these are like some very well I would not say yeah well critical open questions which needs to be answered but we would need people to have the insight here to actually answer these. Yeah, thanks.

Yeah, and and Spencer I think that's a good idea to open a thread for for each of these. Okay, then anything else we should discuss or we just wrap up? Um one thing could I get some feedback on this pull request? Uh for the EIP in general not now but I think it's fine. It would be nice. I'm not sure if it's in your list Maria. As far as I saw it wasn't.

Right, so I think it is part of one of the issues but it's it's I think it's also something we need to figure out. Let me see. Yeah, I think this is this is listed maybe issue three. Um but still I think independently of what we decide from issue three I think this should be clarified and I think your approach makes sense.

But again, like if there's anything anyone else that has any more comments here, please we can we can Oh, yes. I think there's a a hand up. Uh just wanted to ask in case of revert this build gas is returned to like reset like it it did not happen or it is still counted. Uh Sorry, do you mean in a in a call frame or in the top level um execution?

Well, it depends on what this PR means. I mean um state gas still increments execution state gas used, but if it's reverted, we don't count Uh sorry. I guess it's not a good Yes, so my understanding of this API, I can give a short answer. I think it's um the idea is that we were in cases where there was there was refunds and things that changed in in the available gas, then we were sometimes we were not

um translating them into the gas used. Um and so sometimes there were some some some mismatches. So, this is my understanding of of the thing. Um but maybe Stefan, if you if this is incorrect, please let me know. Um sorry, I didn't catch the last part. So, I I was saying that the the my understanding of of your PR is that in some cases where there's changes in the reservoir and the gas left, sometimes those changes don't translate to the gas used variables. And so there's some mismatches there. So sometimes for instance you do a refund to the

reservoir and then you don't change the gas used. Um and so in in those cases those variables need to be consistent. So this is my my understanding of it. Yeah, for sure. So um I think there are so many edge cases that uh sometimes the accounting mismatches and the PR I shared just really tries to clarify exactly how to handle the edge cases. Cool. So should we wrap up?

Okay. Um So I don't know how the breakout thing works, but I guess we can just turn off. I guess we can leave. Yeah, perfect. Awesome. Thanks everyone. Bye-bye. >> Thank you. Yeah, thanks everyone for joining.