All Core Devs - Testing #41

Transcript

Mario Vega: Okay, Josh, I think we can. We can start. Just let me know when we're ready.

Akash | ECH: Yeah, we are live.

Mario Vega: Excellent. Thank you. Hello, everyone welcome to Acd number 41. Today's June 23, rd and we have a a few items in the agenda and the main point being for second 2 and a couple of discussions regarding Eip, 17 9 0 7. So yeah, let's get started. I think we can start with. The the 1st item agenda. I should also shared the link in the in the chat. Please chime in also, if you if you want to leave comments over there. Yeah. So the 1st item is, Fussaka, 2 Barnabas, I think you're the person that can can thank you.

Barnabas Busa: So basically, we are still waiting for client implementations. Therefore, we look like we're not gonna be able to launch today. Hopefully, tomorrow, we're gonna have enough cl tabs. let's say that their branch is ready. But as of right now, we only have Yale implementations done. So we are still waiting.

Mario Vega: Okay, thanks. I also want to show that we have a release for the eels tests. and we also have a hive instance running right now. And Spencer, do you know, do you have a summary of which clients are passing or failing? That you want to share just in general. yeah, if you're talking, you're muted, but otherwise I can. I can share what he relate to me. A couple of minutes ago. Basically, only red is passing. If I understood correct correctly. the hype tests and the rest of the claims are not either running or passing. That's what I understood correctly. But yeah, we can. I can share the link in a few minutes for everyone to see which clients are passing or failing the tests.

anyway. Is there any points that want to be raised about Devnet 2 by any of the clients that we should discuss any blockers, any or or any problems with the specifications or implementations that anyone wants to raise. We can start with. Execution, client clients. I see ben from another mind. Do you have any updates on the status of W. 2.

Ben Adams: Yeah, everything. Everything's okay. Apart from 7, 9 0, 7, which as mentioned in Acde. I'm in the latter half this week.

Mario Vega: Cool anyone from Besu that wants to chime in.

Gabriel Trintinalia: Yeah, this is ready for devnet, too.

Mario Vega: Any issues, regarding specification or any anything in particular. They want to raise.

Gabriel Trintinalia: Not not really. No issues. Just waiting for The 7 h, 7, 9 0, 7. Conversation. But no issues. Yeah. Love for them. That, too.

Mario Vega: Thank you. Anyone from Geth Marius, or is it Matt? In the call.

Marius van der Wijden: I'm not sure if we have a branch yet, but we have 7, 9 0, 7 implemented, and we will prepare a branch with it.

Mario Vega: Thank you. Anyone from from Ref. I see Raman in the call. You have any updates on Devna 2 implementation.

Roman: We we posted the wrench earlier today.

Mario Vega: Thank you. Anyone from everyone. The status on implementation.

Milen Filatov: Yeah, Hi, yeah, so we're gonna we're finalizing the the branch today. there is one failing test which I'm working on at the moment it's for the block, Max, rop size eip? So yeah, once I have this test passing, then we should be ready.

Mario Vega: Thank you. I don't think. Is there anyone else from eels did I miss someone from the eels? Or was anyone from the sales. I see. I see, Enrico. Do you have any updates on the sales status?

Enrico Del Fante: Yeah, from tech. We have. We are working on on Bpo related things. and we have a bunch of things to fix already, not following closely this, but hopefully, we have that sorted out soon. Part of that. Yeah, nothing more.

Mario Vega: Thank you. Anyone from lighthouse available. or status of the new 2. And Grandin, I think I see soldiers in the do you have any updates.

Saulius Grigaitis: Yeah. So so we kind of have a a version that is likely could be a candidate for testing. However, we, it seems we don't have another cl client to test against. So yes, if there is somebody who is very confident in their cl implementation, let us know. We will try to test against that other sail client.

Mario Vega: Oh, thank you. Let's start. I see, Phil. Is there any updates on the implementation?

Phil Ngo: 8 we've been working on our branch. We have a couple more things here that we need to merge to be definitely. But I'll I'll hand off to Matthew to give updates on those specifically.

Matthew Keil: There we go. Can you hear me now? Sorry phone issues. We're have Prs up for several of the features all that's gonna get merged today, and we'll be ready for tomorrow's launch that Barnabas mentioned.

Mario Vega: Sounds great. Thank you. Anyone from any other cl client available to comment on status for Devnet 2 who wants to chime in. Casey mentions in the chat that prism is ready for tomorrow lunch. Great! Alright! Good! Good If nothing else. I think we can. Keep the the chat on the status in the R&D Disco, please post and ping a any anyone from the team so you can. They can know which image they need to use for. W. 2. And yeah, if anyone has any questions about the yield tests also, please ping us either me or Spencer to.

So we can help you out. Obviously, there's like some conversation that is gonna be had today and the the test. But the tests are not gonna change for this segment. 2. Probably they're gonna change for Tabna 3 in regards to el testing. All right. Let's jump to the next topic, which is pure dust, testing

Jochem Brouwer: Can I just chime in quickly.

Mario Vega: Yeah.

Jochem Brouwer: Yeah, because I want to present something about the benchmarks of 7, 9 0 7.

Mario Vega: Yeah, of course, I mean, yeah, of course. I wanted to just clear out if there was any anyone wanted to chime in about your desk or status updates about that. If not, we can just jump into your into into.

Jochem Brouwer: Okay. Yeah.

Mario Vega: Yeah. Bernard, do you know who is the person to comment on Peerdas? Or do you have anything that we want to discuss regarding Peerdas?

Barnabas Busa: I don't think there's too much to discuss regarding periods. To be honest. I think we want to launch that net 2. And that's gonna give us the new 4 digit stuff. And then we can start testing. Really, the value capacity changes that have been introduced interrupted, not too. But at the 1st look Berlin dropped them to already have updated the Cgc values. So yeah, I'm I'm really just waiting for them to to launch so that we can have a nice new network to test everything with.

Mario Vega: Perfect.

Barnabas Busa: Maybe some side lips. Maybe maybe some side lips have something.

Mario Vega: Yeah, they mentioned in the chat that they want to present. yeah, if you want to chime in some lives. Yes, please. Just go ahead, we can. We can. Yep.

J Sunnyside Labs: Yeah. Hi, I will just briefly talk about the analysis that we found from Berlin interrupts, and like the days few days before that. So yeah, I think like. for some of us who went to Berlin knows that. Most of the cl clients we test like for all Dcl. Clients. We tested single cl devnets. So we ran like 18 devnets in total at the end, and we found that all the cl clients, except for Nimbus at the time, were able to reach up to 72 blocks per block. Congratulations and We made analysis on where where were the bottlenecks to please have a look? Every cl has some bottlenecks when it reaches to 72 blobs.

and for the next up we may do network bandwidth limiting tests, but the metrics we get it for full nodes. Don't look that great grand dynamic lighthouse. Full notes consume 20 megabits per second on average. and others were above 80 or 100 megabits per second. So this was the major bottleneck we found. Yeah, other than that there are some, some others that are also part of the report, which I wouldn't say it now, because they are too long. But please have a look. How could yeah.

Mario Vega: I am sorry I cut you off. Please go ahead.

J Sunnyside Labs: Oh. I think Anzker has a question about stability on the realistic networking. So we mainly focus on the limitation side. So at the very worst case, what happens? So we. At the moment we didn't limit any network bandwidth, and we are about to begin constraint constraining the network. But what we found so far is that we probably have the network failing much before than 72 blocks per block. because full nodes are consuming a lot more than expected.

Ansgar Dietrichs: Well, and maybe just to briefly ask, do we? How close are we to understanding? Why? Why, that is the case.

J Sunnyside Labs: In the report. I mentioned that number of column verification is a lot for some of the Cls. That might be the issue. and Not sure why, but it was generally increasing linearly when number of globes increased and Els didn't have much impact on that.

Raúl Kripalani: Yeah. So 1 1 question here, were you testing only with publicly available blobs.

J Sunnyside Labs: Yes.

Raúl Kripalani: Right and were you able to trace down if the failure to propagate, or maybe the amplification was at the El level or at the Cl level.

J Sunnyside Labs: Sorry I didn't quite get the question.

Raúl Kripalani: Yeah. So there's so blobs propagate. So if you're testing with public blobs, there's type 3 transactions that propagate in the El blob pool. Right? So what we're trying to kind of like figure out, maybe as a 1st step is whether the type 3 transactions failed to propagate, or whether the C they propagated. But somehow the Cl. Was kind of like amplifying traffic too much, or something.

J Sunnyside Labs: I I think the letter was the case. So we also tried to see El Usage, but it didn't increase that much. So get. The notable difference was between lighthouse grander and compared to others. So others were using like 80 megabits per second, more on average, than lighthouse and Grandin. what did you mention was linearly, I'm sorry. Thank you. Linearly scaling with blobs. So we used spammer to increasing to increase blobs over time like plus one block every 5 min. And then, when we say, as time passes, we say,

The network traffic was also linearly increasing.

Csaba Kiraly: Yeah. But that's that's normal with actually with like, that's what we expect, isn't it? That's why I'm asking, because because the many things is the only design. Many things are, you know, just getting with the number of drops. So that that's why I want to understand. If if you've seen something unexpected, or it's it's actually what we expect. So the the message sizes, the column sizes linearly growing. And and anything done on the column level is, is just linearly growing with the increase of number of blobs, with the with the one d. Design.

J Sunnyside Labs: Yes, but I don't think we expected this much of network usage, don't we?

Csaba Kiraly: Yeah, no. The question is, was it? Was it the network usage generally scaling? And it's more than what you would expect even with with foil blobs. You just don't notice, because it's below this threshold of of noticing it. Or so, if if that's why, because it was nearly scaling, that means that it's it's just more than even with small number of drops than what we would expect rather than.

Raúl Kripalani: Yeah, I think I think we need to really dissect these results closely. So I can from the peer to peer networking team. We can do that. We'll look closely at the report, and we'll we'll chat with with the Sunnyside labs team to get to the bottom of what might be happening here. There's also kind of like a duplicate effect factor here. I don't know what the status of implementations is, with regards to the latest improvements that that we've introduced, things like I don't want, and and other things which, by the way, would only be visible really, when there is a network latency which I don't think in this particular case there was right. There was no specific network configuration enforced here, as far as I understand.

Csaba Kiraly: No, no limit at all. There's justification. So you have the the geographic distributed. So you have some latency effect. But but.

Raúl Kripalani: Right? Yeah, yeah, yeah. But it was all all on the same as right. Probably run on aws, or something like that.

J Sunnyside Labs: On, digital.

Raúl Kripalani: I think, yeah, right? Yeah. So it's not. Re, it's not real world. Effects. I I do think that we need to look at exactly the commits of the networking stack that the clients were running at that point in time, and the implementation of some of the latest optimizations. But yeah, this this is. This is really good flag. We'll we'll look into into this.

J Sunnyside Labs: Yeah, sure. Yeah, we will collaborate with you guys. Thank you so much.

Mario Vega: Raul do you mention? There's something our engine be get block speed to that you want to talk about. I think we can go into that first, st and then come back to 7, 9 0 7.

Raúl Kripalani: Yeah, sure. So yeah, I think it's, it's the relevant context. So yeah, so I submitted, this pr, earlier, today, this basically proposes a small change to get blobs p. 2. To restore the previous functionality. Where get blobs would return partial hits so when in v. 2, we switch to an all or nothing behavior, where, if a single blob is missing from the El Blob pool, then the whole response would be a null which made sense from the perspective of the current design of pure das, which can only utilize full columns. So if you're missing a single blob, then you're missing a cell in that column, and there's nothing that you can do. So you have to wait for the network to receive that column from the network on the Cl side. But we're working on a bunch of

Optimizations at the peer to peer networking level. Specifically at the gossip sub level that would allow us to do to send cell level deltas. And of course, there's like we're prototyping these, and there's a bunch of testing that we need to do to figure out specifically the timing conditions that would lead to good results. In in good outcomes in in this case. But yeah, the idea is to make whatever is available in the El useful to the to the Cl so that we can leverage optimizations that we can introduce without forts in the network. So this, these would be implementation level optimizations. And and yeah, so this gives us so basically, what this Pr is doing is proposing that we introduce a partial response, flag to the request payload of get blobs, p. 2.

When that is true, we would return the blobs that are available and null for the blobs that aren't, and if that is false, we would just continue with the current behavior which is returning a null for as a as a global response. We would. At this stage we would probably deploy Cls would be using the false value here. But then, as we roll out these optimizations as we and we agree on them and we implement them, we test them. And and we see the lead to good results. They could switch to a true and start utilizing the data that's available in the in the block pool. And the reason that we that we suggested Mikhail had a question of why don't we use partial response through as the default logic? The reason is that currently the Cl won't be able to utilize the the partial responses. But we do want the flexibility so that we can switch to to that behavior to the the partial response, false behavior when?

sorry the the other way around, I think. Yeah, so that we can switch to partial response whenever whenever we should. The optimization. Yeah. So Marius, the idea. So I see you. You asked, why don't we just return partial responses, anyway? And the reason is that we can't use those partial responses. So would be currently would be on undergoing the cost of survey on both sides for nothing. So so yeah, this flag would would make it would make it dynamic.

Marius van der Wijden: Yes, but I think in this, in this case I would just say, let's make this the default behavior and not introduce a another flag like the cost of.

Raúl Kripalani: That's.

Marius van der Wijden: So it's not that much, and.

Raúl Kripalani: Excellent.

Marius van der Wijden: Like my-, my personal preference would just be to make this the default behavior as we had it before the spec changed. and let the Cls filter out while I look at the responses and see how many of those blobs are they- they- they do not, if they have all of them, and if not, we'll just not use it.

Raúl Kripalani: Functionally speaking, that's good that that works for us. I was just kind of like going the extra mile here, just just in case we want to serve save the survey cost. If we get to say, for example. 36 blobs before this optimization, and we realized that the base. 64 encoding is wait like I don't know. It's adding, like maybe 20 ms, 30 ms, or something like that. And we kind of like, you know on on each side. I don't know how much that is. I haven't benchmarked it. but but yeah, I would be currently returning. Those blobs would not be useful right now on the cl, anyway. So that's why I thought this would be a simple.

simple solution to save extra work. But we both know that it's not useful. but I don't have a preference just returning partial responses by default and removing the flag is good enough for us.

Mario Vega: I think the just want to. Raise the point that maybe if we keep the discussion into the Pr, please just chime in. And just to get this either merged or just to not block this this pr, do you need any feedback in a specific role that will help get this pr other measure close.

Raúl Kripalani: Yeah. So I would just like a outcome here. I think we it's a very simple issue to to agree on I. And it seems like, based on the reactions, on the chat. I see that most people seem to be agreeing that we just remove the option so I would just what I'm gonna do. If if people think it's fine, I'm just gonna rehash this Pr and just remove the the flag altogether and just change the behavior to get to to a solution here is just very simple.

Mario Vega: Any opposition on that. Resolution. Okay, that sounds good. Then. Thank you. If nothing else. I wanted just to. Raise this point by answer in the in the chat regarding timeline. Answer. Do you want to comment Or ask the question just

Ansgar Dietrichs: Yeah, I mean, I'm not sure. Feel free to tell me if this is just not a good fit to ask on this call. I'm just wondering right now. Whether we are working towards like a specific list like like check checklist or something of remaining stages here between now and being ready to basically start forking the test nets in terms of Ps, like what exactly we need to to see, also to inform us our our choice on on the the exact details of which values we we use for the Bpo, and if we have a specific like checklist in mind, then a, is that written down somewhere, something that that that could be linked, and B what the rough timeline, I mean. Again, optimistic, of course, would look like for this.

Mario Vega: I don't know who the person to answer this. But maybe just to bring some context. Barnabas, do you have any like comments on on this that you want to bring up. It. It's a sense to me that we want stability right in on at least, yeah. as before. Yeah.

Barnabas Busa: Yeah, we we ideally want a table spec, like a frozen spec, which it seems like we don't even have right now. We most likely gonna be going into the next 3, and possibly 2 weeks from now, or maybe even a bit later. By then all these questions that were raised today should be clarified. And then we can actually discuss about what else do we actually need? Because we are now at a happy case scenario, where everything is working with limited number of blobs testing Cpo numbers is impossible to do on any test net. So we're gonna need to kind of do analysis after before full and determine the Bpo values based on that.

Francesco D'Amato: One thing is, I I think, as soon as definite, the the next 7 is disabled like without necessarily waiting for Devnet 3. Even if let's say, I don't know, this small change is is made that we just discussed, we should probably start like more well doing more of the the kind of tests that now the the Sunnyside Labs people are doing and and just really like, I mean, yes, we probably cannot fully figure out the the safe number of of lobs for the Ppo schedule. But just generally we should start working towards like understanding what the bottlenecks on the Cl. Network inside are better than than we do now. So part of that is just like following up on the I think you know the questions that were raised by the report. But yeah, ideally, it would be nice if we can, maybe

at some point start in parallel, like spinning up some like bigger devnets than even, you know, 50 nodes. And and maybe yeah, just just start understanding really what the networking level or like verification level kind of bottlenecks are. I think then I mean, once we understand that it's there's not. It's it's hard to say timelines, because then it it might take like we don't know how long it's gonna take to address those bottlenecks, but at least like having, like a complete list of all the things that are not fully ready in terms of the the networking performance would be the start, I guess.

Barnabas Busa: Yeah, sure. But I also feel like fundamentals should be working before we doing any kind of tests. So like manager custody is a fundamental feature back feeling, in my opinion, should be a fundamental feature that all Cl support and currently none of them support it. So we still have a bunch of fundamental topics that are not yet implemented by many and doing stress tests and doing edge case testing already when we can't even do backfills is seems a bit stretched out.

Francesco D'Amato: Yeah, I agree. I think we should 1st have at least 7 or 2 being stable and in the meantime, yeah, I mean the same kind of reports and tests that are being done now can can still be done, and we can investigate them. But yeah, I think other than maybe you know what we discussed today. We shouldn't make any bigger changes to to the spec. So hopefully, like Devnet to being stable is, is like the, you know, kind of the last point at which like. Once we see them in the 2 stable, then we really can start doing these other tests? More seriously.

Ansgar Dietrichs: And maybe one last follow up question. Then that's that's helpful. Do we expect to basically just as long as, say, the once we start just testing really out on the realistic networking conditions if we get and and and on foulu and basically we get some sort of numbers that with current client behavior we can support is the expectation. As long as those don't look horrible. We would just then immediately base the Bpo. Numbers on top of that and start rolling towards Maine, moving towards main rollout. Or do we expect to basically try to have some one or several rounds of of feedback, of trying to basically unblock the bottlenecks and get to highest possible performance, and only then lock in the the Bpo numbers.

And and do we know, like where that kind of decision cut off would be in terms of what numbers we would have to see, to immediately move towards Rollout versus keep it improving. or will we just make these decisions on the fly as we get closer to that.

Raúl Kripalani: Think, from from my perspective, what would be useful is to continue working, for example. So both kinds of testing that Barnabas and and Francesco were mentioning are are useful but also like getting finding those bottlenecks and the networking level bottlenecks, and specifically those that are resulting from maybe the design of things that like, then place specific workloads on the network and so on, like those would be really, really important to find as soon as possible. So the kind of benchmarking that Sunnyside labs is doing is very useful at this stage. So we just need to get to more realistic scenario so that we can start seeing those issues, larger scales and also really understanding what the what, the version of clients that were used and the specific configurations at the at the networking level machine level, and so on, were as well. I would say from like this, what is what would happen next?

I would assume that pure das in itself, nothing about the design is likely gonna change so based on what we discover, there will be specific, maybe implementation level changes that we can adjust optimizations. We can review some code and like some some parts of how the implementations are using the networking stack the networking stack itself, maybe even like machine tuning. There's a bunch of things that we can do there that would potentially unlock higher, higher numbers and higher levels of scale, and based on that, I would assume that we that we would set the Ppo schedule.

Mario Vega: Alright, just to try to finalize the discussion here, I think. We can see the outcome of that 2 and then continue to discuss this in Acdc. if that sounds sounds good by everyone. All right. Cool. Thanks. Thanks, Raul and Sunnyside lights for the for the discussion. Yeah. And now, going back to Eip 79 0 7, I think Johan wanted to discuss the current status and the findings of benchmarking. Johan, do you want to share in.

Jochem Brouwer: Yes, let me quickly share my screen. Can you guys see my screen.

Mario Vega: Yes.

Jochem Brouwer: And also my mouse.

Gabriel Trintinalia: Yes.

Mario Vega: Yes.

Jochem Brouwer: Very cool, very cool, all right, Marius and me. We made some. We did some 7, 9 0 7 benchmarks on the Shadow Fork of Mainnet. So we can't test bigger contacts there because we can't deploy those. What we did there is we sent transactions with a gas limit of 30 million. and we have 6 targets, one of xcode size small. What we do here is we call xcode size on the target. This calls for more than 11,000 unique contracts with a size of 12 GB, also for xcosize big. This targets the maximum contract size currently which you can currently deploy of 24 kB. The reason for this is that if you have these 2 results, you could extrapolate

the results to somewhat see what would happen if it would. For instance, double this this size. What we also did is you did a statical on these small contracts, a statical on these big contracts. So 12 kB and 24 kB. And this includes jump test analysis. What it does is at the start of the contract. It will jump to the end of the contract, and this contract itself is completely filled with jump tests. So this is so you should really analyze this and also for these contracts. Each contract is unique. So it is forced to do this gym test analysis every time. and also extra. This is statical, small, aesthetical, big. And this goes into unique context, and it does not do the jump test analysis there, because it immediately executes the stop of code. So it does not have this jump analysis.

Well, here are the results. These are the average execution times of every client. Well, you can see that Basu, for instance, is well, yeah, the slowest. Apparently there are also some some very weird resource. For instance, Gef has for xcode size, small and xcode size big. It doesn't a slower average or sorry a faster execution time for the big xcode size contract. So that's a very weird result. But yeah, what we can see this is already like somewhat alarming. I would say that Bezo has a well, let's say more than 800 ms execution time for these statical big contracts, and this is only for a guesstimate of 30 million. So it's not like our target, which I think is 60 million or something like that.

So, yeah, this is, this is maybe somewhat alarming. We also have, like the worst case blocks. This was more than a second for xcode, small for Bizu, almost a second, for never mind, for the statical small jug test. Well, this is, of course, also a interesting result, because we would assume that the big contracts will be the well. The worst case here Gef. Is more than half a second, for statical big Ref is well, 250 ms for xcode. The size big and irregon. The worst case is 130 ms for statical, big. and yeah, just a disclaimer. These results are preliminary because we also need to dig into the behavior at 45 million guests, 60 million guests, 100 million guests. And we also need to see what happens. And for this we would need a new fork.

because we will need clients which would support deploying contracts which are bigger than 24 kB to analyze this behavior. And of course, also this weird behavior. For ef why is this X code a size small? Well. worse than the xcode size, big. And also like we have to look into the client's caching behavior because this could also well influence the the behavior. And as a general point, I think the testing team has also seen this. Currently, there are some well, some edge cases in the eap and in the implementation, and also for the for the to write the tests that are not. Yeah, not very well specified in the erp. Yet so we should specify these these things in the eap to make this clear.

And I'm seeing a question in the chat. How are the tests executed? Yeah, that's various comments on this. Maybe Marius has to add something here, or someone has a question.

Mario Vega: I think I think Marius already responded, that it was running perfnet

Jochem Brouwer: Yes.

Mario Vega: Yeah, okay.

Jochem Brouwer: Yeah. So just as the disclaimer, we need more time to look into this. But yeah, as we've seen. yeah, the times are very well. not not super nice for some clients. So, yeah. and this is so without any coda size increase for 24 kB context. Max.

Łukasz Rozmej: If I can weigh in it might depend on which hardware are you running? Because a lot of jump desk can be vectorized. And that depends on the hardware. I expect this was hardware support x 86, supporting awx, 2. Hardware. So 256 bit instructions you can optimize for also awx 512 and for arm 70 instructions. But the results will vary. You can optimize potentially the jump desk analysis for worst for best the best case average case, that it will have more jump desk, or more push of codes. And if you optimize for one, you kind of potentially are trade-offing for the second one. So that's another thing that some clients can optimize for one, some clients for the other. But it's hard to have something that

would work for both especially fast. So that that's another potential problem, that the worst case is one or the other right, or either all pushes or no jump desk or full jump desk without pushes. What else? Yeah, that's that's kind of it that it can be hardware dependent. It can be implementation dependent on on the optimization side. A bit just wanted to give more context.

Jochem Brouwer: Yeah, okay, just to weigh in on that. You are right because you can optimize for either the push up code or for the gym test opcode. But of course, what we want to test here is the worst case. So yeah, then then I will just add a gym test analysis for the push also, so that you can't really optimize for one or the other, you will still get like the worst case out of it. Yeah, in the end.

Łukasz Rozmej: Yeah, but it would be good to have benchmarks for both cases, because it can be very dependent on which client chooses.

Jochem Brouwer: Yeah, I agree, I will add this, yeah, good point.

Łukasz Rozmej: Also we will be adding additional metadata and save jump this, probably to to disk along with some kind of index for size.

Jochem Brouwer: Yeah, cool sounds good. Yeah. Just to mention also, also for 7,907. This has the implicit behavior that you have to cache either caches on disk or caches in memory like this code hash to decoder size. Lookup table, because if you don't do this, then well, you can, you will really run into issues that you have to to read massive data from the disk. And this is really implicit, that we that every client adds this, I will also add a test to East for this to test.

Ansgar Dietrichs: Yeah, maybe then I can, because I wanted to basically talk about that specific thing. So to me, basically that the nice thing in a way, at least, about the jump test. Worst case is that the jump test is should at least in principle, not be a new worst case caused by the cip. because the idea is that the Ap charges proportional to the code size. Right? So now, a contract of double the size should also cost double the amount of gas, and the problem that we just have is that you can only really enforce that once you know the size of the contract. But you run the jump test analysis after you know the size of the contract. So in principle, by this point, there's no way of dosing the client basically like, yes, jump test runs slower if the size is larger. But basically this is, then

if there is an issue here, then it's just an existing performance issue which we still have to take seriously and tackle. But it's not a new issue caused by the Erp around jump test, at least, whereas the one thing that you can't so easily bound by this pricing is the reading from from the database itself. As you just said. So basically like, There, you really need. You need to basically choose a strategy, either. You just accept that now, your worst case got worse, maybe not quite twice twice, because it's still a sequential read. So it's maybe a bit faster than twice as bad. But we need metrics on this right, like how bad the performance regulation gets, and they just accept that or you implement this optimization that you have a index that you keep.

That is a mapping of contract to contract of address or hash to the size. My big argument here would be that I personally think we can't treat this index as an afterthought. I think we need to have a agreed upon standardized behavior across all clients. Because if we don't, basically, then again, we have now very variable worst cases. I think we need to have an agreed upon. We need to have this decision. Basically like, do we want this index mandated in all clients by the time Fusaka rolls out or not. and and in general, I personally, if if we lean towards doing it, then I I have some pretty severe concerns around this index. In general. I personally really dislike that it's not

in any way enshrined, meaning that there's no root hash of it committed to anywhere, and that really creates quite a few issues in the short term, for example, already there's some uncertainties around how it interacts with block level access lists, because you basically have to indicate that the code size was read instead of the code itself. So you basically have to modify the format there, which is maybe possible. But then, once you go to Zk witnesses, which is something that we at least want to start doing very soon in parallel also this year, that basically becomes impossible because the Zk in Zk you need to be able to have a some sort of trace back to some trusted hash, and that just doesn't exist for this index. So within the zk proof you'd always have to load the full code. And even if then you actually run out of gas and can't actually use it. So that means that that further degrades zk performance. And in general, I personally think that this index has pretty severe forward compatibility.

Considerations that I just don't see like they are not addressed at all in the eip, which to me means that actually, it's this late into the fog. I have severe headaches about it, but basically sorry this was a bit too long. But, like again, general point is, I really think we can't treat this index as an afterthought. We need to have standardized behavior across all clients, and agreed upon and agreed upon how we handle these concerns.

Jochem Brouwer: Yeah, I would also like to chime in on exactly like this index. What I would personally like to see is that we encode this coda sites into the state tree, and I know this is like a very big thing, because that would alter the state tree. But if we would launch ethereum. Now then, likely what we will do is we will put the code size into the account rop, which we would save into the state tree, so that what we will do is we now have 4 items in the account that will be the nonce, the balance, the code hash, and the state, the statehood and what we will then do. If, if, for instance, we will relaunch ethereum, then we would add this extra item of coda size. And I notice that we yeah, we can't. We like, you can't really add something to the state tree now. But that would be the ideal and the most simple solution, I would say to this problem.

and this is also an idea for my gujang. By the way. so thanks for that idea. Yeah.

Ameziane Hamlat: Sorry, quick question we we already found some issues related to export hash, export size, and we are we fixed few of them, and we are working on on other issues related to code. Cache in general would like to know if, when we push like these changes on the performance branch. How we can trigger the like the same tests. Can I just ping you and then run them again.

Jochem Brouwer: You mean as like a state test.

Ameziane Hamlat: I mean the ones that you just shared related to Yes.

Jochem Brouwer: Okay, yes, yes, I should add these tests to the to the fixtures, I think, to the state tests. Would that work.

Ameziane Hamlat: I'm not sure, because my question in the chat was, How do? How do we collect those results? And Marius said that those results are from Perfnet, which is a Shadow Fork of Mainnet. From what I understood.

Jochem Brouwer: Yes.

Ameziane Hamlat: And that's perfect. That's that's actually perfect, because like the the notes are are warmed up. So that that's perfect for Besu. and would like actually to get the results again when we have the the fixes, like the performance fixes.

Jochem Brouwer: Okay, yeah, that's actually a very good point. Because also one of the reasons why we did this on a shadow fork is because we want this on like a very large database. So, for instance, like a fork of Mainnet, because if you will just do the state test, then your database size is well, not large enough, and we would really want to test this like on a big database like Mainnet. For that. I think this is something for Panda Ops because they run these shadow forks, and then we likely have to update your client to a new version, and then we can run it again and see what the results are.

Ameziane Hamlat: Okay. Thanks.

Jochem Brouwer: Yes, thank you. I think it's also.

Mario Vega: Point is that sorry.

Jochem Brouwer: And there's also ping me on discord if you need something. Yeah, thanks.

Mario Vega: Thank you. Johan, if you could summarize somewhere the list of tests that were run in the Perfnet, that would be nice, because we can focus on either improving those or adding them to the to the releases to even if they are not as significant as when running. Imperfnet. I think it would be helpful to know which ones are. The ones are being run.

Jochem Brouwer: Yeah, I will add this to the to the east. Yeah. Good point.

Mario Vega: Thank you. Oh, alright! Any comments from the clients, or any anything that anyone else that wants to chime in on the results or the I think I don't think we we touch on the spec side. I think one question would be, Johan, does this affect the current specification or do this is this effect? Does it require changes to the eip? in any way that you feel that need to be done?

Jochem Brouwer: That's a good question for that. I'm not really sure. I think that. Yeah, from the testing thing, there are only some points which might be considered unspecified. I think those should be specified. And yes, but for now I don't really have some some big things here to to add, yeah, these are just yeah, benchmark results. Yeah.

Mario Vega: Okay, thank you. From the perspective. I don't think we're changing the the current test. I I it would be 2 last minute. So I think we can wait until tomorrow when definitely launches, and then, if it results that we need to change something to the eip, it would be it would it would need to go into that 3, in my opinion. alright and one last question from answer. How many yield clients looked into the code size index at all. So far. are there any comments on any clients? Is this something that everyone has considered? Or what's the status of the rest of the clients.

if not there's 1 comment from bargain Mouse, that technically should only have repricing of eaps. Yeah, we should make sense.

Roman: Well, well, we we agreed on acd. That 7, 9 0. 7 from light client will go in as well.

Ben Adams: That was the tap on tap on contract size.

Roman: Yeah.

Mario Vega: Hey, yeah. And I just wanted to show me to say that if there's any changes that need to be done because we found significant performance. Well, next is that is that a question of whether we continue with the current state of the eip, or does it need any modification. I think we we still have to discuss this. But yeah, I'm not sure about the the answer to that. To be honest. Yeah. any other comments, about 79 0 7, I I still I still see a lot of shatter in the our messages in the chat.

Jochem Brouwer: Yeah, maybe just to chime in. I think that for Defnet 2, the current state of the Erp is the one which is scheduled for Defnet 2. There is an unmatched Pr, which is what Lucas just posted. It's 9, 1, 9, 9, 1 0. That's like clients. Pr, I think that one was scheduled for Defnet 3. But for definite 2. I think the current spec, and maybe also the fixtures which got released. I think that is the 256 kB limit of the Eap.

Mario Vega: Yes, that is correct. We we have 256 kilobits. So the the fixtures right now in the tests are for the current version that is merged into master of the eip.

Łukasz Rozmej: Yeah, I complained about it like 3 times already, and it doesn't make sense to to go with that fashion and and change. But it was added already, and you know that then this is a roadblock, and we cannot change it. It's my opinion this is worthless information, including it in the 2, 5, 6 version.

Jochem Brouwer: Okay. So I then open up Pr, which only changes this Max code size to 40 kB and then use that specification for def net 2.

Mario Vega: I think, in my opinion, it's too late to change anything for them to.

Csaba Kiraly: Huh! No instant.

Mario Vega: Yeah, I think I think letting it as is for them, too, I think, is is the the only thing that we can go do now and get more information once that definitely 2 is is out. Yep. Also, I think the Pr is for Matt of the eap. I think it would be nice if you can comment on it. But I think he's not in the call. Yep, yeah, okay. Sounds good. Alright. So I I think the the final point is that we we continue as this 4 22. So 256 kilowatts. and the the current state of the master vip that is going to merge, and when we then we reassess either in acde or i async, for the final changes to the eap.

Does that sound good by everyone, or any comments or counterpoints? Cool? Alright? So that's it. And we only have 2 min left. Are there any other comments or quick. final thoughts that we should do before closing the call? If not, thank you. Everyone for joining, and we see you on Acdc. On Thursday. Thank you.

Łukasz Rozmej: Bye.

Jochem Brouwer: Thank you.

Ansgar Dietrichs: What's everyone?

Transcript adapted from ethereum/pm, CC BY-SA 3.0.