All Core Devs - Testing #57

Transcript

Mario Vega: Good morning.

Sebastian Bürgel: Ayo!

Mario Vega: We'll just give another minute until people join, and we can start.

I think we can get started.

Pooja Ranjan: We are live.

Mario Vega: Thank you so much. Welcome, everybody. This is ACDT number 57, today is October 13th. And we have a light agenda today, so basically we'll go over, some Fusaka, status, then we'll pass on to discuss the gas limit. updates, and then, Glamsterdam. And we also have another topic that we want to discuss in this meeting about the RPC testing failures. So yeah, let's just get started. Alright, on Fusaka, regarding the BPO fork updates, do we have any… Incidents, updates, or status that we want to bring up?

Barnabas Busa: Not really. We have the next BPO scheduled for tonight. I think it's in a couple of hours, Yeah, and we have Sapoya tomorrow. Yeah, it's in 7 hours, for the next BPO on Holsky. be duty, Bit of a drop in participation, but it's minor at this point. We expect a smooth sailing for tomorrow's park, and And the fork is on Sepolia is in 17 and a half hours. Last week, we did a Sapolia shadow fork, and everything was okay there.

Mario Vega: Alright, yeah, sounds like we're moving along. Any… Insights from the client teams. Any concerns regarding either… sorry, regarding either Holeshki or Sepolia? Happening tomorrow? Alright, seems like not.

Bharath Vedartham: Yeah, I have one. Sorry, sorry, I missed you. I… one… one thing I want to, like, inform all the client teams is, we have a new release of MevBoost, V1.10 Alpha 6. The difference is that, at this point. we've chosen to add, support to fall back to the V1 Blinded Blocks API if the V2 Blinded Blocks API is not implemented by the relay. We initially chose not to do that because we wanted MevBoost just to be a proxy. But I think on further discussions and thought, we just saw this as a much better idea. So, yeah, so MepBoost now supports, like, falling back to the V1 API, if the V2 API is not supported, but for…

For whatever reason by the relays. We've been checking with the major relays, they have added it, they should be ready. For Sepolia, the flashbots ideally should be ready with the V2 API. So, yeah, this has been tested on DevNet 3, we've, like, tested by disabling, like, the V2 API on DevNet 3, and I generally think seem fine with the relays, with the clients. So, yeah, just wanted to, like, inform all the client teams about it, and all the support… please, like, do update all the Sepolia validators to use this release of MavBoost. I have put that in the Sepolia Validators Telegram channel, too, with all the information about the release. So yeah. Yeah, I just wanted to mention that.

Mario Vega: Perfect, thank you. Just one question out of curiosity, how do you detect that B2 is not supported?

Bharath Vedartham: Yeah, for now, we've just, like, have a situation where if the return status code is greater than just, like, 4XX, like, we fall back to the V1 API, right? So, usually, if the API is not there, we get a HTTP status, like, 4.4 not found, and we fall back to the V1, right? But just for safety, we… what we just did was, if the HTTP status code is, like, greater than 4XX, we fall back to, like, the V1 API. So… but, like, the relay itself is, like, booked, like, then, like, even the V1 will fail, and there'll be a miss slot, so that's… that we can't do anything about.

Mario Vega: Alright. Yeah, thanks, thanks for the context. Anybody, Any opinion on this, from client teams? Are we okay?

Bharath Vedartham: Yeah, so one thing is, like, client teams can choose… so, I know some client teams have fall… like, Lighthouse have supported, like, fallback, and some client teams are calling the V2 API from Fulu. Generally, like, client teams can, with this new change, client teams can ideally just stick to however, like, they're calling the V2 API. Like, ideally, like, MevBoost will handle all the fallback, like. So, yeah.

Mario Vega: Excellent. Thanks. Alright, and also, there was one comment from Prism, they have a new release out, 6.1.2, And the blog has been updated. But also, Barnabas shared a list of the versions that we have in the blog post, so please take a look. If the version that you have here is outdated, please reach out so we can update it. And one another comment is, Lighthouse, be so organizing, do we expect a stable release before Hoodie? Anybody from Lighthouse, Vesu, or Grandin in the call?

Ameziane Hamlat: Yeah, so, as I said, for, for Bessru, We are going to have a new release for, for Hodi and, and mainnet related to, to 60 million gas. Yeah, so, the release is expected to be, out this week. And we'll share, we'll share it as soon as we have it.

Mario Vega: Got it. Thank you. Anyone from Lighthouse? Anyone from Grandin who wants to make a comment? Are we do… are we good there? Alright, I think we can follow async, if not. Okay, any other updates regarding, Holeshki or Sepolia that we want to bring up? If not, I think we can move to the next topic. Yeah, so the next topic is regarding… this message on Discord via Barnabas.

And basically, it's the idea that we want to remove the named forks from the BPO schedule. If the values are not affected. But yeah, Barnabas, do you want to keep more context to this, and just begin the discussion?

Barnabas Busa: Yes, so the idea is that we will be dropping the named folks from the blob schedule in the genesys.json file. We have already done this for Bezu and Nethermind, and Breath and Eragon also seems to be fond of it, and right now the only question is the Nimbus EL and, Geth. So the… the few PRs that are open are to remove Osaka, and also in the future, we just don't plan to include Amsterdam in the block schedule either.

Mario Vega: And just for clarification, this is only if the name fork does not update the values, right? If it doesn't…

Barnabas Busa: The name, the name fork cannot update the values anymore.

Bumblefudge: on the CR side.

Mario Vega: I sucked.

Barnabas Busa: conclude.

Mario Vega: I see.

Barnabas Busa: And that's why I'm proposing to remove this, because this is a redundant tag at this point.

Mario Vega: Makes sense. Any comments on this? Should we move on with this? I think it makes sense. On the east side, I think we have to make a new release to make sure that we are not including the name of Forks, but I think it should be an easy change, and we can have a release in the following days. On the client side, is there any concerns regarding this update? And one comment from Barnabas Marius. Is this okay to you? I think that you're only the one, missing to comment on this.

Marius van der Wijden: Yes, we have not implemented it in guest yet, but, yeah. This is fine.

Mario Vega: Alright. Then I think what we'll do on the east side is also we're gonna make a new release, just to make sure that we can run the clients with static tests, and then we can move on to… Blessing granted. Yeah, and we can move on to updating the notes on the dev notes, I guess. Alright. Barnavas, any other comments, or is this okay?

Barnabas Busa: Yeah, that's okay.

Mario Vega: Thanks.

Bumblefudge: Alright, I think that's…

Mario Vega: That was everything for Fusaka. And I guess we can move on to the next, topic, which is the gas limit testing that we've been doing. I think Camille is in the call, or also anyone else who wants to chime in, but Camille, do you want to begin the discussion? On the updates on this topic.

Kamil Chodoła: Sure. So, as you heard already from Bezo and other clients, everyone is kind of ready for 60 million, for mainnet releases. And in the meantime, we are focusing on stageful testing, which is the biggest, maybe not unknown, but the hardest thing to test right now. But I think that, actually today I have a good milestone met, is that I'm running free EL clients on top of BloatNet in a reproducible way, so in the same way as gas benchmarks are doing. So, for now, we're running Nethermine, Bezu, and GEF. having still issues with, Aragon and Ref, so waiting for some help here on State Blood Research Channel. So I can make all 5 major EL clients being tested this way.

And Joel had prepared some scenarios, actually forks and scenarios, and for now, we do not see any huge bottleneck. And those scenarios are not for sure the most performant ones out there, and for now, Netarmine is having the biggest struggles, but not, like, consistently. So, just to give you an example, once some of the extent scenarios are performing at more or less 250 megabits per second. And sometimes they are dropping significantly to something like 35. Which is still relatively good, and not putting any risk on 60 medium gas increase for now. But I believe that we need to still spend much more time investigating that, because it doesn't align really well with what we've observed on Mainnet.

So, there's still some work to be done out there. But anyways, dashboard is set, 3 clients are running right now, I'm just doing some final fine-tunes, and we'll… the only thing which is left is work on more stateful scenarios on top of mainnet. So, at least the entire mechanism there is running.

Mario Vega: Thank you. Yeah, great work on this, Any comments from clients on this? Do you have, like, a theory on why are we not seeing, like, the same… benchmarks that we are… that we're seeing on mainnet.

Kamil Chodoła: So, I have a few theories for now. One will be that we are preparing some sense preparation blocks, then trying to spam with different contracts to fill the caches, and later on we are doing the actual attack. And one client might cache more while the other is caching less, and for some reason, the actual send path is warmer than expected, and maybe that's one of the things, but it's pretty hard to say, and trying plenty of things, like including restarts, right, without the attack. Or, doing… few thousand of the spamming blocks, which are doing completely unrelated work, and then attacking XAM, but it's pretty time-consuming, and taking some time to… to narrow it down together with Johan.

plus the hardware. I mean, we are using hardware which is below the recommendations, but I need to compare with the notes which were reporting pretty slow times on Xen to see if this is somewhat in common. So, suspecting that maybe my hardware is a little bit better than from the notes which were reported to quite slow times. And maybe we still need to spend more time to see if we can do something worse using some contract.

Mario Vega: Got it, thank you. Makes a lot of sense. Any comments from other teams on this? Amazon? Yep.

Ameziane Hamlat: Yeah, I have a question. Are those con… are those blocks, like, exam blocks, part of the… performance DevNet 2, like, they are really important as canonical blocks on that DevNet.

Kamil Chodoła: The ones which I'm using for testing.

Ameziane Hamlat: Yes.

Kamil Chodoła: No, these are crafted by us. So, we use the same contract, like the real one. The baseline is BloatNet, not mainnet, so we use 2x8, but we craft all of the preparation blocks, and then the actual attack block.

Ameziane Hamlat: Okay. Yeah, I was asking just to, to see if we can help, you know, in, in profiling those blocks. From our side as well, to see if there is anything to… to find from the profiling.

Kamil Chodoła: I can work with you to maybe hook up some profiling tools to see how… how… what we can get from there. For example, trying right now to see if I can hook profiling tools from Nethermind to see how it will be doing. Because it's not that easy just to take that block from mainnet and just recall it once more. Because actual send needs a few blocks to prep… to prepare, some hacking around timestamps of the blocks, because some of the functions from SEND are being able to be called after, right, 24 hours. So, once some block will be called on mainnet, you can just recall some functions, and the result will be the same. We just need to do quite a few of the preparation steps to call it.

More details are from Jochen, probably, here.

Ameziane Hamlat: I see. Yeah, we can… we can do it, I think. Thanks.

Mario Vega: Thank you. Any other comments? There was one comment in the chat that the… Same content is already bigger in Bloodnet, so, I think… We are on a good path to trying to reproduce this thing. The same as mainnet.

Kamil Chodoła: Very important note for me to EL clients will be, like, if you see any interesting stateful scenarios which we can implement right now, because actually what is left right now is implementation, so you can just reach out to us and tell what bottlenecks you see in your clients, and what interesting things we can do to to attack some specific clients in a worse way, which might not be that bad for the other clients to try to stress that as much as possible. And also, the warming up part for me is a little bit more tricky than on normal compute scenarios, because here. what gets warmed is not only EVM, but then we have also loading of the data from disk to memory, and then also there is some kind of hardware warm-up, which affects the results pretty much.

So… yeah, there are quite a few things which need to be considered comparing two compute scenarios where just simple block with invalid state would give you improper warm-up, like… Which would… which would be good enough. So here we have a few levels of issues, which we need to solve, so if you see any… Possible solutions, how to tackle that, just let me know.

Mario Vega: Thank you. Yes, so please reach out to Camille, I'm guessing R&D Discord is the best way to approach. Yeah. We'll keep an eye on the… on the scenarios that we can implement. Alright. If nothing else, we can move on to Glansterdam. Oh, sorry, Marcin, there was, please, go ahead.

Marcin Sobczak: Hi, I have one more thing which has sense to discuss before going to Glamsterdam. We are doing analysis of Modex usages. Like, as a peak introduction, we are… we are doing traces of blocks under… PECTRA consensus rules and Fusaka Consensus rules. We are artificially increasing gas limit of transactions before purchasing to… because, like, modex gas prices increased, and, We think that everyone will take it into account before sending transactions… transactions post hard fork. So, with these assumptions, we are looking for differences, basing on, debug traces. And we found a few hundred transactions which are failing.

after Fusaka, but it's, all of them are, have one pattern, the same pattern, and the same, error message. It's, approximately one transaction per hour, which is failing because of that. And I pasted that, error message is from account abstractions. And all these transactions are interacting with the same contract. It's entry point 0.6.0. I pasted one transaction as an example. And We suspect that it is because we are increasing the general gas limit of transaction, but inside of account abstraction transactions, there are user operations. which we suspect that after hard fork, we have new estimates with higher gas limits, and then it shouldn't fail, but it's, like, we suspect this. So, we would like to confirm that this is the issue.

And generally, share the progress, from this, area. So, Yeah, it is our finding, and we need to confirm that it will be fine after Fusaka. We suspect that it will be, but we are not sure at this point about it.

Mario Vega: Okay, do you know, Marcin, if there's already a reference test for this? Scenario, or similar? If not, we can investigate that for you, but if you know…

Marcin Sobczak: I don't think so, I don't know.

Mario Vega: Alright. Yeah, I think… I think we need to investigate whether this… there's a reference, test implementation for this. Or something similar. Anyway… Any comments on this? Now that we can move to Amsterdam. Alright, thanks, Dr. Martin, for the, for the comment. We can start with, block-level access list, I see… Who's in the call? Felipe? Whilst… And Rahul, do you guys want to kick off the discussion?

Felipe Selmo: Yeah, I can mention, Over the weekend, the… there were some of the test vectors from our release last week. That were flagged for a couple changes needed. But those have already been fixed and merged as of this morning. So, I'm gonna triple check… If there's anything missing. And try to get a release out early this week. For… for block-level access lists.

Mario Vega: Thank you. And also, Raul shared the results of the tests that are running and passing on each of the clients. On the DevNet side of this, Barnabas, do we want to wait for 100% passing of the tests, and then schedule The block level access list definite. Or what would be the next steps?

Barnabas Busa: The BL testnets are taking over by Stefan. Maybe Stefan can comment?

Stefan Starflinger: They're still working on, like, revamping their implementation. There seem to be some improvements that they want to make. So I think we should have at least 3 clients working pretty well before we start the definite. And I think, to your question, I think it would be good to have as many tests passing as possible before we start the DevNet, of course.

Mario Vega: Absolutely. Alright, so I think the next steps on this is trying to get this… The number of dust passing to at least I would say 100%, before continuing. Do we know the nature of the tests that are failing, Rahul? Like, are these consensus tests, or are these, like. Other kind of tests. That are failing. Raul or Felipe?

Felipe Selmo: the… for the DevNet tests?

Mario Vega: For the spec tests.

Felipe Selmo: For the spec tests, the ones that were flagged were, out of gas issues that… On the spec side, we had made some changes that needed to… Be returned. The specs got updated over the weekend, and the tests got updated over the weekend. So they were generating, some factors that were… Not correct. And then we weren't… Properly validating on the, test side? And so, I made sure that we could… we were using the right markers there, so that we could catch it.

Mario Vega: Yeah, and on the client side, do you know what's the… the reason why they're failing? I mean, rather than…

Felipe Selmo: The vectors that were being generated were not correct.

Mario Vega: Oh, I see, I see.

Felipe Selmo: But for the tests that I looked at, I'm not sure if this is the same as the ones being run on the DevNet.

Mario Vega: Thanks. Rahul, do we have, like, a link where we can see the results for each client on this? I think I got it here. Thank you. I already shared. Great. Thanks. I think, please take a look, clients, on this. To see if there's any outstanding tasks that are failing on your side that we need to… that you guys need to fix. And please reach out to Felipe and Rahul also, if there's… Any unknowns on the tests that are currently running, so we can get this to 100%, and then we can schedule the… Local access to this Devna.

All right, any other comments or topics on block-level access list? If not, we can move on to, EPBS side of things. Alright, who's, the person.

Justin Traglia: I can try. Alright. I mean, not much of an update, just the client teams are still working on DevNet Zero implementations. It appears that DevNet Zero by the end of October is unlikely, just in my opinion. Progress has slowed a bit by, like, the off… off-protocol payments discussion debate that's going on. I'm hoping that we can make a decision on this at the upcoming ACDC call. Small stuff, not a whole lot to talk about here. That's pretty much it.

Mario Vega: Alright, thank you. Okay, yeah, thanks for the updates on this. Anything else on Glaston that is not related to Blackload Access List or EPBS that we need to discuss today? Alright, if nothing else, we can jump onto the last topic for today, which is the RPC test failures. There's a blog post in the EATH research site. That was raised by, S.C. Burkel. Regarding failures on the RPC test suite that we have in Hive. I see Sebastian on the call right now, do you want to kick us off with this conversation?

Sebastian Bürgel: Sure, yeah. Thanks for supplement for having me here, Mario. I'm Sebastian, from Gnosis slash Hopper. And, yeah, I think we're all aware that Ethereum is trying to onboard a huge number of individual users. And, yeah, that's… in that regard, we're quite aligned with what we do at Gnosis to build products that actually use an Ethereum chain. Currently, Gnosis L1. And, in doing that, we kind of see, and that's why I'm here, to kind of sound the alarm bells, kind of a… increasing number of issues around, specifically, the Ethereum RPC layer. And, I mean, just to take for a second a step back, right? How do we get, actually, many people to use Ethereum? Well, by Ethereum-based apps, right? And these app developers, unfortunately, they don't touch the network layer, they don't touch the consensus layer.

And as a networking guy, I kind of hate to say that, but that is the reality. They only touch the Ethereum RPC layer. So if that one is kind of critically broken, then Ethereum is in a pretty rough position, and that is, as I outlined in more details in the post, the situation right now. So, I think we made pretty cool progress to get us into a better, space, so shout out, obviously, to ETH PandaOps for the Hive test suite and so on. But we need more than that, right? So if we look at the IF test right now, there's around 190 only on the entire Ethereum RPC layer, which is basically spinning up kind of a fake blockchain of only a few blocks length, and then do, like.

Like, 4 or 5 tests on them. for ETHAT logs, for example, specifically, which is the example that I picked here, but I really only mean it to serve as an example of just how dire the situation around the RPC layer is. And… so specifically. We see clients where, logs just get swallowed and omitted. So, you know, you as a user don't even know that you're missing state, which is a kind of catastrophic failure case for an Ethereum-based application that doesn't have any additional information. And, yeah, on the one hand, you could say, well, okay, that's a client bug, just let clients sort that out. But I find it more interesting, and that's why I posted this blog here.

Because in the process of trying to get to the bottom of that, a few things were sticking out. So the first one is, this was not caught by any hive tests. Because they kind of test for very basic things. Again, which is helpful, but we need to grow by many orders of magnitude beyond that. And the second is, Hive tests are failing. I think it is fair to say that client teams, in many ways, don't care too much if they fail. Because otherwise, they wouldn't have been over 50% red for several months at this point. And the third thing is, when engaging, like, I talked to several client teams, and the interesting thing was, there isn't even consensus. We do not have consensus on how Ethereum is exactly supposed to work.

Meaning, we're missing standardization, and yes, I'm aware that we have the ETH execution specs, which is… has been helpful in finding, kind of, tight mismatches and so on. But what I mean when I say that we don't have specs how Ethereum is exactly supposed to work, again, just as an example. from the ETHGetLogs front, it is currently unclear, and clients have in the past made individual decisions on what happens if somebody tries to, obtain locks, let's say ETH get locks, on the wrapped ETH contract from Genesis until tip of the chain, right? That's obviously labor-intensive, I guess, computationally expensive, and the response size might be huge. So individual clients made individual implementation choices, such as cropping the results, erroring, just taking a whole long time.

or building kind of bespoke indexing solutions on top of it, which has not undergone any VIP, and it's kind of individual innovation, which happens in parallel for every client team. Which is clearly kind of suboptimal. So… Yeah, basically, what I'm saying here is, from my perspective, looking at this, we are lacking two things. One is, in orders of magnitude, growth of what Hive is today. to kind of bring it up to… in parallel, I see it somewhat similar to how the web platform tests work. So the browsers that many of you are using right now has undergone over 2 million tests, right? So Ethereum is certainly not much less complex, so we need Multiple orders of magnitude growth in these test cases.

we need focus on the basic stuff, so if I look at what we discussed in this call. And if I look at the Hive test results, the cutting-edge stuff is kind of okay, right? So the Fusaka tests are kind of all okay. it is the basic, kind of, for example, RPC layer that is not okay. So, that is focus, and then I think we need additional focus on, well, getting to consensus of how Ethereum is actually supposed to work. Because when I do talk to client teams, I also hear, well, you know, Hive tests were basically just the test suite, so of course, NetherMind or, you know, any of the others is not one-to-one following that, and we have our reasons for that.

Which is fine, right? I think many of these reasons are fine, but it just shows me we need to go through a consensus-finding exercise of how Ethereum is actually supposed to work on that layer. And yeah, that's, that's what I see here right now. Yeah.

Mario Vega: Thank you.

Sebastian Bürgel: Happy to take it forward.

Mario Vega: Thank you, thank you. So yeah, a lot of hands up, we can start with Marius.

Marius van der Wijden: Sure, yeah, I think this is a… this is a problem that we know about for a really long time, and, we… We at the Ethereum Foundation have also started hiring, hiring someone for exactly this purpose. But there have already also been a few proposals, from, for example, Jules. On how to, how to, create a new… REST-based API, in order to, kind of, yeah, bring the RPC into the… 21st century, I would say. So basically. when the RPC was created, the… it was… it was basically created for something like Mist. And, the problem is that it…

It's very hard to evolve it, because it impacts so many people in the ecosystem. And, Yes, and also the standardization process is very… difficult, because you kind of… like, RPC is just a second, smaller cousin, to the normal consensus, and so client teams are not prioritizing it as much as the normal consensus. But as I said, I think… I personally think it's a big problem, and I know people have been, kind of. A lot of people have tried to make this a priority, and tried to make the standards, and have been burnt out doing that. for example, like, I don't know, I think ETH Simulate is one of the… one of the examples where it worked, where we found,

found a way to find consensus between clients and create a good spec, but a lot of the other things are not. I also want to go, like, a step further in this. I think it would be good, because, like, as we all know. nodes are not the same thing as they were 10 years ago. A lot of people are not running full nodes, or, like, archive nodes anymore. A lot of people are running with, with, history expiry. and in the future, what a node… Stores is going to be, very different, so we will see more specialized nodes. And I think the RPC Needs to reflect that, in order to make it clear that these are calls that happen on the head of the chain.

These are calls that require, access to the history, and so on. And the last point that I wanted to make regarding the log indexing. So old has, also written an EAP, About that, in order to get the… log indexing that we built internally in Geth, to get it into the protocol. The problem here, again, is prioritization of, of that, against other features, and I think that is, like, dev time is very finite, and, so, yeah. It's definitely an issue that is on top of mind for me, and, I hope to bring some resources from the Ethereum Foundation, in order to In order to get that going, but in the end, it's clients who, who kind of need to implement it, and, yeah.

I think client teams are stretched already.

Mario Vega: Thank you, Maurice. Lukash?

Łukasz Rozmej: Yeah, I have quite a few comments, so… One, I joined the, JSON RPC call, actually, recently, and the temperature check I got there that the… they're working on a very, very small and basic RPC spec, that won't handle everything mentioned here, most of the things mentioned here. And actually, compared… comparing that to the correct node-deep limitations, nodes would be mostly already in that spec. Maybe there would be a very small, changes in the nodes, and the bigger problems they are actually focusing on is the, like, RPC providers are, like, general EVM compatibility of a train. So, for example, some L2s that have very custom and very, Not standard things doing. So that's one thing. So, to make this extremely well standardized on each client level.

Yeah, that's probably what is the problem, to actually build the… the… money and motivation, because it's very boring work to build the whole spec, and then execute on it, and create tests for everything. Moreover, some things I disagree that should be the specs. So, for example, Marius mentioned the AT GetLogs index they built. We built a different one, we will be releasing it soon, but our is not provable, and… but our is faster, right? So, it has some upsides and downsides compared to other things, and I don't think everything should be standardized. I think some figs would be better to… Be left to implementation details, because, different nodes can then, have different advantages and disadvantages.

Another thing, a bit of problem with the Hive test is, for us, is that, they are basically generated through GEF, so GEF is the spec, and while we agree that something between 95% to 99% of what GEF does is the best, and even if we don't agree, we most often align with what GEF does. There are very rare exceptions that it would be good to actually define the spec and potentially change something would get us to be better. And this is especially sometimes problematic for us if we return… the last thing is about the tests themselves being very basic. So, for example, in some rare cases, we want to return additional fields, because, for example, some of our users or customers requested it.

But the test doesn't allow us to fail the test at that point, because the test says this should be the list of fields, and you cannot return anything else, which in JSON RPC, it would be fine to return something optional or something extra, but the test fails. So we have this kind of false, negative in terms of testing. And… okay, I had one more thing. But I think I lost it. Yeah, I know. So we have a list of failing Hive tests, and we actually are trying to, kick down some of them, and I'm seeing it, decreasing every week by a few. instances, so… this is something that Netherbyte is working on and is prioritizing, but…

It is not as high a priority as, for example, all the consensus level. changes. So, yeah, it is lacking, and it will be lacking, compared to the core protocol of Ethereum. And yeah, one last thing. ETH Simulate was actually pretty well tested, and while still we failed some tests there, we are working on it. Maybe there will be some changes to the test here and there, if we decide on that. I don't think we found that examples, we had one or two questions about if the chest should be changed, but we decided to go with what GEF did, because both versions were fine, but, yeah, this is, again, circling back that GEF is the specs at the moment, which is…

Sometimes not the best thing to have, and that's it for me.

Mario Vega: Thank you, Lukash. Felix?

Felix Lange: Yeah, I mean, I strong… I want to respond, first of all, by strongly disagreeing that Gethis is back here, so the… the, The official specs are the… YAML files, which are contained in the execution APIs repository. And when it comes to the tests, it's very critical to say that, yes, we generate the test vectors using Geth, but the tests are not the spec. The spec is the spec. And the SPAC is independent from… from the client. We… we use GATH because For us, it was the easiest to set it up like this, and also because for most calls. the, the, so the testing methodology that is used for RPC tests is that the, the, Some RPC requests are performed, and the interactions between the test and the client are recorded. And then, we replay these interactions against another client, and compare the result.

the comparison of the results has some special cases. For example, for the errors, we do not match on the error message, but we do expect the error code to be the same. And I feel like this is kind of critical, because for the users, the message does not matter as much as the code. So, but even most of the differences in the clients is in cases where, the… requests submitted by the test is containing something invalid. So basically, we are testing the, like. the, like, limits of the spec in the… in as few of the tests we have. So all of the tests where you are just, like, requesting something in the normal way, they are passing. It's mostly about… like, with some exceptions, of course. But then, for the… for the… for example, with NetherMite, most of the differences are in cases where, for example, the request performs… has an invalid argument value.

And then, we expect the client to react with a certain error code, or we expect the client to detect this invalid argument, and instead, it either accepts it because the parsing is too lenient, or it returns a different error code. And I feel like… These failures, in some way, are less important than if it returned, like, a totally wrong value. But, obviously, it would be good to align. And for these things, I feel it's just a matter of agreeing on the correct error code. It's not so much, like, there isn't gonna be a deep discussion. There are some other differences, specifically between Geth and Nethermind, when it comes to the optional fields. For example, things like the total difficulty, we discussed it also recently.

And I have to say that We have to… we still have to find a solution for that that works in the testing. I am not against, return… never mind returning the total difficulty, it's just that it's kind of hard to make it work with our current testing method, because the testing method is just to compare, the outputs, literally. For some calls, we do not compare the outputs. Literally, we have another method that is about checking, basically type-checking the response only, and for some responses, this can work. For example, for estimateGas. We do not require all clients to agree on the algorithm for estimating the gas usage of a transaction. The clients can actually each have their own method for that, so we only check that the result is a correctly encoded number, as is required by the spec.

So, It's, I feel like the discussion is quite nuanced. Like, I personally am the author of most of the existing RPC Compet tests, and I try to choose them very carefully to exercise the behavior of the clients as at the same time, I didn't want to introduce, like, hundreds of tests that would all fail, because when we got started, to be more serious about specifying the RPC with the execution APIs, and then also with the test suit. It became clear that is gonna be very demotivating for clients if they face, like, 300 failing tests, and they will just never get fixed. So we have been very careful about slowly expanding the range of behavior that we are testing. We started very small, tried to make a test suit that just contains, like.

50 or 60 tests that really, you know, are pretty much guaranteed to pass, and now we are at this point where, like, every so often, I will just add a couple more tests, just to see what's the behavioral difference in the client. And then we can, you know, try to get them fixed. I did report over time also many of the failures to the clients, but It just… at some point, also, I gave up on that, because it just seemed like it's not exactly a priority, and some of the things just turned out to be very hard to resolve, or something. That said, there are some constructive things we could do, so we could try to…

Improve the testing methodology by allowing, for example, certain optional fields for certain responses, or things like that. Or we could simply have multiple test vectors, which are allowable responses. The difference then is that we kind of have to manually check The responses to see if they are actually correct. And, this is not easy. So… I don't yet know what's gonna be the answer, definitely we need to make some improvements, but yeah, it's been an uphill battle, and I'm actually pretty happy that, for now, for the basic, calls, we have, quite some good coverage, also, of the edge cases. Forget logs, it would actually be good to add… introduce more tests, so Sebastian, if you have suggestions for that.

we have the repository where the tests are generated. It is very easy to actually add tests for this, and we could then very easily also verify the behavior. Maybe as a final note, some of the clients are currently not even… I guess for the RPC, all of the clients are somewhat able to load the test chain, but I see that, especially, for example, for RETH, we have a lot of failing tests, and… I don't actually really know… Yeah, I mean, we would really have to see, like, the client that's failing the most right now is Nimbus, actually, and I don't really fully know why, for example. Like, there are… some of these differences are, like, kind of hard to debug, so…

I guess we would have to… Yeah. Again, I can only say that, like, the clients should really attend the RPC standardization calls, and then we can see, what kind of… What kind of progress can be made on that?

Mario Vega: Thank you. Justin?

Justin Florentine: The next RPC call is in 6 minutes. So… get excited.

Mario Vega: Thank you. Alright, I see who had the… Lukash, do you want to go? We only have 5 minutes, though, but we can keep discussing this on the RPC call, so I think it's a great…

Łukasz Rozmej: Great thing that we…

Mario Vega: discussing this year.

Łukasz Rozmej: Fine, I added some things into the chat.

Mario Vega: Perfect, thanks. Felix, that was the hand up from the last time, or do you have something else to comment?

Felix Lange: Sorry, yeah, I didn't lower, I'm sorry.

Mario Vega: Alright, thanks. Yeah, I think it would be nice if someone can share the link to the next call. It's gonna be happening in 5 minutes, we can keep the discussion running in there. So I don't think we have reached an agreement at the moment, because it's a very nuanced discussion, but… Justin suggested that we bring this up into ACDE, which I don't think is a bad idea, and also. Yes, just to close… for now, I think everyone that is interested in this discussion, please join the next call. And also, please, join ACD, and we can, also follow up the discussion in there. Anything else from this scythe? I think also from the specs perspective, what I can say is that, we have, this moment… I think at this point.

we are reaching a point where we need a formal, a better specification model, perhaps. So, it might be worth to see if we can bring the ideas that we have put into yields on the consensus side of the specs into the RPC. Perhaps to make the, testing… language more expressive, and we can, even generate better tests with this. So, yeah, I think that discussion is still open. We can see if that would help in the long run. But yes, there's a ton of things to do. I don't think it's gonna be, like, very easy to get this going, but it's doable, definitely. All right, yeah, thanks, Justin, for the Zoom link to the next call.

Alright. Anything else? That can be discussed in this call. Yeah, and thanks, thanks, Sebastian, for bringing up the issue and starting the discussion. I think this is a really important topic, thanks for bringing it up, and also thanks for the very well-written post in ED research. It's been a great, resource to read, and yeah. Yeah. Thanks, everyone, for joining the call. If there's nothing else, we can close the discussion here.

Marius van der Wijden: Alright, thank you.

Felipe Selmo: Thanks, everyone. Bye.

Łukasz Rozmej: Bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.