Nethermind roadmap and Ethereum benchmarking - Marek Moraczyński & Kamil Chodoła - Nethermind

Transcript

Hello, I'm from Netherm team. I will give you a brief [music] 5 minutes overview about Netherm client and then Camil also from Netherm team will share more about Ethereum benchmarking. Given the limited time, let me start uh quickly with what we've been working on. So firstly, Ethereum upgrades have always been and will continue to be one of our top prior priorities. Fusaka, we are just monitoring and waiting for it. Uh it will go live next month on mainet.

And just a reminder, Fusaka will allow to scale L2 signific significantly. Glamsterdam uh with the main feature block level access list and this feature adds storage and state to blocks and thanks to that ELS can process blocks faster and in parallel we already have a multiclient devet and nethermite is there of course and somewhat related to glamsterdam is gas pricing many have been proposed for glamsterdam in this area and the thing is that um current EVM gas pricing not always reflect real resource usage. So just as an example modex

pre-ompile is much slower than other operations uh other op codes and other uh pre-ompiles. So it means that if we execute something with the same gas uh it will take much longer. So proper repricing is critical for further scaling layer 1 and also critical for uh ZKVMs. [sighs] It's also worth mentioning that nethermind client is not only limited to ethereum mainet but we are actively contributing to broader ecosystem.

Nethermind is key component of search based on tao stack and together with tao and now ens we are developing the future of based rollups. We will be pushing performance limits with the nemite clientbased sequencer on arbitum chains. We also support other very important EVM chains such as Nosis chain, optack chains, XDC and now um we all know ZK is the future. So that's why we are working on ZK mind but it's not easy. uh we adapted the client and the main challenge is compiling net to risk v and we managed

to do that for riskv 64 and now I hope that we are very close to proving first blocks uh with zisk which will be huge technical milestone for us and after that we plan to compile net to risk v32 and proving blocks with risk zero and sp1 and a few other things we are working on. So stability and testing first. I know that it sounds really boring but this is something very important for us.

Uh and we want to making running your notes extremely boring and we have fairly uh sophisticated release process but there is still room for improvement here. Before our talk, Sebastian from Nosis was talking about the importance of RPC layer and how it is connected with Ethereum cyber punk vision. So that's why we are adjusting many RPC endpoints for better compat compatibility with GF. Also after Fusaka we plan bigger release with a huge performance improvements in get locks. Uh we worked on a new index here. Um and on data front we also

researching a new ways to do archive node but here we are pretty pretty early and on your notes there is still 500 gigs of history that we can remove with era files and you can expect the progress in this area as well and last but not least performance. So performance is good but can can be even better. Uh so every release we are adding a lot of performance improvements that accumulate over time. Uh we are working on block STM uh which will help us with parallel transaction execution.

uh we are also like exploring different approaches to uh state database including uh developing our own state state database and in terms of performance we are also working on something else uh but not only for our client but for Ethereum as a whole uh we are trying to identify bottlenecks uh we are benchmarking bunch of other clients and Camille will tell more about it. >> Thank you. So the phrase performance is good started as a small joke inside client. So it comes from the comment under one of the pull requests started

by Ben uh made by Ben and it somehow got caught up by other cordeps in the space. So uh we are spending we spent years improving mind client and making it exceptionally fast execution client but at some point we realized that this is not enough. Uh making just one client performant will not be sufficient for the entire network. So we started to work on several tools to make it faster. Once those tools uh matured enough they started benefiting everyone not just mind. And this is when our focus evolved from making Nethermind fast to making Ethereum fast.

So uh let's talk a bit about how it fits the Ethereum scalability story. So uh once we nailed down all of the determined performance improvements, we started asking ourselves how we can make uh entire networks faster, not just our node. This is when our f second focus emerged making specialized dev dev nets and uh benchmarking frameworks which makes everything uh easier to test measure on every single client and making it easier to simply catch backs uh catch performance issues compare perform compare performance across the clients and see how we are different different on the worst cases.

So let's start with the first thing. So let's start let's talk about uh networks which we created starting with perfet. PerfNet is a specialized networks focused on compute heavy scenarios. It's designed on top of mainet state and its main focus is to check how the execution and consensus clients are behaving under very heavy transaction load. It's like our way of asking what would happen on mainet if the load will start to suddenly increase very heavily. Then uh we took the things to the another level with a bloat net. a successor to perfet.

Uh it started as a joint cooperation between Netmind and Ethereum Foundation later was fully owned by Carlos Perez from EF who is doing an exceptional job out there and its main focus is to analyze the state growth and how clients will behave under very heavy state expansion with very heavy uh attacks with state extensive transactions to verify the syncing issues scalability issues and catch them early before they will ever reach out the reach out mainet. Both of those networks are giving us something which we actually never had before. So reproducible reliable uh in performance focused networks which are giving us a space to test and ensure the not only performance but also correctness of our

improvements. So what we have learned from these networks the simple answer is a lot perfet and bloet reveal the performance flaws which are actually not able to be noticed on current mainet in these days. So let's go through a few the most of the most important findings. So set code 7702 transactions starting to degradate very significantly over over a time when we started increasing the load. So this kind of testing help execution clients to improve very significantly even in some cases up to 20 times.

We also noticed that whenever the blob amounts in blob transaction pool start to increase very heavily, some of the clients started to struggle to produce blocks on time. So this kind of feedback helps us to prepare better for for example upcoming forks with Fusaka where the where the blob amounts will start to increase very drastically. From the bloat net perspective, um we had a test we were testing snapsync and we noticed that uh snap healing part of snapsync was able to stall whenever the high throughput of state extensive transactions started to hit hit the note hardly and the gas limit was heavily increased. But most importantly, pertinent actually revealed a consensus issue uh when whenever one of the client

which was triggered by one of the clients improvements and we caught it very very early before it was ever pushed to the production to to the mainet. So we we this show this highlights the importance of such a specialized networks to not have a very critical issues on mainet just have them uh cached on on specific devets for the performance. So let's now switch from networks to tooling starting with gas benchmarks. The the reason we built that tool is was to make something which will make performance testing something very repeatable and reliable. It's not just a bunch of scripts. It's like a full suit which allows you to compare all of the clients uh under the exact same conditions uh with very sens sensit

sensitive uh scenarios which might affect their performance. We work also with Ethereum Foundation on standardization of benchmarks through the ILS project. And huge shout out here to steel team to stateless team and to a panda ops team for making those uh initiatives possible. And here the idea is to align all of the clients under one unified execution layer specification for benchmarks uh ensuring the same data benchmarks and assumptions. So here is one of the examples of how the gas benchmarks are working. So this example is based on a catchak hashing mechanism. So the block is basically filled to the maximum with ketchup opose usages. And going one by one on the top

left we have a all of the clients compared with and scaled to the megagas per second with net leading the pack with 70 meagas per second. Then we have gref, argon, bzon and eph. On the top right we have our ILY indicator panel which is telling us that there there might be some scenarios which are starting to underperform. So 60 megs per second is our current threshold but actually the the problematic scenarios are starting below 20 megastas. So we are still quite above that threshold on that on that matter. On the bottom left we have a comparison over on under certain time range where client can actually verify if they are improving their performance or they are having

some kind of regressions and they can notice that on this panel and on the bottom right we have comparison of the slowest versus fastest. So in this example Ephrex versusmind but the difference here is quite small because it's just two times. We had a scenarios where the difference is actually five 10 times and this is like a information for those slower clients where to reach out for help maybe where to reach out for discussion about some potential improvements in their codebase in in their flows for those specific uh those specific scenarios. So this tool is not just about making uh charts look nicer. They they this tool is having real protocol impact. So let's talk about the major wins from using that tool. So gas benchmarks is like

crucial tool right now to decide on gas limits which cords will suggest moving forward. So before 36 to 45 million gas we use gas benchmarks to execute a lot of scenarios and verify if we are having any underperformance in any client which will be risky to increase to 45. uh later this year we had a Berlin interop where all of the cordiffs met together and started analyzing the data making very specific improvements to make this upgrade happen and so right now this is the current gas limit 45 and we have another step so from 45 to 60 million which is right now actually in progress we already agreed on that clients already released the versions with 60 million but this time for for this step

we also repeated the same procedure but we had ills tests and this is like increasing our amount of scenarios significantly and also we had a stateful test which are which were proving some paths which were never before uh properly benchmarked. So this time with much better subset of tests but with the same datadriven approach we increase the gas uh we are in progress of increasing gas limit to 60. It probably will happen just before Fusaka or maybe during Fusaka or slightly after but just in those time ranges. And what's next? So what about what will happen after 60? So we don't want to stop there. We want to go further. We are building our test cases to be better. We are building much more scenarios to both compute ones and

stateful ones. And uh right now we are considering how to increase the gas limit to 100 probably with some some intermediate step on 80 mega gas. uh after Fusaka hard fork which introduced for example one of the repricings mentioned by Marik about mod XP which is right now one of the worst cases. So this was gas benchmarks and let's now uh talk a little bit about different tool which we've built uh which is taking the benchmarking a little bit step clo a little small step closer to the mainet. These are those are mainet benchmarks which are focused on reproducing the real mainet load on top of mainet database. So right now if you

want to compare two versions of your client you need to spin two nodes which will be having identical uh hardware and just compare them on the tip of the chain you basically need to run it for a few days and analyze the data. This tool is having uh two modes which are allowing you do to do that much faster. First mode is using real mainet databases on certain height and few quite a lot of blocks from mainet moving one after another but we are executing them without any slot time in between. So rather than waiting few days for the same range of blocks we can execute them in a matter of minutes or maybe hour. Uh second mode is actually a little bit more interesting because we are having a capability to merge multiple blocks

together and craft much much bigger blocks than we have right now on mainet and stress test our implementations on the ex execution layer side to catch some unexpected issues which might happen in future. So I have a chart from the second mode which is compressed mode to draw you some picture on what we are testing here. It was 10,000 blocks from made it and from the certain height merged into 100 blocks with average size of 1.1 gagas uh per block and those are real mainet nodes real mainet transactions ordered in a in a in the order what when how they will happening on the mainet and let's to take a look at the results. So right now is a little

bit spiky but the average is on a about 800 megagas per second. Then we have bezel which is on 277 megastas per second. Then is g 210 meagas per second and then we have ref on 73 megs per second. So for ref uh we see that it started pretty high then started degradating over time. So we approached them to to discuss this issue. they were very grateful for for showing that that issue to them and uh they started to fix it straight away. And the last thing which we are involved into is op codes pre up codes and pre-ompiles repricing. As Marik mentioned, we are focusing a lot right now on not only scaling clients or not only giving them a tools to verify how

performant are them, but also about uh scaling entire interum through very very sensitive very specific repricings. So right now one of our tools gas benchmarks is be being adjusted to be able to verify the prices and performance before the repricing after repricing compare the data analyze the data and choose the proper values for each op code. Very likely some of the repricings which will happen on glamsterdam and much bigger repricing will happen on the following fork and that would be it from us. So as you can see mind is focusing on plenty of areas both functional and performance sides. So this is what we are doing right now. Thank you.

[applause] All right, we do have uh time for a few questions from the audience here. Um and so if I could ask you to just step in the center stage there for uh better framing for the folks viewing this on stream here. Um first question is do you have any benchmarking cases for very high gas transactions like 7702 gas sponsored transactions? >> Our cases right now are focused on filling the block to their maximum not only about like very small transactions which are from mainet. We are basically on one tool crafting very specific transactions which are going way beyond

100 megagas or even higher filling them specifically with just one transa one op code type. So we are testing all of the possible scenarios on the EVM level right now. >> Okay. And then the question the person that wrote the question also had a comment I wish you uh could also comment on and give your opinion. Uh they said that parallelization buys a lot of performance headroom but it's best with small transactions. Has that been your observation as well too? >> Uh parallelization is very very important for sure. Uh it gives a lot of headroom for performance. Uh so that's why uh we are testing right now on our benchmarks not on multiple transactions to make it specifically nonp parallelizable right now to test those

actually worst case scenarios to see how the clients perform under the worst possible conditions. >> Okay. Uh next question then. Can others contribute to the gas benchmarks? Is there an API uh in light of some uh bench scenarios that could benefit one client from another client? >> Definitely. So, as I mentioned, we are working in EF uh with steel team on standardization through the ILS project. So, all of the definitions from the tests which we are using right now are living out there and anybody can could simply post a PR with new scenarios just push it as a blockchain X scenario and this would this is what we use for a baseline for our benchmarks.

Okay. And then uh does Nethermine produce any tools for uh measuring the performance of block building? >> So block building is pretty similar process to block processing. Under the hood, it's more or less the same thing. Uh and so in that context, yes, but no, we don't have like specifically uh tooling right now for blog building. Just to add one comment to that is that we are just last week added one uh PR to execution APIs with completely new name space for the JSON RPC which will be used specifically for testing and triggering only the block building logic out there. So it could be theoretically used for that.

>> Okay. And we'll sneak in one last question here because we are at time. Um, how do you address concerns with custom compiler modifications to compile nethermine RV32M or other bare metal targets of ZKMS ZKVMs? >> So, for this you would have to ask Maxim who is not there. Sorry. >> Excellent. Well, thank you both for just a great presentation here. Another round of applause, please. [applause]