Stress-Testing zkEVM: A Deep Dive into zkEVM Prover Testing Best Practices

Transcript

[Music] Hello. Um, so let's get started. I'm Leo from Linear Consensus and today's talk is about uh how to test uh GKEVM prover. So especially uh we are we have kind of a fra framework to test or stress test GKVM prover um internally. So um I would like to uh introduce uh how we test it. So uh this is the agenda today. So I'll talk about the problem or why we built this framework and we'll talk about uh what's the difficulties in ZKVM testing and we'll uh share the details about our

stress testing. So um what's a hot what's hot news right now? So I mean Lineia is now 100% proven. You can go and check our Twitter. Uh so we've been working on uh this for almost six years and uh finally we we are 100% proven uh and we are still ZKVM uh and not we have no plan to move to ZKVM yet. Uh but uh another news is now everyone is talking about ZKVM right so ZKVM versus ZKVM projects are moving to ZKV ZKVM. So um last year I guess in devcon or some some other conferences we had

many zkm friends or projects right now they're talking about zkm so um why why is that so is is building zkm hard yes and is it direct arithmetization difficult yes uh but we think uh zik evm is more efficient so it's yes uh but what about how many projects are working on Zik EVM. We we are not sure. Uh maybe we are the only one or we have some friends. Um so the biggest problem here uh is that we have many challenges in building ZKVM. Um one of the problem is we are lack of uh standardized testing tools and best practices. So no one

shared how they test. So when when they build or implement their constraints and also their prever uh because most team they have uh internal custom test infra and they never share uh this infra and now they are working on GKVM. So um I thought with it's it's good to share our experience today and um there are many edge cases when you're testing constraints and prover. So it's it's difficult to control or manage. So we will also um cover today how we manage that. So these two challenges will be today's topic. Uh but the other the other challenge is like direct

arithmetization of the EVM is very hard because you need many PhDs working on it solely working on it and um also uh when whenever there is zm upgrades you need to modify or update your constraints again. So this these challenges we are dealing with these challenges by introducing a high level language for constraints but this will be covered in another talk. Uh and today I will focus on how to test GKVM. So um what what are some best practices?

So how the heck do you test this stuff? We really don't know. So maybe some project uh maybe they they don't find any other options and just go test in production. This actually happens. I I I'm pretty sure this actually happens and uh many projects they just tonight we test in production. uh it could be mainet world or in test net. So I guess many projects uh they used to test u their new constraints or prover version in test net because it's easier you just deploy it and you waiting for some errors or incidents and you fix it right uh but that's is also um bad idea because you know your new

your users are using test net anyway so um the goal of this talk is we are going to share our experience on continuous integration and testing of ZKVM to help other other teams if there there's any and uh to avoid um their mistakes in testing it and um because you know it's beautiful can uh and uh it's nice weather so I'm not going to share any details technical details today yeah so let's go uh to look at our testing environment so we have a kind of multi-layer uh testing infrastructure so let's say we have three simplified I simplified it and so we have three

layers. The first one here is is about how we test um constraints. So we have this constraint unit test phase. So here we only focus on um implementing constraints based on our constraint spec and we write unit test and especially we focus on covering edge cases. I will go detail uh in the next slides and when we finish this phase we move to the other phase and this is the main phase actually the stress testing. So here we are running many provers and many best nodes and we are generating the traces uh the trace files and then we use it to stress test prover before we move to

devet seapia test net or mainet. So let's cover one by one. So the first thing is constraint unit testing. So unit test here we cover constraint unit test. So the these are the toughest one I guess. So here what we do is we write unit test for um constraint in general. So that that's pack and especially we found out that when you cover only these uh general constraints you will see a lot of errors. So we especially focus on writing edge cases writing unit test for edge cases and boundaries or abnormal

behaviors especially. So we are writing uh thousands of unit tests for only edge cases. So for example this is the left side we have this general case um unit test. So when you to check valid uh valid balance so you can you can spend some balance or amount gas something like that but those edge cases will cover zero value transaction or maximum value transaction. This in reality you won't see this that hap often but uh for testing we are covering all of these S cases as much as possible.

So after this phase we move to the next uh unit testing phase. This is called constraint replay test. So uh we have this uh repos repository. When you go here we have this um function called replay. So this one is when you write a unit test you can actually generate a trace out of this unit test. So when you look at um the diagram we we can actually get those trace files from educases unit test and then we run these um constraint checking tool to see if uh if those traces um satisfy our constraint spec. So if it fails or pass we make a kind of a feedback loop and we

do it again and again and again and um we cover almost uh we cover uh tens of thousand or tens of thousands uh traces before we move to the next phase. So if we finish the first phase the next phase is harder. Um so the next phase we have we are actually running the prover and we are actually using some um basic infrastructure solely designed for our testing. So let's let's move to the the first uh infrastructure which is called shadow environment. We call it a shadow environment because it kind of um a clone of our production environment. So

when you look at there we have a se So uh when you when you look at there we have Sepolia and mainet environment those are version for example 2.0 and we are actually deploying these seapoleia or mainet basus in our testing environment. So it's a kind of a copy paste and we keep syncing those u traces but we have uh we can but we deploy different versions of tracer basu or a state manager or and so on. So whenever we need to uh move to the next release we deploy different versions of these uh actual infrastructure and we reuse it to

uh test our prover and constraint and everything. So uh by using this shadow u environment we can actually we can actually run those um testings. So in this testing we we run prover the real prover and we we can run we can test our constraints and we can also test bas tracer or some errors in state manager and so on. Uh so for example we have this execution prover here and um we have this as a input we have this confilation ranges this this is from uh the real environment. So we can test any confilation range from Sapoleia or mainet. We have historical ranges or or

you can focus on some specific ranges when there was a TPS a high TPS or some incident or something like that. And um uh and whenever you can you can send a you can send a request to the tracer the from from the shadow environment and you then you will get a trace for a specific version of release so that you can run this uh prover with these traces and with this specific uh version of constraints. So when you find these uh errors or some uh problems you you we send this feedback to the arithmetization or constraints team or best team uh depends on the errors and

then we make an update and we redeploy the shadow uh shadow environment again and we keep um working on the test. So when I talk about uh when I talk about the testing ranges uh I said maybe we can test the historical ranges because we when you go to the explorer you can have those historical ranges already but we what we found out is that when you randomly test your traces uh it's not that efficient because as you can see here uh we had this as you can see here we had this uh transaction TPS TPS and this is Ethereum and this is the the pink one is Elineia and

sometimes we have higher TPS than Ethereum. So in this case we had a same a kind of um events uh during this time. So when I test some traces uh around this area so the trace size is very small and there are not many op codes or operations used. So it's not uh efficient enough when you have limited time of testing. So in this case you can just focus on very specific areas like this area and then then they they will have very large trace size and very um a lot of operations there and very diversity dynamic user activities there.

So that's what we found and uh when we test it actually we we run three types of um testing. So one is just random access and the other one is we run over the latest the real time right now the live uh transactions and also on these historically very uh transactions with very high activities. Um I will also talk about uh how we scaling uh scaling out our envir testing environment. So here um so so here each testing node the prover testing node has terabytes of

storage per machine and this machine has almost um two terabyte of memory including the inmemory swap space. So when we are running the prover so we will never have any storage or u memory issue and uh we are using almost 100 uh CPU cores for each machine and uh depending on our release schedule or the importance of the release we kind of scale out this and uh we are spending a lot of money uh only for testing approver and constraints before we move to uh deployment. So here we also have in our prover we also have many

different modes. So we can only check constraints or you can run partially prover you can run partially or you can just run the full prover just like in production. So uh for the constraints only prover for testing one trace it takes less than one and uh for the full version it takes more than 30 minutes to test it. Uh and um so um and uh when we have and also here uh in the shadow environment when we have to test it um we sometimes scale out this environment to have five or more best nodes or uh

best nodes and also in the prover we have five or 10 more uh machines. So depending on our release schedule, we can kind of control this and uh and also uh when when we talk about the coverage of our testing. So for constraint only testing we cover if it's very important to measure release we cover more than 10% of the historical ranges and then uh for each release we are trying to cover more than 10 thou uh 1,000 full proofs generate more than uh 1,000 full proofs so that we uh make sure that everything's um everything is ready uh to be uh deployed and also uh

we We usually use these uh ranges with max DPS and latest ranges to to be sure that everything's all right. So let's talk about some lessons and uh discussions. Yeah. Um so what we learned from our testing is that um a good GKEVM testing framework will always save your cost and your team uh because you know you're not testing it enough and we just directly go to test net or mainet uh this is disaster right and if you get any error from for example test net uh I think it will cost even more uh because

human cost or you know infrastructure cost and also So uh your users won't be using your in um you know network for a while. So that's really bad. So we are trying to spend as much uh spend lots of money and effort on solely on testing. And what we found another one is that when your network is idle uh you better focus on just traces with high activity because uh when you you're testing over network uh with very low activity is almost meaningless. Uh testing over that kind of network is almost meaningless.

And um we what we found another uh is also testing test net is not enough. So you you just deploy test net but because your test net will have very limited activity so it will not find you any errors before you move to uh the main net that could also happen. So that's why you need to build a very good um testing framework. And here's some discussions. So what we thought is we need a kind of standardized or open source testing framework for better ZKVM ecosystem. And uh also I thought maybe we need a very good benchmark uh test data so that researchers or GKVM

projects can just use it when they build their constraints. Uh they need some test data for testing it. And um maybe this isn't just an idea. Maybe if we can build a kind of infra where you can borrow some transactions or trace generate traces out of other uh Z I mean EVM compatible chains that will be also very useful and um we we are currently working on high level languages for or writing constraint. So this is conclusion. So before uh we just deployed it and just pray for um no errors but right now uh after this testing framework and testing phase uh

we are happy we always happy to deploy and uh we are 100% proven since June 9th and so far no instant so far. So yeah um there are not many zuk evm projects out there so I hope we can um keep building together the ecosystem and happy to collaborate with you all always thank you so much [Applause] we have we have one or two minutes so if there's question yeah thank you very much for the talk you mentioned that you have this 10% historic data that you use for testing testing right my question is how do you

pick those traces for testing because I mean you can take 10% of only transfer of you know ERC20s that's correct yeah I mean yeah that's kind of limitation I could say uh just we just randomly picked 10% including those uh very high TPS yeah that's what we do usually but this doesn't guarantee completeness of the entire EVM operations right yeah that that's that's correct That's why yeah I mean if there is a 100% test data set that could be perfect right so yeah we are still uh figuring out just one more followup so for for G there exist this testing suite like retest ETH that is you know that

verifies the VM execution would that be a useful thing to be used for testing here yeah u well I guess so when we can integrate that uh into our testing framework that will be I think yeah really helpful. Yeah, we we'll work on that. Thank you. Yeah, thank you so much. [Music]