Episode 356 - ZK Benchmarks with Conner Swann

Transcript

[Music] Welcome to Zero Knowledge. I'm your host, Anna Rose. In this podcast, we will be exploring the latest in zero knowledge research and the decentralized web, as well as new paradigms that promise to change the way we interact and transact online. [Music] This week I chat with Connor Swan. Connor is in the midst of building Proof Lab, a project still in development that will be tackling ZK system benchmarking.

So think comparing different ZK VMs on various dimensions. Benchmarking in ZK has been consistently controversial. From the early proof system comparisons to the ZK rollup benchmarks to the ZKVM and now the ZKML benchmarks, there's been a lot of debate and discussion in our channels about these topics. So, it was really great to dive in. We talk about Connor's past work, what led him to be interested in the topic, some of the challenges of benchmarking, and how benchmarking can unintentionally be subjective. We then dive into dimensions on which benchmarks for ZK could be created, looking specifically at the ZKVM case. Connor has also prepared a survey form. And given that benchmarking can be quite contentious, I thought this

might be something listeners of this show may want to fill out, especially if they have strong opinions on the topic. Now, before we kick off, I just want to let you know about ZKS 13, which is happening in Toronto on May 12th. ZKS Summit is our bi-annual event where you will learn about the latest in ZK research, the newest applications, and find out who are the most important players in ZK today. It's also a wonderful way to get to know the larger ZK community. So, if you've never been to a ZK Summit, I hope you'll join us. To check out the speakers, program, more info about the event, and a link to join, please visit zksummit.com. I've added the link in the show notes. Now, Tanya will share a little bit about this week's

[Music] sponsor. ZK is finally easy with Noir, the fastest growing zero knowledge programming language. Build privacy preserving apps without any ZK experience. Apps today all look alike, especially across chains. Noir gives you simple privacy tools to build something new. Noir is rust based, backend agnostic, and already used by leading ecosystems like Starkware, World and Aptos. Aztec Labs is running a 4-week program, Noirhack, with 200K in funding and grants. You'll get hands-on support from a dedicated team of 10 Devrils, weekly deep dives, partner workshops, and a demo day where you'll pitch to judges from Bass, A16Z, Crypto, IOSG Ventures, and Paradigm. Applications

close soon. Sign up now at noirhack.com. That's noirhack.com. nirh.com. And now, here's our episode. Today I'm here with Connor Swan who's building Proof Lab. Welcome to the show, Connor. No, thank you for having me. It's really exciting to be here. Uh, it's actually my birthday today. Yeah, it's so cool that we're having you on for your birthday. Connor, you and I know each other from having worked together back in 2021. We worked on the Plumo project. You at the time were at Cello. I was sort of like free I was freelance helping out on a specific project which was the launch of the trusted setup for Plumo which was a very

heavy trusted setup and I think originally the scope of this project for me had been something like 4 months but it ended up going on I I don't remember exactly but something like a year and a half I think. Yeah. In 2021 it was hard to build ZK. Let's just say that. Exactly. But it was kind of a fun time because the group like the crew that came together to do it, we really got to know each other. Um, it was yourself, Kobe, Yas was there, Prne was there. And I know that there was also the amazing like developers from the Cello team.

What we would do is we'd actually like week to week be running these trusted setups like a lot of people running them in parallel. So there was like a big social like element to it. If the listener here doesn't know anything about trusted setups, I can add a link in the show notes to some episodes I did long ago discussing trusted setups, but yeah, Connor, since then I actually don't know that much about your journey. So, what have you been up to? Yeah, I mean I've uh I've been on a journey uh since since um at CE I I'm an SR sort of by trade and so I ended up um kind of bouncing around a couple of the different engineering circles over there and um I uh I finally ended up on the the bridging circle which turned into Nomad.

Um that whole story is history. But um after after Nomad I I spent you know a couple years in in AI B2B SAS land. So you left you rage quit crypto itself. I mean I I rage quit crypto just like a lot of people did I think that cycle after the whole FDX debacle 2022. Yeah. But you didn't stay away it seems. No you can't keep me away. And uh kind of the thing that that really brought me back was uh zero knowledge virtual machines actually. Cool. Um I had I had met the the risk zero team at ETH Denver in 2022. Yeah. Um, shout out to Paul.

And I wasn't really ready personally in in my head of like I wasn't a Rust engineer at that point. And you know, I've learned a lot since then and it got to the point where I really need to be working on this technology like this is kind of the gamechanging moment I think the inflection point for for zero knowledge infrastructure. Very cool. Who did you work with? I spent some time at Succinct. Okay, cool. and uh learned a lot about sort of like benchmarking the concerns of distributed proving in the cloud and that was all just really really cool stuff to me and went off to sort of like work on my own thing and uh that's sort of what proof lab became.

The the thing that I'm really interested in is what makes ZK do the hockey stick thing. Um, okay. You know, if like ZK podcast had the the developer report last year. Oh, the ZK. Yeah. State of ZK, you mean? We do that every quarter. Yeah, every quarter. Great. Yeah. And um we're seeing the same sort of developer growth that we were seeing in AI in about 2015. Wow. Um and that was sort of like buy signal in my head of like, okay, this is interesting. It's it's literally doubling month over month. And that's sort of important and I think that's the catalyst for that is these programming abstractions that make it

really easy to build zk stuff. And I think zk is the endgame. Cool. I'm curious now Connor like what's your background? Like what part of the stack are you working at? Like are you an engineer? Are you devops? Are you like what what kind of role did you train in? And what part of this have you been working on? That's a great question. I'm like I call myself an S sur by trade um site reliability engineering. Sort of a thing that was popularized by Google. And that's sort of the person that comes to a team of engineers when some metric isn't up to snuff. Okay. And you know it involves sort of picking apart the

pieces of the system that you're dealing with and identifying core optimizations or bottlenecks and addressing that so that the userfacing need is addressed. So I I came to ZK with this sort of mental algorithm already built out of like we need to focus on the users and you know providing business objectives and not necessarily like you know I have a PhD like I I didn't study cryptography in college. I just I'm a generalist computer scientist that really likes to optimize. Cool. You sort of just mentioned this hockey stick and maybe we should talk about what you mean by that and cuz I think what you mean is growth, right? Like you mean like adoption, like the adoption dream that we have. Yeah.

That at some point these systems are going to be so easy to use, so common that it's powering so many of the maybe the world systems. Right now, we don't really have that. We do have some like interesting cases where they've gotten lots of users, but it still feels a little bit outside of the mainstream infrastructure powering the internet, for example. But yeah, what what does that look like? Like what would that even be if ZK's so permeates the world? So, I'm not a fortune teller, you know?

I don't have a crystal ball and I can just look in and understand like what will ZK do? But what I can do is look at the past and sort of understand, you know, what patterns have occurred in in the recent past that, you know, and what sort of maps to ZK. And that case in point example for me is is AI. Okay. Um, like we just kind of talked about. And what happened with AI is you had a set of like infrastructure application cycles that occurred over the course of a decade or more. How did it get to the point where there was a large language model in every single iPhone? And that was like a slow process that sort of took a bunch of research and then

product breakthroughs. And I I don't have a strong answer for what a ZK looked like. But I do know that the thing that unlocked AI for the world was actually PyTorch in my opinion. It's a software library came out of Google. What year? Uh that was 2013 2014 I believe. There was there was a couple kind of major versions of it and uh Vitalik actually had pretty cool blog post uh called the glue and co-processor architecture where he sort of like looks at the way that modern computing is is developing and you've kind of got these easy to understand glue layers that a typical engineer can sort of interact with and it plugs in with these highly optimized co-processors for you know to

use Vitalics terminology and uh an AI. This was PyTorch and then highly optimized CUDA GPU kernels to do like highly parallelizable vector multiplication. Um, and we saw that first for neural networks like uh RNN's things like that. And really what was required to make this thing go mainstream was the large language model. And obviously like we we saw how that happened and that was an emerging phenomenon like that wasn't really planned for some exec at open was like we need to release this right now and let's make chat GBT and it changed the world. Totally. And so I think ZK is going to do something similar and the ZKVM is the catalyst for that. Wow. So

is the ZKVM the PyTorch? ZKVM is the PyTorch. Yeah. The ZKVM is is ZK's pietorch moment. Cool. There was also though in AI and so I'm not that deep in it but I I do at least vaguely remember that there were there were these moments of like extreme hype and then marketing and then a little bit of disappointment when like it didn't quite do things that people thought. There was this sense it was going to be magic. It wasn't quite magic. I I'm just thinking like I don't know how long ago this was seven years ago, eight years ago where like this really interesting AI kind of character comes on Twitter or something. and they managed to turn it racist within like 20 minutes or something like it just sort

of like it's like this this toy that's going to get fooled. It wasn't quite at a state where it was magic and started to be like solid and that's post pietorch. So there's like these hints of what was coming but they didn't quite make it. Do you think we're also seeing something like that here? Yeah, I think that's a product problem. It's not really like a core tech problem. it's that there's not very many people on the planet, you know, in 2022, in 2023 that had any experience building, you know, product experiences for normal people that had large language models at the core. And I think that we're just now getting to the point where the the AI shops or you know the AI labs and uh the product organizations are getting to the

point where they're actually building cool stuff that is worth using because I don't know I think we're we're all sort of familiar with the idea that you need to sell a vision in order to raise you know investor capital and I think time and time again the the vision necessarily doesn't match the reality and I think it's okay you know we just have to come in with with reasonable expectations but then in ZK are we still in the selling the vision, but we don't yet have our product. Uh emphatic yes, in my opinion, um I think I think that we're at the point where we need to uh we need to lose a little bit more money before we can make uh some some money in in sort of distribution and adoption.

Yeah. Damn. I now want to talk about the I mean the topic that I wanted to cover with this show and I think one of the topics that you cover with Proof Lab, which is benchmarking. At what point did you start to think about benchmarking in the ZK context? I I I have idle hand syndrome. I I can't not build toy projects and things. Um so one of the things I was working on over the summer is like a ZKS SQL co-processor that you can run against you know any normal Postgress database that gives you verifiable data querying and data uh insertion. Is that a bit like what Lrange did? I feel like there's a couple people that have done

this. Space and time database is is another one. And proving time is so slow that it's not even worth trying to ship a product around that right now. And okay, um we're we're at the point where like SP1 on a single GPU will do like 1.4 MHz. And that's just like that's like a Casio watch, you know? Um and it's that's like the the Apollo the Apollo lander. Very slow. Like so slow. Okay. In in in terms of like the amount of under like the actual native performance is just like orders like many orders of magnitude 10,000 times faster. And so we're at this point where like year overyear the provers are speeding up 10,000 times. You know it's

actually incredibly fast. It's the Moors law. You know more eat your heart out. Like this the rate of development is so incredibly fast right now. Cool. But we're still in like the 80s when it comes to like compute throughput equivalency. Interesting. So that's what got me into understanding the trade-off space here. Like ZKVMs are so much more complicated than like a normal computer. And like maybe I'm over complicating something trivial. You know, I think somebody who's got like deep compiler and you know, software computing experience like may not think it's that complicated. But I think to a normal person arithmetization and memory arguments and polomial commitment

schemes like that's really complicated stuff and there's a bunch of different tools to choose from. It's really hard to reason about. So that was sort of where I started off on my journey of like figuring this stuff out and looking at benchmarks. I mean was it and and I think so far we have mentioned primarily like the ZKVM benchmarks. I mean the reason for this episode in a way is like the conversations around benchmarks and the announcements that teams have made showing benchmarks have created at times quite a lot of controversy in the community or conflict. What will often happen is a team will have done something. They will have optimized something. They create a report, a blog post that showcases, look at this

optimization I' we've done and look at how we now compare to other peers of ours in terms of performance, in terms of speed. What will often happen is those other teams will say something like, well, the choices you've made in in running these benchmarks really like show you in a great light, but we have all these other great things that you're not showing. And then a debate will ensue. Yeah. And eventually it will die down but it will it kind of comes back like if another team releases another metric, another benchmark which showcases its talents in a slightly different way. And I I do think even before the ZKVM benchmark kind of conversation, there was also the ZK

rollup benchmark conversation. And before the ZK rollup benchmark conversation where was the bridge benchmark and the bridge wars of the 2022 you know bull market and like I don't know I come from I come from a place in crypto where like this is not foreign to me and like the marketers at all these companies would say no press is bad press this is actually great for us let's keep doing it. Yeah, you want to be included in the benchmarks at the very least, right? Like you don't want to knock, but if you're in the top three, you know, it's still good. The issue I think is in reproducibility.

Like that's like maybe like one of the top sort of like five core things that I think is um a miss in terms of ZK benchmarks in general. So um yeah, and uh I come at it from like a pretty particular point of view. the S in me forces me to consider the user first and not necessarily the theoretical architecture, the polomial commitment scheme of it all. Um, and I think that that has some pretty practical valuable additions to sort of like the discourse here. Um, some people like the academics of the ecosystem would tell you that there's no point in benchmarking any of these things because they're all so wildly different. It's like comparing apples to oranges. But

the S in me would say That's both fruit. People need to eat fruit. And people are going to reach for one fruit probably and so they should know what they're getting. One fruit. Exactly. Yeah. Um and uh in my experience uh with Nomad, if you're not familiar, you should go Google it. It's not what this podcast is about. We did do an episode on Nomad many years ago. I think before pre pre the the big hack. The the thing with with Nomad is that the theoretical security guarantees of a system are only as good as its implementation.

And um Justin Dowler would agree with me like he you know he's written a lot about how implementations are confounding factors. And there you know shouldn't necessarily be used for practical evaluation of a system and I kind of disagree with that. I think that actually there's a middle ground here of like theoretical understanding, you know, formal verification for example is really important here and then there's like the practical implications of like how do I put this in and how do I run a business on top of this system like ZK verifiable verifiable computation is like the next iteration of the internet in my opinion and if it's going to be that core to the

way that we transmit data you know on a massive planetary scale, it's really important to understand the trade-offs and like the actual performance profile. I mean, I should I almost wonder if we should pick one category that is being benchmarked. So, we've talked a lot about ZKVMs. I think you know a lot about that. So, we can start there like looking at ZKVMs when they're being compared. What are the different dimensions upon which they're being compared today? I mean there's the way it's built which is important but to me personally the thing that is the most important is speed and cost. Okay. Like these are the two things that the user cares about and on which kind of compute

environment. Right. Exactly. Like like the availability of compute is directly related to speed and cost. You know like if we're using like a ingoyama you know one of their crazy ZK chips A6. Yeah, exactly. Like can I even get that chip? You know, like the the proof live benchmarks that I'm building um we're running on very typical publicly available cloud hardware. Okay. Um the Google approaches to to benchmark everything on on commodity hardware, you know, like they built Hadoop, like the the distributed storage system that Google uses is all run on commodity hardware. And I think ZK benchmarking should be based in generally available stuff. But would it be CPU and GPU?

Yeah, I think that's that's super important to understand sort of the performance profiles. Uh especially in a world where Meta has bought out all of the Nvidia GPUs for the next 10 years. U might not actually be possible to to acquire those. Um so how you're even saying like average GPUs, not the highest level of GPUs either. Yeah. Well, the interesting thing is the highest level of GPUs is taken with respect to AI and AI imprints. So memory is actually extremely important in in the AI use case but that's not necessarily as important in the ZK proving use case where it's actually uh cycles comput cycles how many you know floating point operations can I get in a

span of time because the overhead in zk proving is in the virtualization the emulation of the uh the trace generation kind of all of all of that side of things the the zk component And that's basically compute limited. The thing you're benchmarking for, what you've just said is its speed, its cost on commodity hardware, GPUs, CPUs. But before this, you you had mentioned this reproducibility as something that's like key to a benchmark like a benchmarking report. How do you when you're doing this, do you then have to make sure that this is reproducible for others? Cuz I mean, you have your configuration, you've set it up from your location with

your servers like Yeah. How would you make sure that what you're doing is actually also reproducible? Yeah, the before I sort of talk about like what I do, um I think that it's really important to sort of like plug teams that are doing this already in like probably a much more sophisticated way. um powder labs and uh openVM for example like make it really possible like not just easy but possible to write code once and then run it in any of a number of ZKVM backends and that's very sort of uh in line with Vitalix glue and co-processor model you know just being able to swap out the underlying pre-ompiles the underlying co-processors underlying whatever as things sort of develop and move forward so

reproducibility to me is being able to run the same code on any of these benchmarks A benchmark report is a declaration of performance. Okay? And what you're saying to the person who is reading that report is if you run this code, you get this. So reproducibility is sort of like in line. It's spiritually understood that you should be able to get this result. And it's marketing. It's it's trying to get somebody to use your thing. Yeah. And I think that sort of the reproducibility, the transparency, the honesty in all of that is the only important thing uh from that perspective. And like all of this comes from people who have done it for 30 or 40 years. The semiconductor industry

SPEC um has producing it's an industry group that has been producing benchmarks for the purpose of evaluating things like CPUs, virtual machines like the JVM. Mhm. People have been waring over benchmarks for my entire lifetime and then some. And um people will continue to war over benchmarks, you know, in perpetuity. So it's important that sort of um I think the the ZK community bands together and builds a set of universal benchmarks. I don't know. Proof Lab can be the start of that, but it's not going to be the only uh sort of contributor there. But the way that it's working right now, isn't it almost like every team is creating their own benchmark report and their own benchmark like strategy and then they are optimizing

based on what they're benchmarking. Yeah, I want to just share I mean I I'll try to link to this, but I posted something last week about like benchmarking being a point of contention. I got a lot of great answers. I'm actually going to link to this in the show notes on that post that I shared like Paul who you had mentioned earlier from risk zero actually posted a little comic with three frames. The first one is situation. There are 14 competing standards. And then in the middle, someone's like, "This is ridiculous. We need a universal standard." And then situation. And the third panel is there are now 15 competing standards. Now, you could just replace benchmarking report styles there. And I think you have what we have in a way. So why is that different? Why is what you're doing or why why do you

think we actually do need some sort of like industry standard? Won't we just fall into the same problem? In supply chain, we call this the profit disincentive for cooperation. Oh, where individual actors, rational actors are disincentivized to work together to grow the total amount of revenue in the system. And like I think that is just like exemplary what's going on here. What the semiconductor industry learned is that verticalization enables growth. Okay. So because you've got these giant companies that sort of specialize in one thing, you may have two or three other competitors, but like to grow the pie, you can't be fighting with each other. You need to be cooperating when it counts. So one of

the ways that for example the semiconductor industry has band together to fix maybe not fix but address the problem is through industry organizations that set standards. standardsm bodies are there for a reason and it's to enable commerce and uh as long as we're bickering on Twitter about benchmarks we're not working and growing the pie and we're not growing and onboarding users although aren't we getting to better benchmarking like isn't it sort of a process of like well maybe we've been too academic or maybe we've been too focused on CPUs and there like we are moving closer towards specialized hardware for these things like yeah I feel like there's there's a

few different arguments that I've heard from teams who you know would argue that a benchmark isn't isn't done quite correctly. Yeah. So the approach to benchmarks internally at a lot of these organizations is we are subject matter experts on our thing our prover system our approver our hardware or whatever and we are not subject matter experts on the other competing things. So we will take maybe the example project from a competitor and then we will optimize the hell out of our own project and then ship that entirely and it's like well your doc said this so we just copied it verbatim and it's like you know it's a bad faith example of performance. Yeah.

And I think like a great example of a sort of industry aligned thing that does this right is ETH proofs. Uh Justin Drake has been working on that at the uh the Ethereum Foundation and that is a system that is designed to prove Ethereum blocks and I think they prove like every hundred blocks or so and groups prove that are building provers. It's on them to come and implement their best case solution as long as it fits a set of constraints. You know, we're not allowed to distribute those along many computers. It's got to be one instance, one GPU, whatever. And I think that that is like a normalizing that's a regularizing force um in an industry where there's a lot of sort of froth.

That's ETH proofs that you just mentioned. I actually hadn't Yeah. ETH proofs.org. Yeah. I'm not exactly like the goal of what they're doing is like they're serving a purpose, right? It's not just for benchmarking. It's like they're doing something where you can start to see how these proving systems are working, how effective they are, I'm guessing. But can you explain exactly what they're trying to do? Just like what is their what's the core goal? Yeah. So the the first call it battlefield the first battlefield of of ZK it seems uh is the ZK roll up. Yeah. Um the the whole point of ZK in sort of our social scene is the the ability its ability to scale a blockchain and to

well it used to be privacy and it still could be privacy. I think it's still I think it privacy is the big one, but like put that aside for a second and like let's just discuss rollups because um that's where all the money that's where the revenue is going to be coming from is that the the the layer ones and the layer twos like are going to pay fees to get their blocks snarked or started. Um, and understanding the performance profile of that particular thing is really important when you're a rollup operator and you're picking which proving system to use or you're the ETH foundation and you're trying to figure out you know what's proving systems to support at layer 1.

What ETH proofs is doing is uh I'm going to explain the ZK the ZKVM use case but like if you know you could you could build a custom circuit for this and it hasn't been done in a ZKVM you take uh maybe Helios or Ref uh Revom uh you take a consensus client and you shove it inside a ZKVM okay you take the Rust code you compile it for the ZKVM you take the inputs which are maybe the last block header the current state route the next block and you put it as input into the ZKVM. ZQVM executes the code and then generates a proof of that code.

Okay? And obviously that's like that's the thing when we're talking about a ZK rollup. That's what we're talking about. We're talking about generating proofs of correct block transitions. And that's really cool because you can take that proof and you can verify it and you don't have to recomputee the transition yourself. Yes. That's literally what we were working on at Cello back in 2021. Yeah. was was being able to do exactly this verify block headers and do a quick sync. But ETH proofs is doing you were saying it's sort of like using these systems like I I want to understand how it's benchmarking like how how is it actually I want to understand what it's doing cuz it's not a rollup it is proving Ethereum blocks full stop full stop not rollup blocks just like just

just Ethereum mainet blocks rollup blocks if EVM is EVM you know it's going to have similar performance overhead profile whatever and it takes one out of every 100 blocks sends it to approve prover has it prove it records how long it took it it takes and then computes a relative cost it's on this type of AWS instance for example you can compute based on how long it ran that's how much it costs and so speed and cost those are the two things that are important here who's doing that who's sending it over there and are they using different systems at the same time like are they using the like are they proving it in parallel multiple times with different systems just to kind of see which one performed in a certain the the current

three provers that are supported are Succinct, Snarkify, and ZKM. Um, and just looking at the the website right now, it doesn't actually look like all three have been reporting proofs the last call it 100 blocks that have been proved on here. So, still uh you know, a little bit sparse. And I think that what is required here is more VM organizations, more prover builders really caring about what the optics are of ETH block proving for example. Yeah, I mean all of this benchmarking, who is the audience for it really? Is the audience the person who uses an application deployed on those VMs or is the audience meant to be the developer

who's potentially going to build their application on using these VMs? Yeah, it's definitely the dev like what what is happening is this ecosystem is just competing for the mind share of like the same 200 developers right now. Um, and I think that's I think it's okay, you know, although isn't the shouldn't like that number be higher because the easier these things are like the easier it is to just build an application, shouldn't that market actually be quite large? Like you don't need to know ZK that well to do some of these things, right? That's the goal. And I think like the the product organizations uh sustain is doing a great job um rising a great job at producing frameworks for this like ZK rollup frameworks like uh you're already going to probably hire a conduit for

example to run your your ZK you know blockchain and sort of like it's important to understand the performance profile. I don't think that the ETH proofs block proving use case is like the end all be all of benchmarking, but I think it's a really good example of like an industry organization that is aligning incentives that is sort of standardizing a benchmark that is relatively understandable that doesn't require a PhD in cryptography and I think that is a net good thing but we need to do better. We need to do more than that. Interesting. I think there they're almost it needs to be a spot that people are paying attention to in choosing maybe in the way I I sort of said this in in that post but it was like the L2 beat for overall rollups

that was like it became a source of information for a lot of people into how things are performing where they were on certain metrics uh you know this sort of scale of how much of a rollup they are how much of a zk roll up they are what have you like what features did they have they had built some sort of dashboard that helped to compare it and I don't I haven't so far seen anything that quite does it in such a like easy to understand way on the ZK front in the benchmarks and also I mean you could do it maybe for ZKVMs but there are other dimensions that people also seem to be exploring like the recent benchmarking conversation was about ZKML so I don't know yeah so um I think that that's a really sage observation that you just

made in that the benchmarks that are most relevant are the ones that are closest to the real world use case that is developed. Okay, I think that um you can implement ZKML in a ZKBM. You can implement um ZKTLS in a Z in a ZKVM. You can implement ZK email in a ZKBM. And if you can do that in a portable way, you get a really nice, you know, zeitgeist relevant benchmark that speaks to users that like they understand how Zikls works and that just kind of natively makes sense. that that's what the SPEC has been doing forever. You know, they're doing, you know, weather simulations and they're running like actual real life workloads because

that's the whole point of doing a benchmark in my opinion. It's not just to run Fibonacci to the millionth digit or whatever. Yeah. So, what is proof lab then? Because you're not it's it's not ETH proofs. I guess you're not doing what they're doing. So, what what are you doing? The goal is to be that that L2B style like I I can look at like a nice little radar chart and I can sort of make a relative size estimation. Yeah. Judgment call of like should I choose this one to build a prototype or should I use this one to build a prototype. The approach that we've taken thus far has been to try and build an SDK that makes it easy to write portable code. Um not necessarily at the compiler level. I think that that's kind of

overkill right now considering how much churn and how uh you know high velocity things are uh compiler tool chains are all over the place. So just make it really easy write one rest program pop it into uh SP1 or jolt or zero and that's like the benchmarking use case. You know we've got I think maybe 10 or 15 benchmarks and then sort of the secondary goal here is to produce that breakdown of we've got the benchmarks which is like the real world workload assessment. We've got like uh systems systemwide security analysis which is like really important understanding what parts of the glue and co-processor architecture are like you know affect the trusted computing base and like you know as these things change like the

only thing the only people that are reading most of this code are the people that wrote the code and like the auditors and I think that building a solution a project team that is just focused on doing some of that work is like not only valuable but important to the longevity of this industry. Yeah, but the key components here are like the workload pattern analysis like understanding uh for example memory access. Um one of the things that I've been working on this week actually is um creating a worst case scenario for ETH proofs. Okay. Um basically getting catch 256 hashing is really inefficient inside of a ZK circuit. M. So if you can get a block to do a lot of that and force the

prover to generate a bunch of hashes, you can create a worst case scenario for relatively cheap because the the ETH client natively has no problem running these hash functions because it's built inside of a you know pre-ompile or whatever. But the prover has no such optimization right now because we're we're early. They will eventually but they don't right now. So you can three or 4x the proofing time or more may might be able to create an out of memory sort of situation for relatively cheap 25 bucks 6 million gas. Interesting. So you're like pushing this you're kind of making it really awkward for these systems to use for the ZKVMs to actually do fast. You're making it like uncomfortable as uncomfortable as you

can. Yeah. Yeah. I I literally It's called Z Carnage stress testing uh the ZK systems through maximum pain. Cool. And the reason for that is like I'm not trying to be mean here, but it's better that me a researcher does this than some bad actor from North Korea because there's some economic incentive later. You know, I'm not making any any um statements around sort of the actual incentives or why you would do this. Yeah. Or it could just happen accidentally, but like at the worst time possible. Like that's the other that's like almost the more likely way that those things happened. Like there's I've experienced that personally. Yeah.

Yeah, the the last kind of two big ones are sort of like implementation trade-offs, you know, like are you trading off security for speed? It's really important to know. And then sort of the last one is like is there in integrated verification framework for formal verification is really important here. Um formal verification is something the semiconductor industry leans on for shipping all of their chips. like a lot of these chips that they ship are formally verified before they even hit the production line and that's like of the utmost importance and I think that um if I wanted to shout out a particular team Risero is doing a great job of that they are spending a lot of time and money and energy trying to get their their circuits verified formally verified will that then be a

must do you think for all the teams eventually that is something that they're going to have to do another another blog post for the the the show notes is going to be Justin just released uh I think last week or two weeks ago um his idea of what the stages should be. We got stages for roll-ups. Maybe we need stages for thinking about ZKVMs. Interesting. And a big chunk of that maybe 50% is around sort of the verification posture. Is it possible to verify the claims made by the developers of the system? Wow. Do you think this is a bit of an aside, but I'm just realizing like the ZK EVMs the rollups the initial idea was I mean it was like there's the EVM we need to extend the

EVM we create a ZKP version where we compress it and we use it for scaling but once ZKVMs emerged like it totally takes on a whole new way of thinking about these systems right like you're not extending like the base of Ethereum if everything was built still on Ethereum which we don't know But the Ethereum base could stay EVM, but everything that comes off it could be whatever like more real world languages and more real world systems almost. And it is such a paradigm shift. Yeah. And I I think I personally find that really exciting.

The thing the thing about the ZK EVM is when you scope down the use case of the virtual machine you're building, it gives you a lot of opportunity for optimization. Hm. So I think that um a lot of the the ZK EVM implementations will probably be better for an EVM use case than a ZKVM in the long term because you you are just focusing on EVM use case whereas risk 5 is a larger instruction set. It allows for more in my opinion more expressive systems and applications. And I'm not making any claims around how like I think a ZKVM should replace the EVM underneath to generate proofs. Like I don't

necessarily think that that's true. I think that um what I am most bullish on is the ZKVM's ability in bringing offchain data into an onchain context and allow for entities that maybe distrust each other to trust the data that they're transmitting. H Ben Fish had a great tweet uh from Espresso um recently around how verifiable computation is the most useful when you've got greater than call it 100 untrusted parties that are looking to you know verify your claims. Mhm. So enterprises maybe not necessarily the best use case cuz they can negotiate they got lawyers they can negotiate these trusted information channels but I think the killer ZK use case is going to

be probably privacy in the small medium business context like that's my nois prediction maybe hash that put it on chain and we'll talk about it in a year nice going back to that post I mean I had some people answer when I asked like is it possible to do these benchmarking or is it too complicated some were like it's easy but others just did say it's actually really complicated because there's all sorts of different points of benchmarking one could do and here's a few that were listed. I just wonder what you think of them. You could do prover time, proof size, verifier time, proving key size, verification key size. Is there a trusted setup, which I think mostly not, but say there was. Uh memory usage, I think you did mention the

hardware that it works on. Uh single versus multi-threaded choice of crypto library. Have you thought of any of those? Yeah. So like the answer is yes. I have thought about like a lot. By the way, just shout out to Alen from Aptose who who posted that. I want to give him some credit. Yeah. Yeah. The important thing here is to build like a multivariate analysis framework that takes into account sort of all these things. So okay um some of the things that you mentioned are in regards to the performance of the system. So what sort of hardware do I need to buy to run this thing? and some of the crypto library choice the formal verification. Yeah. So like let's let's talk about proof size in a second. The implementation choices

are really important from a security perspective. Uh supply chain vulnerability perspective. North Korea's you know making commits into open source software these days. So like it's really important that you choose correctly and you check yourself on a regular basis. And then there's kind of the integration concerns of like proof size for example. like proof size is really important when you're integrating with a blockchain. Grath 16 is currently the deacto standard when it comes to snarks to put on chain but all of these EKVMs are producing starks that then have to be wrapped. So there's an extra, you know, recursive component of generating proofs that go inside of other proofs and that has a whole performance component to it.

Um so like the proof lab benchmark currently looks at literally all of those things. Yeah, we look at proof size, we look at the different kinds of proofs, we look at the proof times, we look at verification time, we look at proof sizes, and you know, I can tell you that we've got ECDSA, we've got uh EDDD SA, we've got Fibonacci like everybody, we've got the Helios light client. Um, you know, I can do ETH proof style block proving, JSON parsing, uh, catch reax RSA 256R1 SE Shaw Tenderment, you know, like all of these things. Like when I'm talking about real world benchmarks, this is what I'm talking about. What do you have to build to benchmark that? All you got to do is

import a library. A lot of the time, like that's that's the best part. You're like, I got this. I got that. I'm like, what have you built, Connor? Okay. Yeah. So in in doing the the research for that that ZC carnage you know kind of uh project attack you know experiment. Yeah. Was reading the the Revom and Reth codebase and understanding for example uh did you know that the EVM as it exists in the ref codebase uses a uh mod exp is a is a pre-ompile inside of it does a modulo exponentiation or something. Um. Mhm. It uses a crate, a Rust crate that was written by the Near team. Oh, like in the same way the EVM can just import code and run it. That's

it literally exactly what is going into a ZKVM. Okay. Is just the exact code compiled for the ZK ZKVM executed and then proof. Hm. Are you trying to build a tool set where anyone can run their own application on these different systems in one go and then you see the comparison like is that the goal of proof lab? Yeah. What what pietorch and what transformers did for AI is very much what I would love to contribute here to ZK which is hugging face transformers allows you to swap out any sort of LLM. you know, Deep Seek comes out with a new, you know, Nvidia disruptor. You just drop it into your pipeline. It works exactly like it did yesterday, but it's just now better. And

you know, calling back to Vitalix glue and co-processor architecture, like the idea here is to make it so you don't have to rewrite all of your code. You write using the proof lab SDK and it slots it into the most efficient prover for that particular use case. Whoa. Uh maybe you've got, you know, security concerns. We're not going to choose the most risky one. Maybe you've got performance constraints. Maybe you want to prove on a Raspberry Pi for some reason. You know, choosing the back end for that would facilitate that use case is important. Um, and that's where Proof Lab kind of sits right now. Interesting. So, you are are you like a middleware?

You could say that. I think I'm just glue. Dirty word actually. I'm just glue. You're glue. Okay. You're the glue. That's interesting. Do you think by creating a system like this, I mean, if it was if it became popular, then would people be competing for those use cases to be used on their systems in a way or would do you think that they're just they like all the teams should continue to build along the metrics that they're optimizing for? They will find potentially use cases that will fit their optimizations and that they shouldn't worry about like the other ones kind of. Yeah. So like I think maybe like rephrasing what you just asked like where does the money come from in ZK like what are people competing for and like yeah people are competing over ZKVM performance but

really what they're trying to do is capture users to acrue value to their prover network. this enables that. Like we're still going to need to distribute a proof generation like today in 2025 and probably 2026 and 2027 like we're not going to be talking about prover costs because it's distributed but like you're not going to be able to prove that locally in you know maybe a ZKVM given the current overhead. So making that switch easy, you know, for example, being able to switch between boundless or SP1 prover network or Furma or one of the other, you know, Lrange prover network, any of these sort of existing or, you know, soon to be mainet systems, developer choice is important and that's one of the things that I really care

about. Going back to your your initial idea like the application developer doesn't really care who's doing the proof where they're just creating an application that is supposed to have a certain speed performance like etc. What you've described is that it would be sort of like a switcher between those things depending on the case. So that's why I'm wondering if you're actually interfacing with the prover network then. Yeah, I think we're going to get to a point where like right now the cloud cost of ZK infrastructure is not factored into the cost of proof generation that's currently being subsidized by venture capital. Sometime in the next three years, we're going to see the fee switches get get switched on and we're going to have real competition

for pricing price. Wow. And the this is the exact same math here for AI inference LLM inference is the same for proof request. Interesting. So just like we saw commodification of inference uh and a race to the bottom we're going to see a race to the bottom with proof systems and proof networks and proof generation and being able and having agility in capturing that value is it's incumbent upon any startup founder to make sure that they have that ability. Hm. Why does this open up more people?

Like why does this open up the space to more people joining? Kind of like going back to the initial thing you were talking about like wanting the hockey stick and wanting the more users like cuz right now sometimes I look at this and I think like it's an amazing system but there isn't that there aren't that many users and it's still sort of a theoretical use in a way. It's like theoretically there's going to be users. Therefore, we're going to build this so that it can handle all of this. And a lot of the economic modeling is based on that idea as well that there will be a lot. There's a few examples where we have seen that happen. Like there are certain applications that do need a lot of proof and we see what they're using that for. But yeah, I'm just kind of

wondering how like if by creating benchmarks or by creating even like you know more efficient systems where you're getting the right ZKVM for your use case, why does that create the hockey stick? I don't think it necessarily creates the hockey stick, but it is a required feature of maybe the soil environment for the plants to grow. Okay. So like the the plants here are developers and developers need to be able to make informed decisions and if they can't make informed decisions they're not going to build anything. Yeah. Like what hugging face did for the AI ecosystem circa 2016 is what enabled that hockey stick growth. You need these these applications these systems these businesses that are focused solely on the consumer use case. And it became

clear like hugging face kind of seemed like a stupid idea until chatbt came out. Wow. And it was like, "Oh man, this is a great idea. You're a genius." But it took them like 5 years, four years to get to that point, you know? Wow. Cuz were they building they were building like the grand hall for an audience that had not yet arrived kind of? They were pitching a vision to VCs. Yeah. And you know, I think that their business model is still sort of like in flux and in question because their valuation is $4.5 billion. But I think that similar environment needs to exist. So, we're still waiting for ZK's chat GBT moment, but when that occurs, I think Proof Lab will be an extremely valuable tool.

Cool. Well, Connor, thanks so much for coming on the show. Oh, thank you for having me. Thanks for sharing with us all of your insights into benchmarking, a topic that's obviously very key, very important, very much discussed in our community, and it was really fun to catch up. Yeah, seriously. Maybe I'll have to get out to a ZK summit sometime soon. Yeah, there's one coming up Toronto. Go figure. All right. Thanks again. And I want to say thank you to the podcast team, Rachel, Henrik, and Tanya, and to our listeners. Thanks for listening.

[Music]