The State of Source Code Verification Closed, Painful, and Ready for Change - Kaan Uzdogan

Transcript

So hello everyone um my name is Khan um and welcome everyone again for this beautiful day at the solity summit hosted by the solity team and the argo collective we sourcify is al we are also part of the argo collective and uh I'll be talking about I will giving an overview of the current state of the source code verification why it's closed painful and ready to change and what we are working towards doing that So the other day I was looking into the RC1 1155 standard and I was I came across the this forum post in the open open zeppelin forum. Someone explains nicely what the ERC does and

thanks very nice of them they've written a contract and they have uploaded and deployed it in the girly blockchain in the girly test net. So I click on the links and I come across this. So, it's funny that we have this today in a massive cloud cloudflare outage, but this is unrelated to Cloudflare. This is permanent. And if you want to see a contract on the girly girly blockchain, you you are greeted with this nice screen. Then I have to then I thought to myself, actually on girly test net, we have these contracts, but where are these contracts? Where have they gone? Like

are they in the graveyard? I mean, you can say this. This is a test net, right? Khan, don't be don't be too um don't be too serious, but this reminds me and kind of indicates me what can happen also in the mainet. So, what what do we do if this happens on the mainet? And don't get me wrong, Ethereum actually has a good verification culture. The community demands contracts to be verified. We don't interact with contracts that are not verified hopefully, right? Um, and we have a good established culture, but we can do better. So, yes, we can say the Ethereum contracts are open source, but the problem is they are open

source through closed source verifiers, which doesn't make really sense and then the data set is closed. So of course you can see the individual contracts one by one and that's the whole point right like you want to see the source code of the individual contracts but try getting those contracts and try getting a data set and try running benchmarks as morid show has shown with a data set try to search try searching contracts the verified contract so it's it's not possible with a closed data set so we can do better h but before I will lay out and go into detail details of these problems. So the very obvious one is open source, right? We we want

everything to be open source. We trust only the open source software. And right now the largest verifier is closed source. And as of today, the verification process is a black box. That means it's a black. It's black box. It can't it's it's not reproducible. And this needs to be changed and open sourced. So the other problem is the closed data. As I said the contracts are accessible one by one. You can see them, read them of course that's the point but the data set is not accessible. It's siloed. And for this data to be useful for data um data engineers, security researchers or compiler engineers for

anyone that needs an overview of this data. Right now this is not available. Maybe you have sourced this data somehow yourself by regularly checking the verifier like scraping. Um but this is also not legal. But then this is the data set is right now inaccessible and the problem here is that the EVM data is onchain. Yes, the it's accessible and anyone can build an explorer. So it's not a big issue if an explorer is closed source. But the problem is with the verified contracts because the verified contracts are offchain and they're siloed. So it it means you need to have you need to ask

and you depend on whose database these contracts are on. That's the that's the main issue with the with with the closeness of this data. And that also means when it's gone, it's gone. Right now again these toolings are not available anymore. They're dead and when it's gone it's gone. And the last point is about tooling and the suboptimal devx. I guess everyone many of you here try to verify contract once in a while and it is not fun and it's difficult to know why it fails what went wrong. you need to get API keys and also we can do much better in terms of what we provide to the developers and the tooling. The current API APIs have been there for a

while and they haven't improved and we can actually provide much more data like source maps, bite codes, storage layouts, immutables and the complete standard JSON input and output for better tooling. And the other issue is if we leave things as they are, it's not going to change. So the current players are not incentivized to open up the state themselves. So someone has to do it. Someone has to push it. Not only push it, but also tell community what's going on and so that the community can demand this from these players. And in general, it's not a sexy thing to build like contract verification is somehow taken granted. And there are more interesting

things in the overall ecosystem to build on. It's not the frontier. So that means someone has to take this in hand and try to fix it. That's what we do at Sourcify and also what we do with with other verifiers at the verifier alliance. So sourcify shortly is a was a solid side project within the EF. It started as a solid side project. Then within the EF it became its own team and now we spun out with solidity into the Argo Collective. Um and since then we are part of the Argo Collective also as a nonprofit uh organization nonprofit team and we don't generate any value any revenue and we are a threeperson team and

everything is done by done by me manual and Marco we are all developers. Verify alliance on the other hand is the alliance we formed with block scout routes scan and us and also helped by SAM if you recognize him. Um and in short it's a it it allows easy and unified access to the source code of EVM contracts and in the essence it's a shared database it's a postgress database of all verified contracts that all verifiers have. So whenever sourcify gets a contract we verify it okay this matches then we put it in this database so that other verifiers can pull from it. For example

blog scout then can look for a contract in this database and pull into their database. And the nice nice thing about it is we also publish this data daily in the parket format. If you haven't if you haven't heard about it, Parket is a um modern data format that allows easy processing mostly used by data engineers. So we provide this data in this nice format so everyone can make use of it. Okay. And what are we doing to solve these problems? First of course we open source everything. Um our main verification logic is open source and the nice thing about lip sourceify is it's a browser friendly um package that means you can run it in your own browser

and do the verification on on your own browser and this is actually what we do what autoscan does scan is a local first block explorer it runs on argon so you run an argon archive node and you spin up a autoscan instance very easily and you have a local explorer that you can use easily and it's as I said local first but what you can do with autoscan and sourcify is you can fetch the sources from sourcify autoscan already uses sourcify to show the code but in addition you can fetch the sources on the browser compile the contract on the browser then do the verification once

again on your own machine in in the browser So this allows you to make sure that actually the verification works without even trusting source servers. So we set the verification logic then we have other modules such as the sourceify server the repository UI and we also recently started we took over the openchain.xyz many of you might be familiar with the open chain for the Ethereum signature database. We took the domain and the server over and we implemented the same API but we also enriched this data set. We took over the openchain data set. We took also the four byte directory data set and we also added the verified contract. So whenever

a contract is verified the signatures are already automatically in this data set. So you still can use this the old domain api.opchain Open chain XY Z but over time we will be um transferring this to the sourceify domains. We also have a UI for this and right now you can see we have over 7 million signatures from different sources. And the next thing is the self to source instances. Of course source source is open source and we want people to run their own instances. And right now we are aware of these six public instances from Hashcan which is a block explorer from the hedra chain the monet test net

block explorer running chain parseek finance vchain and also the walnut runs a source instance to verify contracts. Okay so we open source everything. What is next? Then we verify everywhere. So we really believe this is key to open up this ecosystem. We believe a contract should not be verified in one place. It should be always verified in all the verifiers because it doesn't make sense to have it in a single place when you can verify it on in many places. Right? So what we do at sourcify since recently I would say when you verify on sourcify we also share this with other verifiers. So for example, you sent a you submitted as a contract, we verified

it, but also you can see on our UI what's the verification status on other verifiers. So we also share it with etherscan, we shared with route scan, we shared block scout so that you users can also see what they do. But the limitation there is with the scan API keys, you can only post 250 verifications per day regardless of the tire. Even if you have the most expensive API key, we are limited by this. So at least we can only share 250 per day with them. And of course, the second thing is the verify alliance. So whenever we verify something, we post it in the verify alliance database. We also listen to the database. So whenever blocks go or route

scan verifies a data, we also pull from that from that from there and then we also verify it ourselves. Then we have built this remix plug-in. So previously in Remix you had different plugins for different verifiers. You had a Etherscan plugin, you had a sourceify plug-in. We said okay like this really doesn't make sense and it's not it's very suboptimal developer experience. So instead we built the unified verifier unified verification plug-in ourselves and this now includes all verifiers by default checked. So whenever you want to verify a contract, it's already by default on all verifiers except you need to add an API key for either scan of course.

And the next thing is the the tooling and recently couple weeks ago hard shipped the sourceify support and right now there is also parallel verification for source and other verifiers and foundry is also going to support parallel verification or verifying on multiple places soon. And the last thing is opening up the data. We share the whole database daily in parket format as I said both the sourcified data set and the verify alliance data set on these URLs. So you can go to these addresses to these URLs and you will see a manifest file explaining how the data is structured and how you can download it.

And right now sourcify has around 10 million above 10 million contracts. This number is also similar in the verify alliance data set. And sourcify has data from over 180 chains 180 a EVM chains. We also mirror this data in BigQuery and in in Google so that anyone using BigQuery can make use of this data openly. Um as you know BigQuery works quite well in large data sets but also nicely you can pull this data set on a Jupiter notebook and I think on on Google it's called a collab notebook. So you can spin up a notebook pull the I have a pointer yeah pull the data from

from our data set and do any data analysis you want to do. We also have a nice AI playground to showcase what you can do with this data. If you go to our website at sourcified dev, you will see this data playground and here you can talk to an LLM from um free models from open router and you can say for example give me give me contracts with solidity version 0426 or which contract is the most popular contract on the source data sets and this will give you an SQL query and you can execute ute this query on our big query the mirrored bigquery data

set and it will say okay the most popular contract is gonosi safe proxy and we have over 2.5 million going safe proxy instances so this is to showcase and allow users to easily ask questions to the database and showcase like what you can do with this data set and what else is possible so the open data open data matter because it not only allows these but also enables a lot of innovation in the EVM tooling. So with open data we can do EVM tooling testing in bulk. Right now for example Nomic is making use of this data to test their um compiler Slang.

We can do data analysis on this. We can for example feed an AI model and train train an AI model with this with this set of contracts. We can actually the thing that I'm maybe most most excited about is that is building a better decompiler. So I think right now the decompilers have been have haven't been progressing that much and with the data with this data in hand we can build much better decompilers. So if anyone knows how the compilers work or really excited about this, please find me and let's see how we can do this.

Another thing is and another possibility is to find vulnerability patterns. Uh imagine some imagine a security uh pattern is found. You can do a search on the whole data set and see which contracts are actually affected and that would make things quite easier. So these are these are the only things that I can think about top of my mind. I'm sure everyone in this room has different expertise and I'm pretty sure this data can help everyone build better tools and use this more efficiently. So let's use this valuable data set to move EVM tooling forward. And hopefully next time you see a contract you ask yourselves

where are these contracts? Where are these actually? and hopefully they will not be gone for forever. So, thank you. Um, happy to take questions. I guess we have some on the Q&A. If not, you can already scan this and shoot me some questions and catch me after the talk. >> Actually, we do have questions from the from the app. Um the first question is do your four bytes directory also register the error four bytes from new contract being verified.

>> Yeah so we do uh have errors in the forte directory. The thing is to maintain compatibility with the open chain API. I'm trying to find that slide. Oh yeah. So to maintain compatibility with the open chain API we don't have a specific error field. So when you get a response from the API you will see a function and the functions four bytes and an events with 32 bytes because openchain didn't have that we only have functions but we mark them as error signatures in our in our database. So these are for example so if a so if you give us a if you give us a signature the signature itself does

not give any information if it's an error or if it's a function right you take the hash of it you can use the four bytes or 32 bytes but if it's associated with a verified contracts looking at the API we can say okay this is an error signature this is a event signature and that's what we do here in the left side or in the box you can see found in a verified contract API And these are the ones we marked as function event or error. But in short, you won't find the errors in the function field in in a separate error field. But you will find in the function field.

>> Thanks Khan. Um, another question is the public shared posgress database verified by a majority of verifier alliance or does it take just one verifier to include a verified contract? It's it has to be only verified by a single verifier because in the database we write the data and we also write it's written by whom so we know where it came from and others are free to pull that data but then it's not it doesn't enter a cycle. So like it's only it's only written once and the first one that writes it except uh the case when when a contract is verified partially and it gets fully matched. So in short I skipped this information but there is a

integrity hash at the end of the bite codes if you know that. So if we verify one contract without that and then it gets later on verified with that which gives us higher cryptographic guarantees then we write it again and that's a stronger verification. >> Okay. Um, is there a good alternative to now duplicated uh contract century that can be searched or explored efficiently? >> Um, source data sets. I think that's the answer. The we we tried importing the smart contract sanctuary but I think some either compilation information was missing. So we weren't able to import most of the contracts and verify again.

And we also talked to I also talked to Tintin and we weren't able to do that. But I think I assume we have good coverage and the data set is already there. Our data set and it's I assume probably larger than the smart contract sanctuary. >> Okay. Does sourceify plan to support IPFS? >> IPFS. So we already support IPFS. I also skipped that part in this talk because I can't tell tell everything. So whenever you verify a contract, we pin those files on IPFS and you can if you're familiar with the metadata hash, the integrity hash I talked about, you can pull that metadata file from IPFS and you can pull the sources from IPFS. But the problem with

IPFS hashes is you still need to verify them, right? Because I can't put whatever the hash at the end of the bite code. So you still need to verify even though you pull the pull the sources from IPFS. So yes, we publish data on IPFS. >> Um, two more questions. A little bit of a spicy question. Has Etherscan ever commented publicly about the concerns that you've raised so far? the so whenever this came up at least on X their response is we do provide contracts and you can have it the open source license contracts but the problem is if you click on the link and get it first it's not a data set it's just a list of contract addresses and it's like 5,000 contracts for mainet and you still

need to retrieve them with an API so it's not nicely formatted it's not a data set it's just a list of contracts and I don't think saying we can't share non-opensource license contract is a valid argument because I would assume if you're publishing a contract your very goal is to let people see this right and I think it's okay to have this in a data set I think licensing issues are around usage how you're using this data how you are reusing it so and even if that's like that the data still should be formatted and I don't think we have 5,000 contracts in the main net that are open source So I I'm not fully satisfied with that answer.

>> Um, another one to squeeze in quickly. When looking up four bytes with collision, do you share data about relative usage or presence of the different versions? >> Can you repeat again? >> Um, when looking up four bytes with collision, do you share data about relevant uh relative usage or presence of the different versions? >> Yeah. So um, in the first iteration, we didn't ship that. So whenever there's a collusion you have no extra information except if there's a collusion you get a you get a extra field saying this is actually verified with a contract but this if you have a full bar signature that is not on a contract it's not it's marked verified false. So this is the

only information we have but if we see people need more clear information around collisions we can say okay like this this signature has been found in thousand contracts. So, we can say, okay, like there's a 95% chance this is actually the signature and there's a 5% chance. I actually did a nice data analysis on the collusions, like around 1,000 signatures I found that that are not spam. And you can check out our blog post about that. And it's a nice way to see how the collision space is limited. And even even if it's not spam signatures, we get around thousand signatures that are they look really real legit and like unintentional collusions. So check out the blog post.

>> Um here it says the timer has run out but I think we have another minute before we need to start the next talk. How is sourcified data set verified? Do I need to verify it myself too after downloading it? >> Um depends on who and how much you want to trust. If you trust us you don't need to. If you don't, you can spin up a sourcefy instance, do it yourself or as I've as I've shown like autoscan does, you can plug in the verification library we have and pull the data, pull the sources and compile it yourself. But I think I would say the easiest way is to spin up a sourceify instance and verify yourself.

>> And the last one, has source thought about um productizing it and in what ways if yes? Yeah. So as of now that's something I think about. Um I would say for short term we are not looking to productize it. I think the priority is to solving the ecosystem problem and once we think we have made progress on it we might think about having this in a more sustainable way. But either case our priority will always be having everything open source and everything all of the data open. So we will never silo gatekeep or close source things.

>> Thank you so much Khan. Thanks for answering all the questions and that was a super insightful session.