Building and Publishing an Index of Every Address on Ethereum Mainnet

Transcript

um hello everyone my name is thomas rush i'm going to talk about indexing the ethereum mainnet and importantly for me how to be how to index the ethereum main net without becoming a web 3 api provider that's very important to me so uh first of all i wanted to ask if anyone's donated to my uh git coin grants in the past a couple of you i wanted to thank you i was going to announce a token but i'm never going to announce a token for my project so sorry about that but so i have a long history in the computer field i've been doing computer science since 1979 so i've seen mainframe computers go to

desktop i've seen desktop go to web i've seen web go back to mainframe basically and i hope what we're seeing now is mainframe back to desktop so that's what i'm going to talk about i i believe so my driving vision this this is a a nevada box do any of you have avato boxes or dap note boxes so you're running your own um you're running your own ethereum node and it's running on this machine basically you buy this box and you plug it into the wall and you can run an ethereum node yourself so there's a node called aragon which allows you to run what's called an

archive node which has all the data that's possibly available on the ethereum network and the software that i built wants to wants to live behind this box i want to only communicate to this box and the ethereum network to get the data that i'm using in the application so that's kind of the driving force the reason is because when i joined ethereum or when i first got interested in ethereum everyone was talking about these ideas of self-sovereign data perfect instantaneous accounting on every transaction which is on the chain but if you try to pull that data off the chain um you may have discovered that you can't really

balance your account you can't really do that off off-chain without going to a third-party site which to me never made any sense i want to build a piece of software that can reconcile directly from chain data because i think we can build what we used to call truly transparent or radically transparent organizations where any person that's in the organization can do the accounting for the organization can see what's happening on the organization without having to go to the organization to get the information because by in that way we used to believe that we could do coordination on a massive scale which is an organization

that can't lie to us which means that we excuse me which means that we all have a belief that the organization is behaving the way that that we want it to behave so that's kind of where i'm coming from and um i'm going to talk about my work on trying to use the node software for the source of data and finding that the node software doesn't really give you the source of the data you need and then how once we get that data how can we use it in a way where we don't become a web three data provider so we built a piece of software called shifra it's called true blocks the whole

software package there's a command line called shifra and it provides a bunch of different sub options one of which is called scrape that i'm going to talk about one of which is called init another called export and then what we did is we took the rpc data or the rpc endpoints and we uh expanded them and kind of added features to allow us to do an indexing so one of the commands is called sheep for blocks and we added an option called unique which will look at the entire block data using the rpc and extract every unique address in the block

and what we do is we say where did we find this address so here we found it in a log here we found it in the input data which is a thing that i think most indexers don't do currently which allows us to actually get more data than other indexers get so you can see we're looking at we're looking at very obvious places like the from and the two and where a contract is created but we also look into the input data and the log data and we're finding things like this where solidity has created this is clearly an address this is clearly not an address because it has a huge number of zeros that's probably a number

this is very clearly an address so we call that an address even though we don't really know it's an address so we took 200 of the the top bitcoin grants at random and we and we uh ran it against ether scan asking against all five of etherscan's endpoints can you tell us what um what transactions happened on those addresses and we got this column of numbers then we ran trueblocks as indexer as well and this is the difference and in every case all 200 cases we found more addresses than than etherscan was

returning enough so that excuse me i'm sorry enough so that um every time we try to reconcile these accounts they actually balance down to every decimal place in every transaction which is not true of ether scan because there's missing transactions so we get about 115 percent of the total number of transactions that ether scan reports and we always get more transactions than ether scan and we're finding obviously that about 60 percent come from input data and another 20 come from log topics and log data and another 20 come from traces

so we built basically a better index and we did that because we want to find every appearance everywhere on the chain because we want to create radically transparent organizations and we are able now to reconcile these histories of transactions down to 18 decimal places for every transaction on it on a on a thing so now we want to publish we want to say this works well if you have this box but but can we give this index data to people that don't necessarily have this box because the index is very useful to people to build applications with

even if they're hitting against the infuria node or something like that they're hitting in a much more targeted way so they're not rate limited as much because they're not asking for as much data from inferior for example so we looked at existing solutions there's basically two two types of solutions one is a web api like etherscan another is a coin incentivized thing like the graph and they all have they have these problems that we wanted to try to solve web apis have privacy invasion kind of built into them you can't go to a web api without having an api key or exposing your ip address basically so they're inherently going to be able

to in the future watch what you're doing which to me is a really bad design they also have the ability to deny you access to the data for example rate limiting is one way but they could also say we don't like you we're not going to give you the data if they want they're also creating data that we call unreproducible which means that they do the extraction from the chain but they don't show us their work they don't show us how they did that extraction so there's no real way to know that the data that they're giving us is accurate especially given that we can't reconcile the data so that's a real problem for us as well coin incentivized systems to me

it's kind of like i've already paid for this data as our whole community has already paid for the data by gas why do we have to pay again to get access to the index that that should be i think part of this node it isn't part of this node but it could easily be and we should make it part of this node i think the other thing is that if you have to pay to play that game if you don't have money to pay you literally can't play that game which means that if you don't have money you're not going to get access to your the index that you want you're actually going to compete with people that are going to pay more for their data and you're going to get priced out of that market i also think it's way complicated way unnecessarily complicated because

because they're not making reproducible data they have to build a system where they have an arbitration they have disputes they have resolutions and this ultimately falls back onto the graph and to me that's just as bad as a web 2 site that isn't proving their data so uh so we wanted to build something that answered these questions these are the questions we were trying to work on can we actually take advantage of web3 to to solve some of these problems can we build an index that's uncapturable unlike a web api if we build applications that are truly behind a node we've built an application that's perfectly private because i don't expose

my ip address to anyone except for this node which is um in my control i wanted to see if we could build something that worked better if we got more users unlike a web api where every time you get a new user on a web api you have to somehow provide more resources to handle their requests and you have to that's that's just built into web web 2 technology we wanted to make it so users didn't have to do anything basically they just have to use the software it automatically does this we wanted to make it fair so that a user like uniswap who has a blog a transaction in every block

is using the chain more heavily so they should kind of carry the responsibility of the index more more thoroughly than someone like me who interacts on every 3 000th block or something so i should carry 1 3 000th of the burden can we make our results reproducible so that people don't have to trust that we build this data correctly and then really importantly to me is how low can we get the cost of publishing this index i think i hope i can show that it goes almost to zero the cost of producing and publishing this index so um this is a block so and we've done

shifa blocks unique and we've extracted the list of all the addresses in those blocks a regular indexer is going to simply sort that list the trouble with it and what we wanted to do is we want to try to put this data onto ipfs so this list is immutable it never changes so we can easily put this on ipfs this list every time we add new records to the sort we have to resort which means that this ipfs hash changes every single time we add new data so what we did is we basically stopped adding data periodically every 2 million records every time we get 2 million records in this sorted list

we stop adding new data we write it to ipfs as an immutable hash and we call that thing that that does the command she for scrape so you can just get our software run she for scrape and it's going to create what we call a time ordered log of an index of a time ordered log so that's what we built basically because we wanted to have immutability built into our code additionally when we go from here to one of these chunks we put in this chunk we put the instructions that we used to extract and build this chunk and we do that by putting the git commit

hash of the code that built the the chunk so when you download the chunk you get not only the actual data you get the git commit hash that built the chunk and you get file formats and enough information for you to reproduce the entire data set okay and then we um old fat old-fashioned web apis simply give this indexed thing to the end users we wanted to give chunks so we have this sheep for export and we can build a piece of software now that lives here behind the node and i think improves the situation somewhat because

at least we're proving our data or we're making it reproducible so we write each individual chunk to ipfs and then as we go we're keeping a list of all the chunks and we write that to what we call a manifest so each time we add a new chunk we write a manifest we also write that to ipfs and then we publish this hash to a smart contract that we call unchained index and in this way we've completely removed our ability to withhold this data now because we've put a hash to an ipfs file that has a hash that has hashes of every piece of the

entire index is available to anyone who wants to look at the unchained index smart contract and now you can run either your own receiver of this data or you can use our shaffer init to read the smart contract find the manifest download the entire index to your local machine here and now you you're able to use the node in a way where you can access any piece of data you want about any address that you want completely locally which was sort of our goal is anyone with me here except for james then on top of shifra we build a server

which is literally identical to the endpoints of shifa so that you can serve applications so i want to just pause here and look at this and ask how much does it cost us to run this system so we're running this box we're running aragon on this box we're running true blocks on this box all of this stuff from here to here runs literally for free or for however much it costs me to buy that box and how much it cost me for electricity which is nothing so i can publish this thing right here for free it cost me nothing to build

this i want to build it because i want my application software i'm going to sell application software i want my application software to have access to everything they want in my application software so we do have one cost which is publishing this manifest to the unchained index we have gas costs but they're extremely minimal because that smart contract does nothing but accept the hash so it's very very cheap so um any human being on the planet earth can look at that smart contract pick up that manifest hash

pick up all of these hashes and download the entire index of ethereum's mainnet for every address on the chain literally for no cost so that to me is important it's interesting it's it's a direct result of ipfs it's not that we've done anything special it's just that we've used ipfs in the way that ipfs is built to be used it has some other interesting aspects if this user is me i'm going to look at this and say i don't need eighty percent of these

index pieces i need one or two or ten or 15 i'm going to download them here and pin them on ipfs by default we do that by default so that this user becomes part of the solution of distributing the index because he's downloading his part he's been downloading his fair part of the of the system and he's um making it available to other people because it has 2 million records in it only 10 of them are his now what the one trouble we had here is this index is 50 gigabytes big which is a little big

so we also added a bloom filter on top of each of these immutable chunks which is an immutable bloom filter and we also put that in ipfs and we put that in our manifest and now becomes one gigabyte so when a user comes and has says i want to participate they download one gigabyte of data on their hard drive they can do a query that says which which pieces of the index do i need for my address and they can read it from here read it from here look at all of the blooms and only download the big pieces that they need for their uses so you get a fair system

where tiny users or people regular people download their own fair share and pin it make it available to others large users get almost every piece of the index and pin it and make it available to users which to me seems fair and if it's fair i think people will tend to want to use it because they're not feeling that they're taken advantage of so uh so this is kind of it all works um i have a demo but i don't have a demo so i'm just going to wave my hands but um so did we take advantage of web 3 kind of in a

get rid of web 2 in all of its problems let's just see what it looks like to build a web 3 thing yes can we make it uncapturable not really because we have to publish the manifest hash to the smart contract although if we disappear anyone could do that two minutes is it private yes it's private because it's behind the node it works better the more users you get the better this works which is a outcome of ipfs not of us the users don't really have to do anything because it all works by default and you just have to have an application that wants to read this data it's equitable it's reproducible and it

costs nothing believe it or not or next to nothing it's so cheap that i'm willing to build it and give it away for free because i want my applications to work i want my applications to be on a node and i want it to work and i'm more than willing to give it away for free because it cost me nothing anyway so i don't understand why everybody wants to have a coin incentivized network where we have to pay for something that we can literally build ourselves for free and i think we should do this demo demo demo the demo shows that there's no reliance on third parties there's we can cache the hell out of

this data because we're on a local machine so we can make it as fast as we possibly can because we don't have to go across the network repeatedly we go once across the network to the node and then we cache it locally this works with infuria as well you'd be surprised at how fast reading from a hard drive is when you don't have to compete with 10 000 other users on a server to read from a hard drive and it's perfectly private we can get every transaction and reconcile them we can actually surprisingly find every token that you've ever owned and show its balance history because we can see every single incoming token it becomes this is a very interesting thing the application becomes user

centered so instead of me wanting to understand what happened on ens and having to go to ens.io or wanting to know how to interact with uniswap and going to uniswap.io which is a privacy invasion i can see everything i've done here and just walk through my transactions interactively and the thing almost becomes like a spreadsheet instead of a web page because it's so fast because all the data is local so just think about infuriat for a second am i done i'm very close to done um we're gonna we have a front end work that we're doing we're working on a

docker version we love aragon because it allows custom stages so we think that part of this work can get put into a custom stage in aragon so that it runs this scraping mechanism and we have this idea of traversers which is the graph does have a great feature where you can look at each transaction and modify it in any way you want the indexer can but we can actually do that here after the extraction we can customize the data in any way that you the developer wants to provide to your users i think it's really important that we all start thinking about running our own node software because to me that's decentralization is what's the point of

what we're building if every user has to come to a website to use our tools it doesn't make any sense to me at all we should think about running our own nodes if we do we're going to find that the node doesn't really deliver the data so you need an index so i'm i'm suggesting that trueblocks is one possible solution to that and then we can build what i call truly distributed perfectly private local first applications which i think we should try to live in that world because we want to understand what those applications look like and we want to see what we have to do to make those applications work for users there's contact information and i'm done thanks

[Applause] after we have no time for questions yeah i think it's um you can hit me up outside you