Indexing Blockchains

Transcript

Hello everyone. I'm Kristoff. I'm with AWS and what I want to talk about today is indexing blockchains. Um, as you probably know, blockchains are very good at storing data but not really good at retrieving them. So, we'll need some sort of solution for that. Um there are of course a lot of services that provide that. Um you can always use them and that's always the first choice but there might be reasons to build your own indexer and that's what I'm going to talk about today and how would you do that if you were using AWS. Now if you have a very simple data flow that's what it looks like. We have a a mobile client or desktop client. Then we have a front end and then of course we have the blockchain and we're pulling

data from off of it. Now the easiest way to do that and that's what you normally start out with would be to query a smart contract directly. You can just issue RPC calls against the node, get the data and then display it. Um there's all kinds of problems with it. You're restricted to individual RPC calls. You have to do them one by one. It's very resource in intensive for um rich queries because you're actually quering the node for every single bit of data. So it's really not not a good way to do it. So we'll just skip over that.

Second one is of course use an existing uh indexer or one of the open source frameworks something like the graph that's probably the most uh well-known one. Good thing about that is you have full control over the mapping. um or if you're using one of the existing providers, you can just basically use what they are providing and don't have to care about their um own infrastructure provisioning, infrastructure deployment, maintenance and all that. Every once in a while, you might get into the situation that that data isn't enough or it isn't good enough for some reason. So, you really want to go and build your own indexer.

What are the problems with using the existing ones? First of all, the indexer usually pulls or pulls the data via RPC still. So while you have an indexer in between your blockchain node and your front end, the indexer itself of course pulls data via RPC from the nodes and then provides it in a somewhat better and easier to query format for you uh for the front end. uh but still we have these RPC calls and if you've ever tried to do a back fill via RPC calls that can be very very slow and can take a lot of time also you have a fairly late transformation in the game so you're extracting data from the

blockchain and normally because the indexer is does the extraction and the transformation all in one you basically whenever you um whenever you change the transformation you want to try something view you basically have to reindex the entire blockchain again and again if it's mainet that's a fairly long time we'll come to that a bit later yeah that's transformation changes um require reindexing of the whole chain and then late transformation is fairly late in this ETL process we do the transformation but we could possibly move it much closer to the actual node to get it quicker each one of each one of these three issues one by one this This is a normal

architecture. We again the blockchain index is pulling the data from the blockchain node. Ideally what we'd like to have if we want to have an ideal unidirectional ETL flow extraction from the node to the indexer and then to your front end or middleware. We don't want to have this pull component here but rather we would like to have the node pushing the data into our indexer. that wasn't available for the longest time. Nowadays it is. Um, and what we're trying to use here is the rest Ethereum client, the Rust Ethereum client that does provide a means to actually have the node push data into a userdefined sync so that you can use that data more

or less immediately. Transformation, excuse me, usually happens fairly late in the process. there is a tendency to move that earlier to to where where the data is actually generated to the node. I can highly recommend this blog post that's not by me. It's by Andrew Hong. He outlines this whole graphic that's also by him and it's it's a fairly good read. So I I highly recommend it. But that's also what we're trying to do in in the architecture later. Move that closer to the note.

And finally, the last one. If your transformation changes, you want to make sure that you don't have to reindex the full blockchain, which we would normally do because the extraction and the transformation is really one process. What we want to do instead is have the extraction part getting the data from the blockchain, store it somewhere, and then have the transformation part that only u uses that storage instead of going back to the full node. So that's what we're trying to build here. We want to build our own indexer maintain full control over the mapping of course over the transformation want to have that unidirectional data flow from node to indexer to front end early transformation if you want to and uh

transformation independent of the extraction part. Now if you do want to do indexing there's one part I need to mention. Of course, we have a full blockchain sitting there on the node. That's the history that we probably want to have. And then there's also future all the blocks that are coming along. And there's really two solutions to that because they have different requirements. If you're looking at the history, that never changes. There won't be any reorgs in the history or anything like that. It's just data there that sits on the note and you need to somehow extract it.

And you can possibly do that in parallel because we have block numbers. So we can always order that later and we can just I don't know saturate the node with RPC queries get all the data as quickly as we possibly can store it into in our intermediate storage and then transform it. So that's one requirement for the future blocks. Of course, we need to be reorg. So blocks can possibly be invalidated or disappear if there's a reorg and or um solution must be aware of that otherwise the data doesn't isn't right. So two different requirements, excuse me, two different solutions for

it. I'm going to go through that architecture what it looks like. First I said we want to have the intermediate storage where we extract all the data to so that we can then transform it. What we're using here is Kafka. Um at AWS we have a managed Kafka service that makes it fairly easy to set up a Kafka Q. If you've ever tried to do it manually it's a bit of a nightmare. It's fairly complex with a managed service is fairly easy to do. So that's what we're using. This is what uh where our data will be sitting indefinitely.

So we'll extract it from the node, store it there, it sits there and then whenever we do transformation, we'll um base it off of that. First thing then the back fill we have a block to node. In this case, uh I'm using Cryo, which is um an extraction utility uh that Paradigm made. Uh it's it's a public GitHub repository. You can just download it and it's it's really really fast. It it saturates a node when it tries to extract the data. So don't try that with a public blockchain service. It should be your own node, but because it really sends many requests to the node, but by

that it's fairly fast to extract all the data. And what you're getting is a filebased data that um sits on your EC2 instance in that case. And then we can push it over to Kafka. That's for the back filling part. Once we have that, we have all the historical data in our Kafka. Now we want to make sure that the future data gets into Kafka as well. For that we're using or I'm using the RET Ethereum uh client that has these so-called execution extensions which is in a somewhat simplified way I'd say a plug-in mechanism where you can write uh plugins in Rust that are directly

compiled into the node and basically instruct the node to whenever there's a new block to send that data to not only its internal database but somewhere else too. In our case, send it to Kafka. This is just a very very small piece of what it looks like, but you can see it's it's it's a fairly elegant solution, I'd say, because the execution extension gets these notifications. If there's a new um if there are new blocks, it gets this process committed chain. If there's a reorg, it gets this reverted chain and the new committed chain. And if there's

a revert, it gets this reverted chain. So you always know what new blocks are there or if there's a reorg. So it's reorg aware. That's really what we need. Um right. And then of course because it's written in Rust, possibly you can also do transformation here already if you if you were so inclined. I haven't done that here. The only thing that I'm doing is really extracting the data and writing it more or less raw into Kafka because I wanted to compile the note once and then have it running and never touch it touch it again more or less and then have all the transformation on Kafka.

So that's the next part then now we have all the data sitting here. Now we want to transform it. Uh we want to have custom syncs for it. So the first one that I want to and that I'm going to show later is um extracting specific data. Um for example, I've chosen the the Cryptounk transfers and the USDC unis swap uh swaps, extract that data from Kafka and then write it into a database. This is just a relational database. Of course, it can be whatever you want because with this uh Apache Flink, what we are having there is really Java code that runs and

that can do the transformation for you and that's what it does for me. So in this case, we have this custom sync that accepts all the data and then of course our front end will only ever query that because that's where the data is in properly transformed format and Oh yeah, that's what it looks like. Uh this is Java code so don't have to read through the whole thing. But basically what we're doing here is first block here filtering for a particular contract the punks contract in this case and filtering for particular events and then using those events and basically write them into the database.

Each event is one record in the database with block number and everything in between. So that's what the Flink code looks like and of course because it's Kafka we c can have many independent consumers running on top of that Kafka queue that are really one starts extracts all the punk transfers the other one starts extracts all the unis swap transfers and they don't interfere with each other nor do they change the data on Kafka the data in Kafka is always there as our single source of truth if you will and that's it really for for the simplest part. Um I do want to mention one other custom

sync that you can implement which is called S3 delivery via our Amazon data firehouse which basically takes the data as they come along and stores them in packet file format in S3. So this is really the long-term storage if you if you ever want it. it just can sit in S3 and then S3 glacia for the for the really long time start forever at low cost in Kafka of course um I've now indexed mainet and that is about I think eight or nine terabyte or so of data sitting in the in Kafka which is a lot to be honest so at some point you'll

need to decide on how long you want to keep it in Kafka but for now it's it's of course just all in Yeah. So that's basically the architecture that you how you can build it on AWS. If you then look at it from the ETL perspective, now we have a properly separated concerns if you will. We have the extraction part, we have the transform part and then or load part. We basically store it in some sync so that we can use make use of it. But the important thing of course is that we've really separated extraction from transformation because the extraction is really well it takes a long time to do. Transformation

is much much quicker. So we can add new transformations without doing the whole extraction every single time. So that's the the really nice property of this whole thing. So just timing wise to give you an idea how long that takes I've uh indexed mainet from a fully synced node uh to around block 22.63 million extraction via cryo takes about 9 hours to go through the entire chain and write it into into files.

Then ingestion into Kafka from that takes about five days in in its current implementation. But what I've done here is really sequentially ingested it into Kafka on purpose. I haven't done any kind of multiple partitions which you could potentially do and and parallelize it much more. I haven't done that here. It's really sequential injestion into K cafka. And then transformation and store cryptounk transfers takes about a day or so for the existing chain. Now just to give you an idea what that does, this whole thing here basically goes through every single lock that has been generated since Genesis extracts the punk transfers from it. Um

as you can imagine we have 22 23 million blocks or so possibly 100 transactions or so and maybe two or three locks per transaction. So we are in the billions of lock of flock events that are stored in Kafka that we need to go through if you want to do that. So that's why it takes so long. So finally summary before I can actually show the demo I've said all that we've decouped extraction from transformation plus we have a unidirectional data flow no more RPC cores whatsoever and we have independent consumers so we can add new consumers as we come along the really not so nice thing about the whole thing is it's a plethora of

languages so deploying the infrastructure is a CDK script so Typescript the uh Rust execution extension is is Rust and then the um Apache Flink is Java code. So you'll need to work through all of that which is a bit bit of a nightmare to be honest. But well here we are. Now what does it look like? This is just a very simple front end obviously based off of the data. Anyone a particular Cryptunk ID that they're interested in between 1 and 9,999?

33. 33. Here we go. Now, what we're doing now, of course, is querying the database, nothing else. That's what it does. Uh, and basically what we see here is the minting event and then all the transfers that have happened since then. What we also see is we currently um have indexed the database to block 23,830,000 something which is if you look at the cryptounk's uh ether scan page exactly the block that the last punk

transfer has happened. So that's exactly what we want to see. And then the final bit USDC price that's the V2 unis swap contract on main for USDC uh Ethereum transfers. That's a pricing of various uh swaps that you can see. It's really just showing that there is data to see. And what you also can see is the one is if I'm changing ether to USDC, that's the ether price then per USDC.

And that's that's this scale. And then the other one is the other way around. If I'm changing back from USDC to Ether, and of course, we see that those lines are really mirrored because that's exactly what we're expecting. The prices at the same time should be approximately the same. All right, that's what I wanted to show. I'll switch back to the slides quickly because the only thing I wanted to mention is this is our web three blog that we have.

um hopefully fairly soon I'm going to publish this whole CDK script um on GitHub and write the blog post about it. It isn't ready yet. So that's why I can't show the blog yet, but it will appear on this blog page. So if you want to try it out then and actually deploy the all these services and play around with it, this is where it will will appear hopefully fairly soon. I did see we have about five minutes left, I think. So if there are any questions, I'm happy to take them. Okay. Who's that? Yeah, there's a mic coming.

Uh thank you. Uh I have a bunch of questions. Um when you when you go back to the slide, like what what data are are you actually ingesting into uh into your system? Is it just receipts? Is it logs? Is it uh logs and state? What I'm ingesting is I'm I have three different Kafka topics. One for blocks, one for transactions, one for logs. Mhm. And that's basically what I'm what I'm getting there. And the messages themselves are the basically JSON records of those three three things.

So, and then of course for the demo I'm I'm really only querying the uh the lock topic because I don't need the other two. Um I don't know if you know, but filtering out the logs for the cryptounk transfers takes like 30 seconds on the guest node right now. So, it's I I'm fully aware of that. I just wanted to make to see how long it takes on that architecture. So, yeah. Yeah. But I think if you create like we have this new filter index which uh um which blow filters. Yeah. Yeah. Yeah.

Yeah. That that will be much much faster. Of course. Yeah. Um, that will be much faster. Uh, all right. No, I think that was this. Okay, perfect. I just got the notice that was the last question, unfortunately. But I will be around if there are any more. Thank you.