Blockchain Archive

Transcript

Hello. My name is Joshua Foster. I currently work for QuickNode and I want to talk today about blockchain archive. Give you some practical experience that we've seen as one of the premier blockchain providers, RPC providers in the space. I worked for QuickNode since December 2020. And we run a lot of blockchains and so we have a lot of different experience with archive. So first up, I want to talk about flow. Flow is a blockchain and what they use is they use sport concept. So periodically, it used to be you know every like 6 months and now it's about every year. What they do is

they sort of capture the current state of the blockchain and run a process and create a sort of a sort of a fresh snapshot for the next spork and then effectively archive that the existing spork and move on to the next one and so each one is just sort of like a continuous sort of process. What that does is it keeps the database small but and then any archive is just what they've done is they've basically just put them in sort of a read-only mode.

So the the blockchain is actually done really well in terms of performance. It's been very efficient. Um and then for anyone who wants to sort of access sort of historical data, then it uses a historical sort of fallback where if they are called sort of something that doesn't exist in the current spork, then it will sort of fall back to the previous sporks. And so what you have to do when you run the client is you limit the number of fallbacks that it can do to to keep the client performant for data that it doesn't know about.

But Flow it's it's it's allowed Flow to sort of continue to be an efficient client throughout throughout its lifespan. Next is a Hyperliquid. Hyperliquid also uses sort of or uses a state snapshot. So when the client starts up, it goes and pulls a snapshot. And it has no history at all. So what they do is is they actually store their full history in S3 and then someone has created a fork of ref and like it's called Nanoref. What it does is it pulls from S3 and it provides sort of like that full history.

The other thing that you see because there's not really a lot of sort of standard options available for serving up this data. You see you actually see a lot of custom implementations where someone goes someone else goes and writes software that pulls from the S3, makes it available to the consumer. And this is also allowed Hyperliquid to sort of achieve very low block times and but the downside is is that you don't necessarily see a lot of standardization in terms of you know using this client versus that client.

So that's something that's going to need to be improved in the future for Hyperliquid. Monad Monad is also a newest chain. They have a 400 millisecond block time. They did a a lot of innovative work in terms of their database. They also use a snapshot for their base client. So when that base client restarts, it has no state whatsoever. So it pulls a snapshot. Starts up and that's all it has. To mitigate the issue of historical information cuz customers don't just want you know just the latest, they want to be able to query historical stuff as well. They also implemented MongoDB as a

secondary client and what what it does is if the client doesn't have the data, then it falls back to that database and which is actually also pruned. It just stores a longer a longer period and then their third and final sort of historical state is stored in S3 which the client is also able to sort of seamlessly pull from. The one limitation is that it doesn't store states all the states. So if you try and do let's say an app call for let's say 3 million blocks in the past, you're not going to be able to do it because the S3 doesn't have that data and neither does the database.

But it's it keeps the client performing well and synchronizing um quickly and keeping that 400 millisecond block time. Next one is Near. Near is [clears throat] not an EVM chain. It is probably the largest chain that we run in terms of local state. So there are other chains that have more history but Near you effectively run a client and everything is sort of local.

And the way that they achieve that is they achieve that with hot cold storage. So the current state is sort of tracked in a local database. And then they have a cold storage which is a separate key value store that runs you know we generally run it in like a sand because it's 120 terabytes. You can't get 120 terabyte drives and you still need fairly high performance because it's a key value store and so the more data that you put into it, the slower it's going to get. So uh Near eventually is probably going to need to adjust in terms of how they store this historical state. But at by separating it up from a hold and a cot or a hot and

a cold, they're able to keep synchronization and then just archive is just going to get slowly degrade until it's it's it's too slow to sort of query. But that's something that you you also see with with other chains as well. And the last one is Solana. So Solana uses a snapshot to start up. Their state is really large and so it can take you know a certain amount of time and they're doing a lot of optimizations to ingest that snapshot state to improve startup performance.

We're seeing like I think they mentioned something about like 15 minutes on on like the new Dancer. So what that does is that that keeps the database consistent. It it allows you to spin up validator nodes fairly fairly quickly in sort of uh the grand scheme of things. Um And they've actually they're actually going through a transition right now. So uh the what they've historically had is is they've historically had this archiver backfiller service and they would populate a big table.

Big table what we found is is very very expensive. And the other thing that we found is is that it's also very slow. So anyone wanting to query a historical block, they would be between 5 and 10 like seconds in order to actually just retrieve it. So about 3 years ago, we started doing an alternative implementation like ourselves where we implemented our own data store to mirror what's being done in big table in a similar type of structure. And we still currently have that in implementation.

You can also see that competitors of QuickNode have also been implementing their own implementations. And so like we had a competitor recently say oh we solved archive for Solana and archive is now 20 times faster and our 3-year-old solution still outperforms them. So there's a lot of diversity that's coming into the system because people are trying to figure out archive. How do you store infinite amount of data? So right now we're using um a distributed database for storing it but we store it locally and so we can control our costs, we control how much

data goes in, we can scale it up as needed and we're not paying a lot of egress costs which is what you would do in big table type of solution situation. Um There's also a new implementation in Solana where what they're effectively doing is is they're replacing all of the RPC. So instead of the RPC being served from the client, everything is going to be downstream of effectively a stream. So you can stream data directly from the client using either GRPC or at a lower level using an API. And so it provides full access to everything that's in the blockchain. And so you can write downstream services. And so what they're doing is is they're creating a lot of

sort of secondary services that perform significantly better that you can scale up horizontally. They're elastic. You can write database storage stuff. So the the sort of standardized implementation that they're building is this car files and that's called Old Faithful. Where they're straight up flat files where there's data, they're binary, there's a spec specification. And so what you're probably going to see is is you're you're going to see in the future that that's easy to go and read the spec, start implementing different things, doing better performance, your own implementations. So

those are sort of like the five different examples that I have. Next I want to talk about a little bit of recommendations where I think that we need to go in the future Um, from a blockchain perspective, if we want our blockchains to like we want our blockchains to succeed. If they're going to succeed, that means that they're forever. So, by being forever, that means we need to do a couple of different things. And so, the first one I think that needs to happen is this good data verification. So, we've we've had a couple of different chains you know, recently where you've got two different nodes. They're both sinking.

They'll both say that they're healthy and one returns some data and one doesn't. This is the you know, database corruption issue. In order to sort of get a long-term archive solution, you really want good data verification for all of your old data. You can't do that just assuming that everything was good in the past. You have to be able to say, okay, well, this file has got this hash. This file has this hash. Which should it be? So, you need good data verification for archive.

You also need eventual consistency for the database. So, every chain that we've tried to run on top of let's say like a sand, you never have a consistent database. So so what happens is is when when you your sand is trying to sort of deduplicate things to to reduce costs, it's not able to do it because this database has this little bit of extra information. This database has this little bit of information because it's stored in a key value store.

It's just not the same. And so, when you have that, you don't it just it just starts bloating and bloating and bloating the data store and you don't actually get the cost cost efficiency of being able to reduce and deduplicate information. So, in order to achieve that, you really need not only eventual consistency of the chain, but you also need eventual consistency of the database and the database files. You want it to be O1 lookup. Login is just not sufficient for an infinite amount of data. So, you want to and and you can see that there's there's movement in this direction. The tree DB is is you know, you know, for archive on the PBSS is is

a good step in that direction. But, I think we can we can do better by constraining ourselves just for archive and focusing on sort of where what that does. And then the last is is that you really want remixability. Like this is a lot of data. You want it to be an spec driven format. You want it to to you know, have a good API. You want to have multiple different APIs against it so that you allow future users, future implementers to develop new things to use it in different ways that you don't necessarily expect.

So, some of the recommendations that I have specifically that we need to improve is so those were the goals. These are the recommendations. We need to have immutable data. Right now, receipts, they're not immutable. You know, there's a couple of different chains where you can you know, go a day later and then receipt then gets updated with some sort of additional information. That needs to be immutable. You know, it needs to be immutable at least potentially at a finalized state. The other is is you know, faster finalization and also finalization that honors the finalization contract.

We've had chains where you know, 32,000 blocks in the past all of a sudden, you know, this this thing happens and it needs to go 30 it needs to do a reorg at 32,000 and well, that's not what the finalization said. So, you need you need good guarantees on the clients on the chains to be able to sort of guarantee the finalize. Once you have immutable data, once you can you can guarantee that that's not going to change, it's much simpler to implement a sort of an archive solution because so the next is is using a cold storage type concept. You do see this in some things where like raft for example,

you know, has the the static files. And so, you can see movement in this direction. So, for basically all the different blockchains, they need to move to a cold storage type solution for for archive so that you have the flexibility of storing that old data on you know, a sand or some other type of storage solution that allows you to effectively infinitely scale the amount of data. It should be an array database because it should be append only. It give append only gives you the ability to you know, do block offsets. You know exactly where the data is. You always get O1 performance.

You can build indexes on top of that for the you know, the different access that you have. But, at its core, you can always say I want block five. I can go to a block five and and it's sort of always the same performance. We need hash hash hash base verification for those files. So, we need to be able to say I need to be able to you know, if I've got 10 nodes, I need to be able to quickly be able to say is this this node good? Is this node bad? What's the differences and and continually be able to monitor those things and do that in a very quick way.

And it's a lot of data. So, you know, a hash base verification for the at the file level versus you know, you know, traversing the tree to see is this this data valid. Simple file base archive storing. Sharding Eventually, single machines are not going to be large enough to be able to hold all this data even with sand stuff. So, so having file base stuff, having the ability to shard based on files, spread them out to different machines is just going to be that long-term enabler for basically storing an infinite amount of data.

And then the last recommendation is we should also update our RPC APIs. So, when it's archive, there should be constraints put on. There should be optimizations put on of what you're doing with when you're requesting archive information. By doing a sequence based one, you know, most of a lot of times if you're doing let's say data data analytics and you want to suck out the whole data in the chain, you're going from genesis to tip. So, creating APIs that sort of represent that that allow customers to do those sorts of things.

Users to sort of iterate over the information instead of you know, randomly accessing the stuff is going to be important when you have an infinite amount of data and and making RPC APIs to match that is going to be useful. So, is there any questions? I'm so just curious for the archive data. Looks like it's optimized for like historical blocks, but also increase the historical state.

So, some chains do and some chains don't. So, in the case of like Monad, you don't necessarily have historical state. That's not stored in the S3, but it's stored in the Monad database. But, like if you do raft like Karna does have the state. Now, if you if you want to be able to do things like F call on historical state, then yes, you need to you need to persist that as well. And doing that in a sort of an array based format, a sequence based format with O1 lookups, it enables you to do that.

Whereas right now, it's you know, like it's just a you know, it's a ticking time bomb. And so, my understanding is that for some chains that even quick note doesn't cannot offer the historical state cost, right? So, in that in that sense. Correct. Okay. There's no more questions, I think. Thank you. Thank you for coming. Yeah. >> [applause]