DAPPCON 2024: Storage Diffs As The Ultimate Source Of Truth For Analytics - Tomek Mierzwa

Transcript

hello everybody thanks for waiting thanks for being here uh thanks envio for the great presentation and thanks for your question because I will try to answer this question as well during my presentation I'm tomc I'm a co-founder and CD of token flow and I'm going to propose you a slightly different approach to onchain data analytics that is based mostly on the storage information okay it works okay at token flow we provide data we provide data for complex use cases

use cases that require more data more detailed data that you can get from events and calls and what we do we also try to be fully multi-chain which means that we aggregate data from multiple chains into a single database so all the analytics that you do you can actually do across multiple chains at the same time and uh I would like to focus mostly on our approach to analyzing data using the most important the ultimate source of Truth which is the blockchain storage as you know very well every blockchain is

actually about the storage this decentralized Ledger it means that we have some data that we store and it's changed in time by well defined actions like transactions but actually what matters is what's stored in blockchain how much tokens I have what rights I have what identity I have everything is written in the blockchain storage and actions like transactions they change the storage and actions they emit events some of actions and M events not all of them and only in a way that was designed

by developers because events are defined by developers in smart contracts so you get events when it was designed by developer and they contain the information that was provided by the contract developer and events are very popular like the most of the data analytics today is based on events and events are great they're very simple to use easy to use they are well known and if you look at the most of the dashboards analytics everything is about events today but sometimes events are not enough and sometimes events are not good I will even say that events are bad I will try

to give you a little bit more context for this as you know very well events cost gas to emit event you have to pay for it and today this cost becomes important and more and more projects they started to think how to reduce this cost and how to stop emitting event or limit the number of events that's emitted by the contract even today we talked to one of the projects that's building the Dex that is by design eventless it will not emit any events and probably some of you are also

familiar with the concept of Shadow events this is the nice concept introduced like a year ago which is approach that is based on adding some new events to existing contracts in parallel implementation of the blockchain so you take existing contract you modify this contract you run this in parallel environment and you can emit your own events whatever you want you can include in this new smart contract and this is the trend that you can see on the market that people are looking for a way to avoid costs that

is related with uh emitting events today and this is one problem one problem you will you're going to see less and less events going forward in the new contracts new implementations another important problem is the fact that they are Developer defined so you get in the event what developer wanted to show you and you have to trust developer or you need to deeply understand the whole contract to be fully sure that everything that you get in the event is actually complete it's true because it's it's something that's developer

generated event is not the contract storage event is just some representation of what happens in the action provided by developer another problem is even if developer is fully trustful all events are great still sometimes you may find a need to change existing event to add some additional information to existing event or to add new events to Smart contract and you cannot do it right you cannot do it today because events are implemented in smart contract so you cannot change it so our Maybe bold argument is that events are actually completely redundant

and completely unnecessary because everything that you get in events you can get from contract storage which is not easy but it's possible because the storage is the ultimate source of the truth everything that's important is in the storage and if you imagine if you have access to every single change in storage history you can get exactly the same information that you get from events and much much more but working with storage is very hard this is the reason why so many people today use storage for analytics the blockchain nodes were

designed as anvia said before me for doing different things not for getting the data out of the node and getting storage data is the hardest part of it problem is very storage is very hard to extract from the blockchain uh storage is implemented in a way that makes it really really hard to work with first the most important problem is as you know most of the storage is in the hashed form if you have this the hash map so you have slot and keys for example you can you have balance of address it's actually gets hashed and

this is how it's written in the uh blockchain storage and it makes it not usable at all this is the reason why everybody says that you cannot iterate over hash Maps you cannot ask the contract for list of holders this is because the storage is hashed another problem with storage is that it's completely unstandardized for events for example if you have erc20 events you can expect some standardization you can expect the transfer events will be the same for every token for contract Contra for storage there is no standardization at all so everybody tends to implement this

in custom way also it this is this problem of as I called it semantic ambiguity is because the same storage the same physical storage can be interpreted differently it's not me interpreted differently by two different implementations of the same contract imagine if you have the diamond pattern so you have one proxy that calls multiple implementation they all work on the same storage and the meaning for the storage element can be different for different

implementations storage is huge if if you take all possible variables of all smart contracts and multiply it by every action you get really really huge volume of data that's actually not possible to to work with easily and also working with storage is hard like analyzing storage is hard because events are very simple we learned how to work with events but working with storage is is hard and what we do we try to provide a solution for this so our approach what we do at token flow we first of all we implement the custom Tracer for

evm chains which means that we use standard node but with custom Tracer which can be implemented in the node and it enables us to get the every single storage div in preh hased form which means that we have access to every situation when anything in blockchain storage was changed and it's even true for the transient storage that was introduced recently with the latest hardw so we have every single change and it's in preash form so it's actually usable then the next step is storage decoder that it tries to make this very complex information usable by exposing

all the key maybe also the variable names for verified contracts so we expose the storage in the form of storage divs at the lowest possible level and then we build this multi-chain data warehouse which has what every other blockchain data warehouse has Warehouse has we have all the blocks information transactions internal calls events but also this storage divs information on top of this we also provide some analytical front ends for analyzing this data but still it doesn't solve the problems of complexity we learned hard

way during the last few years working on this that even if we expose all this storage information it's still hard to work with this it's still quite conversome and you need to have like very deep understanding of contract logic and you also need a lot of SQL skills to actually build any queries on top of this uh storage so today we are trying to introduce completely New Concept concept that we call EOS and EOS are artifacts that are similar to events but they are built directly on storage information so we take what we have like

every single change that ever happened in blockchain storage and we on top of this we build additional layer of processing that simply transforms this row information into something that looks like events so these are the artifacts there that are emitted in specific moments in processing and these moments are unlimited we can send Eco at the beginning of the block at the beginning of transaction when storage specific storage variable is changed we have full flexibility when we emit such uh Eco and we have full

flexibility to what we include as a content of the Eco so think about EOS like events that are generated directly from Storage information of course it happens in like in postprocessing which means that we have all the storage divs information and then using this data we build some formulas to generate EOS this is very important because this happens in database which means you can easily build Eco that you can calculate historically for the whole history without any need of reprocessing data because all the data is there you just

apply some database operations to calculate this using the storage information that's like the basic thing for doing anything about it EOS as I said they directly generated from the blockchain state we have access to every single change in blockchain memory we have access to every action every transaction every call even to existing events we can build EOS on top of existing events think about transfer event you can amend existing transfer event and every time when transfer event is emitted you can emit Eco which has

exactly the same information plus additional things like token name token decimals receiver balance after the transfer sender balance after the transfer you can add multiple things just to existing uh existing event as I said they can be generated for history just by database operations which means it's relatively fast and cheap to generate them and because we have access to whole history we have full flexibility when in which moment of

processing we plug such Eco and what we include into this at the end of the day you get something that looks like events so artifacts that are precise precisely like hooked in time and have some Dimensions some measures that you can use for your analytics what's very important such approach gives us access to variables of every contract on the blockchain so you're are emitting EOS from one contract but at the same time you can queries storage of every other contract on blockchain and if you think

about it you can also build multi-chain EOS so you can for example generate swap Eco that has all the information that typically are include in swap event but you can add also onchain price and you can add onchain price on other chains as well because if you have everything in the same database and you can align those blockchains with time you can extend the content of Eco with information from other other chains as well so this gives us the dis ability of easily generating multi contract multi block

blockchain artifacts then then you can use for your analytics exactly as you do it with events today some very simple examples think about signer change Eco we trigger this by change in the storage in the hashmap that stores the list of signers for a wallet so we don't actually we don't care how complex is the wallet smart contract how many calls you might have how many events are generated we are looking exactly at one specific piece of storage and every time

when we change something in the signers list we emit Eco which contains information about time cender of transaction for example signer address type of change was it adding or removing the signer cost of gas whatever you want to add to this have full flexibility another example I call it whale transfer we look at the P token and we want to emit Eco every time when there is a change of balance for holder that holds more than

defined number of peper tokens so Define when you do it and what's the condition for generating such Eco and again you can include whatever you want like time function that was used to change the balance sender recipient amount sender balanced post transfer recipient um balance post transfer usdc price of Pepa onchain price of this token transaction Center gas price gas use sky is the limit so we believe that this approach gives

us completely new abilities because we can analyze the full blockchain data and we can get this information that's very important because analyzing storage is the core of many analytics today and what we do today think about it if you want to have the balance of a token in time what people do people take the millions of transfer events and aggregate them in time just to recreate the balance at any point in time with this approach you have access directly to the balance you don't have to

remember what types of operations can change balance you know very well that for usdt and for we it's not enough to take transfer events because you need to also take deposit and withdraw events to recreate balance so you need to have additional knowledge about how this storage is changed with this approach you work directly with storage it is it is much easier at the end of the day and much more reliable and also what's very important you see what you want to see not what somebody want to show you so this is this approach that requires very detailed storage history in preash form but this is what we have

today we have this information and on top of this we are building this additional layer that enables actually to use it for analytics in a way that's similar to well-known events so thank you and I'm open for questions thank you Tomic um any questions I think there's some right there thanks stach for for the talk um so my question is do I understand correctly that you have traces right so like you have to run your archive note and have

your custom trace and like I'm trying to understand how do you get the context for for this um storage change so like you identify moments in time when storage changes but you also need this additional context to be able to construct Echo so like do you take it from traces like that's why that's very that's very important uh part of it because when you execute transaction you you you execute transaction as a whole it contains inside the internal calls right you you execute them one by one and every internal call it may have multiple storage changes and state changes we distinguish storage and state state is like the eer balance nons and

storage is the uh smart contract variable so when you execute every call you have the potential storage Dives that are ordered and keeping this order and also being able to align this with the order of calls event of order of events gives you this ability that you know exactly where in this like processing stream this operation happened so if you for example generate your eco on the change of the storage variable you know which change it was like is in the context of this specific call this specific transaction so this is this is very

important to have not only access to this uh storage information but also this context as as as you said which means like which transaction it was which call it was what is the like the overall state of the whole blockchain and this is exactly in the data so every action and by action I I mean transaction call emitting event or changing storage it's ordered so we exactly know when it happens in this execution plan so yeah that's and and you can this is very important that you can use like all this information are not constrained with the scope of information that you can include in this

generated artifact another important Point here is that normally if you work with the node you get the you you can get the storage information at the Block Level so if you for example you have the get storage at RPC function so you provide the address which is hashed with keys and stuff so you need to know keys to actually do it and you provide block and it gives you the storage at the end of the block so you are blind to what happened inside the block with our approach as I said we have every single change so it's enormous number of information but it gives it you cannot

hide anything if you change something at the beginning of transaction and reverted this back at the end of transaction normally you will not see the change in the node because from node perspective nothing changed in this block we see that it was changed to something and then reverted back back to the original value and as I said everything is ordered so you can easily get this context for for every e probably answers uh we have time for more one question uh a short answer please no worries no worries oh thank you great talk I have a lot of questions but I don't want to borrow anyone so

I'll limit myself to really short one uh time and space how much space all of this stuff does take and how quickly can you answer a query for example I have a contract I have a storage slot I want to know uh the value of the stor slot at a given appointed time let's take ethereum as an example this is the the Big Chain so the whole processing because as as you mentioned this requires like the initial processing you cannot easily get this information you have to reprocess every single transaction and this reprocessing takes some time for for ethereum it took us around a months to process the whole history but we do it only once like if we do it once and we

do it well we never get back to this then we process head which is easy because it's not so much data and we update the database uh a few blocks behind the tip of the chain the whole database for ethereum today is around 100 terabytes I think and the speed of query ring of course it depends on the how much money you can throw of it because uh we store this in multiple databases and distributed yeah yeah uh and for example in Snowflake which is one of the distribution channels you

actually can Define what size of Warehouse the computing power you throw on this query so if you want to pay small money it will take longer if you pay big money it will take uh it will be faster but with reasonable size of the warehouse you get answers in seconds so this is what you actually need for for your analytics this requires some like good understanding how to build the SQL uh queries to do it and one of things to provide this EOS is to like do it for you so the the standard way of processing storage is kind of hidden for you and you are

seeing something that looks like events just includes the storage information uh in the events so uh it's a lot it's a huge data it's lots of processing but first it's doable and it's like in reasonable time it's it's not something that you cannot use for like uh uh analytics and and actually working with with this data and and it's life it's for few chains we we have it we are adding more and more chains and we can do it for every G Aron based chain uh we are also working on uh Reds so every uh blockchain that uses this

technology can be easily like processed just by adding this additional uh Tracer I see thank you thank you so much Tomic any other questions you can always approach Tomic we'll be around I'm here and we also have boats down there so we are open for any kind of questions thank you thank you