The Onchain Data Frontier - Mats Olsen

Transcript

you should be live hello hello let's see how do I get a slides up there Perfect all right uh for the longest time this talk was uh named TBD until this morning actually when I sent in my slides and it's now called the onchain data FR so um I'm matz co-founder and CTO of Dune hopefully you know what Dune is uh I moved to Switzerland in 2023 and I'm super happy to be based here along with my co-founder um our mission is to make crypto data accessible we founded Dune in 2018 today we have about 5050 employees we raised um most of our money at the peak in 2021 when things got very

crazy um we are a big web page in crypto and we have a lot of people who essentially come to Dune and know us as like the front page of crypto so millions of people come to Dune every month to understand onchain activity um onchain adoption um understand applications tokens all kinds of stuff on 93 blockchains I think it's the it's the number today most people don't know that dunit is also crypto data platform and that hundreds of the apps that you use use rely on Dune to get their data for their analytics and for their product stack

and recently we've delved into building developer tools so in November at duncon in bankco uh we released echo which is a developer platform that provides very low latency access to sort of curated apis for app Builders today we're going to talk about um the status qu for onchain data we're going to cover how literally every single app in crypto gets data how every Analytics tool in crypto gets data and then we're going to talk about what we think is next and what we're building so the aster here is that we're at ethereum STC so obviously we're talking about evm chains uh on

Dune there's about 50 of them in the world there's probably several hundreds of interconnected l1s and l2s and l3s and we're going to talk a little bit about the basics first and then look at the frontier so first um the way I think about the world is that blockchain nodes are really right optimized uh databases in order to build a new Financial system um to sustain the load of I don't know something like Visa or any popular app in the world you have to optimize for writing and so every single blockchain we use connects to a RPC node that's optim optimized to write quickly and unfortunately reading

data is clunky inefficient and it's bolted on as an afterthought the indexes that power the blockchain are there to power fast rights and in so far you can read data from the blockchain you have to read it um sort of in the format that it exists on disk to support rights so um in the evm world we throw around Json RPC a lot Json RPC is a technical term to describe a way for computers to interact but in crypto we mean a set of standard uh methods API calls that you can make um to send or receive

data virtually all data purposes in the evm world are covered by four methods so e call allows you to read data from an account e get blocked by number gets you to well get all the transactions and the block data from any given block get Block receipts allows you to get the logs so where things like events are emitted and then you have the special snowflake which is called debug Trace by uh Trace block by number which allows you to trace the execution and understand essentially um what a transaction does calls contract a calls contract B calls contract C this is this is virtually what the world relies on these four methods this

is reading if you want to read directly from the network and uh that goes for uh sort of apps and it goes for analytical tools as well there's a caveat here that there's some like convenience methods for like filter logs or something like this if you're building an app but generally speaking like the data that you get out of these four methods is what you have to work with so in the analytical world uh there's providers like Dune that essentially expose data that's all derivative of these four methods and the reason Dune exists the reason other tools exist is essentially because you cannot do an analytical query on top of a blockchain

node it's just not meant to do it it supports Point lookups by block number and that's it Dune and other tools allow you to count the number of uh daus uh on Unis swap that traded more than $10,000 every day this is just not possible if you connect to an RPC uh but dun and other tools um sort of transform data enrich data in order to provide something that's more useful but the way we divide our data in Dune is like we have raw data which is essentially the raw output of those the latter three uh RPC methods that I showed you we have decoded data which is where we take the abis of smart contract

facts and we decode um so that the data is no longer just arbitrary hex encoded bytes it's actually uh typed information that you can query and then we have curated data and this is where like 90% of the utility and queries that are run on Dune and other platforms are done at this layer this is where you combine raw and decoded data with metadata such as token prices token symbols I don't know all kinds of stuff to create things that a human can actually look at and answer questions from like Dex trades nft trades nft mints stuff like that and it takes takes quite a bit of work to get to to that point

actually just quickly glance over these data sets to show you what it looks like so here we have an example of uh transactions on top and logs in the bottom this is what you have to work with when you call these RPC methods and it's all hex encoded bytes so doesn't mean much to many people you have to have some understanding of like uh the smart contract that you are calling or that's emitting these uh logs um that's where abis come in U application binary interfaces allow you to take that binary data and uh convert it to numeric data um textual data Etc addresses um I pulled a slide from our

sales stack um in here so on on D you have like millions of smart contracts and events and generally speaking they're divided like this so you have divided by within the ethereum blockchain you have Unis swap B3 you have every contract and every contract has an event or a function call and together across the universe this is like 1.5 to two million like tables you can query on D so it's a lot but as I mentioned curated data is where most of the magic happens for most people and this is where you have to like combine um a lot of sources so for instance you cannot just go to a Unis swap contract um look at its swap events and know which tokens were swapped it's just

not possible you need information from the factory contract so at minimum you need to join two decoded tables now uh you probably also want to figure out what's the symbol what's the name of these tokens and so you need to get in more metadata it's typically uh collected from E Call Etc so all in all like data sort of Flows In A as a pyramid to produce this curated data there's tons of challenges in this Paradigm so there are hundreds of evm chance the number keeps going up and if you want to analyze an app chances are you want to do it if you want to analyze the revenue of Unis swap you don't want

to just do it on ethereum you want to cover all the markets for curated data like these categories they spawn over night like yesterday was Texas tomorrow it's nfts and in a week it's uh I don't know mean coins Etc like like all the like the utility of the data really lags um and even worse like when all these smart contracts are deployed we're at the mercy of developers that are emitting the necessary data uh in events so in an analytical tool like Dune and others you cannot really read the state of the of the blockchain at the time of a query like technically you you sort of can sort of uh build a tool

that does that but it's really clunky um in addition you can't really easily create data here so that's sort of the analytical world as uh Dune has entered the data for apps space I want to talk a little bit about that so the way we think about it is that applications need to read data um in a highly efficient manner like you need data specific to that app which is what we call protocol data so usually consists of uh event logs that are emitted from a set of smart contracts but you also need access to data about the account that connects to the app token balances for instance you cannot go to the blockchain and say give me all the tokens this account um or

this wallet has there's no method for it you have to rely on a provider um such as dune or build it yourself from from scratch then you have asset data and you have a myriad of providers such as coin market cap coin gecko Dex screener I don't know who sort of in different ways try to provide you with token metadata prices URLs Etc and all the apps need this like when you connect your wallet to Unis swap they need your balance they need to see your onchain activity they need the logo of uh usdc they need the I don't know the price of USC Etc like there's a lot of data that comes together to create a good

ux um protocol data um so essentially like data that's local to a Smart contract protocol one or more smart contracts together form uh a protocol uh generally used by an app um this field is sort of dominated by tools such as the graph or Ponder um which are Frameworks to create apis generally speaking it's all in typescript and it works exactly the same as dune and any other analytical tool for every block you call a number of RPC methods to get data except in this Paradigm the user is allowed to filter transform do side effects such as calling other apis

Etc um and then the data is eventually inserted into a database gives you an API on top um it's like either you use one of these tools or you build it yourself like there's no functional decentralized application that connects directly to RPC I remember we have the CEO of Aragon here Aragon is one of the first decentralized applications I used and it used to sync the blockchain into your local storage in the browser so in order to vote in your Dow you had to wait for your literal browser to catch up with the head of the chain could take days so some of the challenges that on

like um app teams face are that indexing tool today is they're limited to events and function calls so we're we're still at the mercy of emitting the right data in our events and whenever we need to make changes to our index or to our database typically you need to reprocess the entire blockchain and since you have a dependency on these like right optimized databases the nodes this is not very slow sorry this is not very fast it's very slow and and it could take weeks for you to get a new database that will power like Unis swap across multiple chains and then yeah the fact that applications are deployed across many

chains you have to manage connections to end different um RPC nodes maybe multiple API keys different payments you need to map your contracts maybe your uh contract on chain a is version 0.01 on chain B it's 0.02 it's just a little bit of a mess okay so that sets the state for stage for what I call the onchain data Frontier what is the onchain data Frontier well selfishly it's what I'm building at Dune and Adrien over there it's also from Dune so uh we believe that the future of onchain data is one where you're no longer at the mercy of the um sort of blockchain node that runs

on Alchemy or quick node or in your dap node um you're no longer at the mercy of the Json RPC API it allows you to go deeper we believe uh that the future allows you to read the variables defined in smart contracts any smart contracts that you can write your data access logic in solidity work seamlessly across the hundreds of evm chains do backfills on these apis in minutes and get real time apis without any hassle might sound like a pipe dream but we believe that the frontier of launching data exists in a paradigm known as simulation so what is simulation essentially simulation is

replaying uh blocks transactions and in our case inspecting them pulling out data at different points of the execution and doing this in a highly paralyzable environment so we're essentially building a future where you can inject codee into the evm without deploying a contract on chain and paying gas for it to emit any data point at any point during the execution not just an event not just a function call and we think it will revolutionize both analytical data that you'll use on d.com and application access to onchain data which we're building under the

developer platform Echo and essentially the way we think about it is that we're building evm Lambda functions so if you're familiar with uh software development you've heard of a Lambda popularized by AWS essentially a piece of code um that's called on some sort of trigger it's just a simple little code self-contained that does one thing and it's connected to some trigger so at dun this is powered by an instrumented hyper parallelizable evm that is incredibly fast and allows this Flex data access and so essentially what we've done is we've taken the evm forked it built our own and injected

code that allows us to inspect at any point given the execution uh what's going on this allows for like flexibility um that's not seen and since we're not participating in a decentralized network um we can sort of get the performance that application developers need so this is acquired this is technology that we've acquired through a company called small Excel Adrian over here is one of the Key software Engineers of that company you can try this on sim. it's live today for ethereum base and World chain actually so a little bit of a coincidence and while you should expect major changes to sort of the ux or the

UI of the product like the the core functionality of creating a Lambda and fetching data from any point during a smart contract's execution is still there so it is quite simple but also quite complex to explain how this works so but essentially you write the self-contained piece of solidity um that will uh receive a context with data about the step and um the transactions execution so it's similar to a debugger if you're a programmer so you create like a essentially like a little I don't remember what what it's called but you say I want to hook on this line and I want to see all the data at this point

in time and then we allow you to transform all this data read the state of the contract you're in read the state of any other contract and then emit the data so you can um do arbitrarily complicated things in here and it's incredibly fast but essentially you also need to Define your hook source so if you're Unis Swap and you want to act on like Unis swap pairs you could just select I want this to run not just on this address but any contract that implements the Unis swap pair ABI and then you can Define do I want to call this piece of code when a log is

submitted when a function is called when a variable changes it's not possible today um and then after this the data is submitted to our internal sort of Kafka topics and you can essentially do whatever you want with it pipe it into your database pipe it into our database build an API on top of it use websockets Etc so what are we building with this technology so what's next at the frontier so on the one hand side we believe that uh for analytical purposes product analytics for to measure onchain adoption as smart contract protocols become ever increasingly complicated we need the technology to

introspect and to instrument smart contracts and so we'll allow you to do this and write the data back to Dune and allow you to join with you know the two million other tables there so you don't have to create all the data you need you could just augment the contract and go back and join it with Dex trades or any of the other curated data that you use on Dune secondly this is new to Dune we're sort of creating a framework on top of this that you um work on in your own terminal in your own editor it's a local codebase workflow um in a way that allows you to declaratively say

um these are the contracts I care about this is the data I want to pull this is the destination of it and you can deploy it to our back end uh from a local CLI or through GitHub actions sort of framework defined infrastructure and you instantly have the apis your protocol needs uh fully managed and real time and then for us we're also building out these account apis um asset apis that we talked about a few slides back and we're obviously using this instrumented evm to make these more powerful essentially the way I think

about this is you can run any piece of code on the blockchain with without deploying a contract without paying gas without being at the mercy of a public or private variable um yeah so pretty excited about that and if you want to chat about it you can come find me after this um also uh mats.com pretty simple um and yeah thank you [Music]