[BUIDL Asia 2023 Builder Stage] Firehose a Standard to Index Blockchain Data

Transcript

and it's after lunch which is like delicious lunch right I don't feel like speaking but you know let's do it no I feel like speaking uh how many of you guys know about the graph phrase of hands oh five people wow that's that's that's a great progress and that should be good enough for us to you know really understand about how the graph works okay I'll give you a little tldr a one minute thing so we talked before this talk we talked about the ZK you know stuff which is pretty cool it is one of the focuses of the industry to put information inside the blockchain while layer 2's why layer ones and to scale that but how do we get that information

from the blockchain to the user that's called indexing right and the graph actually indexes data if you have been a web 2 developer if you know how web2 works then you might know that writing information inside database is important and that's why ethereum is important layer twos are important all the polygons mean as cellos of the world are important but similarly if you're writing data inside the blockchain you need information to read that you know you need infrastructure to read that information from the blockchain too so the graph is the middleware between the blockchain database and your front end it's a middleware to get the information from the from the blockchain to your

world right and basically we have iterated through a lot of times to understand how you can get the information from the blockchain as of now the standard is you actually have a RPC you actually query that particular RPC and get the event data and event information but that's not very efficient because in smart contract you write events events make your particular transaction and your particular smart contract get a lot of gas right for example if your transaction would have costed two dollars it would cost 2.5 dollars so putting events inside a particular smart contract is not efficient first of all and the second biggest issue is that it's very slow if you want archived at

RPC it's very very slow it's very very tough and given that we are the biggest nerds involved in the blockchain we tried to like you know make make things more efficient because imagine if there are 100 million people on the blockchain you might be able to write information with all the layer to scaling happening but you will not be able to get the information and if you cannot get the information then it's of no Worth right it's like Instagram where you can post pictures but you cannot see the pictures so that's why we innovated on the RPC stuff and got this new indexing standard known as far host so how does fire hose Works how does the current graph infrastructure works it's like a ETL Cube extract you know then

you transform then you query it that's it etlq extract transform load and queue query extract transform load and query this is also linear way of indexing data you get the information from the blockchain transform it into what you want then load that information in database and then query that information extract transform load and query but that's not really a very scalable way why because this is very linear if you have an event which has a lot of data you will have to go through this linear path of first extracting the data and transforming then the loading into a database and then querying that information how do you make this efficient the answer to for all the computer science

graduates is parallelizing the process extraction should be parallelized loading should be paralyzed and all of these things should happen in one go in one thread right it could be multi-threaded infrastructure and that's what the graph actually built uh so I already talked about what are the problems with the RPC but the solution as we talked about is fire hose for extraction you need if you want innovation in indexing you have to innovate in all these four processes extraction should be much more efficient transformation should be much more efficient loading should be much more efficient and then querying that information should be as efficient as a computer science person you have to parallelize these process

and each process needs to be very very very very nicely efficient in all the ways and right now the extraction is not efficient the rpcs the inferors the alchemies of the world are not very efficient and the reason I told you is that they rely on event indexing to get you the need that's required you cannot do that if for example if you want in information about an crypto Kitty event you did in 2017 that RPC will be calling all the events of the cryptokitty smart contract from 2022 to 2017 that's like five years to get you that information that's not efficient so the answer to the RPC latency issue and the cost

issues and the performance issues is fire hose what is basically Firehouse Firehouse his flat files instead of using RPC to make a blockchain data available to you you use something like fire hose to make that data flat so you get still get the information from a file from the blockchain data but it is very very flat what are the benefits of our house it is basically a flat file you get you run a blockchain full node let's say polygon full node get the information and store it in flat files it enables parallel Computing like we were talking about etlq is all parallelized and the last

one is that it's blockchain agnostic so right now all the RPC stuff we have is all evm right but there are other blockchains like near which are not evm there are blockchains like Solana there are other blockchains in the world which are not evm how do you create a standard for them do you create a indexer for all the new chains that up and come no you make sure you have a database standard so that so you don't have to keep and keep on you know innovating and creating new indexes you make sure your indexer patches information from a generic data and that's fire hose so basically if you are creating a blockchain you don't have to worry about indexing all you have to

worry about is making your blockchain compatible with fire hose so basically a particular blockchain anybody anybody can be compatible with the graph if they are compatible with fire hose they don't have to create a new indexing standard if they are compatible with farhos so how does the firehouse architecture actually works right it's a full node which reads information from a reader reader takes the information from the full node right and then it creates a single block file that single block file is either consumed by indexer if it's the latest information or it is going inside uh like you know the object store

for getting the information stored right that's how flat files are created you have a block that's created that is converted to a flat file that flat file is either used by indexer or it is you know first of all the all the useful information is taken and the flat file is deleted that's how the architecture of firehose works and to make your blockchain efficient and compatible all you have to do is you have to just make your stuff compatible with fire hose basically you have to make your blockchain let's say near blockchain compatible with the reader that's all that's required what are the benefits of our host why would you want to do it because you want the graph to index your blockchain you

want an indexer to get your information from the blockchain to the front end right and we have had so many inquiries of let's say near and you know Solana and all those people who want their information to be indexed by the graph standard up till now we could not do this because we were relying on rpcs for getting the information but now we can because we have created this new standards which is not rpcs rather it's fire hose so with farhos you can get your blockchain compatible with the graph first of all secondly it can power substreams right and I was talking about the etlq extraction is fire hose transformation is substreams

transformation is the one which makes your you know information more efficient you have the raw information already how do you transform it into what you want that's done via something called as substreams and the results we have got from substreams are crazy up till now pancake swap uni Swap all the big dexes used to exist on the graph but we had these quick like you know complaints from all the fast chains and for all the fast dexes that the graph is not fast enough if it's not fast enough it cannot keep up with the blockchain if it cannot keep up with the blockchain it takes a huge time to sync so right now if I start on the graph node and start

indexing uni swap sub graph like a thing from scratch it would take two months until I can get the information that's not efficient in two months anything can happen in web3 that's not the right way to make things work what's better is basically you make something like substream with substream and the fire hose combine the parallel etlq model we were able to make the two month indexing the two-month data retrieval processing all these things happen in 20 hours and that's what the graphs new beauty is you have fire hose you have sub streams to get your complicated fast chain information indexed from two months in two hours so

all we require all the chains is to integrate with firehost architecture and they will have the fastest indexing in the world so you need if you have let's say Stockwell right you have a starkware node and you want to make it compatible with the graph what you do is you enable your starkware client with farhos right that can actually take information via graph node and give you the right information in API key that's required as of now we have already started building on it it's not you know something in stealth that we are doing it's already live in production and these three beautiful like you know dabs run by misari are being indexed by the graph

so why are substreams and fire hosts so uni swap has a misari uni swap has a sub graph which is being indexed by uh like you know and queried by the misari team to show their analytics page for all the data that's that has a substream sub graph right so it's already in production it's already live and as I told you we did this experiment of indexing blockchain data with uni swap with our current subgraph standard and then with the substream standard and with the sub stream standard we were able to do it in 20 hours so that's that's what we think is the future it's already live battle tested on the hosted service to try it out you know the whole all the hosted

service ethereum subgraphs are fired by are powered by Firehouse so you should all try it out CTA or call to action for all the chains that exist in the world right now is that make fire hose a standard in your graph in your chain client node so if you have a new chain coming up which is going to be the future of web3 I mean that's what most chains claim when they come up and uh like you know when and it might be right we want to believe in you if you want to do that if that's what is what you think will be the future then have fire hose integrated in your client so that it is compatible with graph node and we can do the indexing for you for your dabs so that's that's about it this

is how the architecture works I don't want to bore you into like you know the details of the stem but uh yeah we already have some progress ethereum guys uh one of the core teams which is chain safe which also maintains the Aragon client right now has worked with the graph Foundation to uh like you know have the Aragon implementation of fire hose so basically the Aragon fire hose is right now live and you can see it on the graph dashboard as well as well as in this particular article but we have already made progress with ethereum and evm chains we are already talking to uh there is multis which is another you know important chain in the European region right now and they're pretty

predominant there they have integrated this fire hose and there are few others that we are talking to right now which are evm who want to integrate with firehouse and they can do it like you know with the current infrastructure uh that's about it that's uh far host standard for indexing blockchain data if you have any further questions you can either ask me on telegram or tweet about how good a speaker I am on Twitter it's time to clap [Applause]