Efficient data query through dense subgraph with OpenZeppelin and TheGraph

Transcript

okay thank you so thank everyone for coming i'm adrienne kubwa i'll be discussing subgraph then some graphs and present you with a new tooling open zaplin is coming up with so stay tuned until the end so the first thing i'd like to discuss is graph density and this might sounds like a fancy thing for a graph series thing that some of you might have studies in university but actually it has very concrete consequences and the first example i'd like to present you to explain graph density is example of wikipedia and when you go to wikipedia there are a lot of links like a lot there are internal links to move around inside a page and there are links to other pages

and if you ever played wiki race or if you know six degrees of wikipedia you know that you can go from any webpage of any other webpage in wikipedia very quickly by just clicking on links and i think this is where the value of wikipedia is it's not just about the data in the content but it's about how the data is linked and the more links they are the easiest it is to be build queries and to run from one page one piece of data to another and this is something we also see in the blockchain space when we go to etherscan i'm sure a lot of developers are using etherscan daily and again just like wikipedia there are a lot of links from smart contracts you can go to any transactions you can go to any addresses you can verify

any every transaction sent from an address or received by an address and for the transaction you can have a look at what storage slots were modified who was involved sometimes which token were moved and you can go to transaction from transaction you can go to blocks from blocks you can fetch all the other transactions and so on so it's very easy to jump between entities and this has been very powerful to debug or to understand exploits better and when we consider sub-graphs density is also very important so subgraph for those who don't already know are the name that are is given to the indexing layer in the graph ecosystem and basically what subgraph do is basically they index data

from the blockchain and provide them in a structure way and this is very interesting because sometimes you can just query the blockchain to read the storage but sometimes you need more structure you want to know the history of a loan in maker for example or you want to have more data than what's simplistic queriable in in the storage state and this is where subgraph comes in and this is where you basically describe your application and your the way your blockchain operations perform and and create a you can create a ui on top of that and it's used by a lot of project code out there so obviously it's really interesting and there are a lot of sub graphs but there are very different qualities of

subgraphs and so here is an example of a subgraph that i found that is featured on the under on the legacy explorer where you see that it's about contracts about orders and about transfers but when you have a look at what's actually inside the different entities you don't have any links here you have an address which is just a byte the token orders are just strings they are not actually token ultra this is probably the address and the ids are probably the address we don't know that for sure but it's very likely but this means that from a token contract if we want to retrieve the top 100 holders well we have to to read all the orders which are a lot and are not sortable they're gesturing

so you can sort them alphabetically but it doesn't make a lot of sense and then you have to do one query per order and get their balance and then you will have to filter that and try to sort it it's a mess so please don't build this kind of subgraphs on the other hand you have these subgraphs and it's pretty similar you see there are erc20 contracts there are 1820 balances and if you want to have a look at the balances you have a list of objects and the objects contain the account which is the address basically and they also contain the value both in decimal and in exact format and so if you want to know the top 100 holders well it's basic it's really simple for any contract you can just filter the balances by value decrementally take the first

100 and the database layer will take care of that you don't have to do any other transaction in one single query to the database you get everything also you can read all the transfers and if we go further into this subgraph we know that there are events and there are a lot of events registered and for each event you have a transaction object and for each transaction object you have a list of events so just like on it scan you can really get complex linking into your subgraphs and if you go to the account object we saw previously it's not just an address it's a history of basically everything that's pointed to an account so if you know that this icon has a balance you can also get the list of all the balances for all the other tokens you can get the list of all the

approvals and you can get basically a lot of things and i believe this is the right way to build subgraph because this helps you to create a lot of links between your entities and it helps you and your users build better querying systems so basically my guideline to build a group graph and this is what you should leave the room with if you are trying to build your own subgraph is try to have as many high-level entities as possible like token balances loans if you are writing for example a subgraph for for a maker or similar financial system but also have low level entities like addresses even transaction and try to have as many links

between these entities as possible so that you really end up with a dense graph of data that is easily queriable one thing's also is that's more technical but when you are building a subgraph you can listen for transitions in because transaction record you can listen for events you can query the blockchain to read the state of a function at a particular point some features are more expensive and less efficient to do so as much as possible please try to stick to only getting events and only rebuilding your data from events and that also means that if you are building contracts and you think that at some point you will be listed on top of that try to think about what needs to be in

the event so that it will be easy for you to build a subgraph in a second so those are guidelines and it might feel difficult and abstract to to use them all but here it's what it allows you to do this is an example where given a nearest 20 smart contracts we are able to get the first 100 balances sorted by balance and we get the address and the value for each one of these we're getting further we can get for an account the list of all the erc20 contracts that that are indexed by this subgraph obviously with the details of the rc20 and we have the list of all the inbound and unborn transfer and for each one of these you have the transaction

with the timestamp block number and everything so you can almost recreate an explorer from that and if you're going further and if you are using open zip in contract you can even have a look at these erc20 contracts and say that oh this contract is also an access controller so i want to get the value of the list of all the admins admins are basically the one with world zero and for each of these admin i want their balances so very quickly you can have a deep look into the structure of your application who are the administrator what power they have what token balances they have and everything and this is basically what i've did in many of my subgraphs that you can find on the explorer so i'm known to i've been built

a few sub graphs and i try to use this design every time i build a subgraph which might be for a specific application like poap or just a meta subgraph like the one i have for erc 721 which lists this entire nft space in the in the mainly blockchain but what about you what how would you do that because that might sound difficult so this is where open depletive graphs comes in and opens up in subgraph is a new tooling that opens up in provides that you will be able to use for building your subgraphs and the idea behind the open the princip graph is i want to open the clean sub graph to be to the graph protocol what opens up in contracts are to smart contracts these are not the

be-all end-all of smart contracts but they are very effective pieces of software that you will be able to use to create smart contract complex smart contracts and to create the corresponding subgraphs and the idea behind open deployment subgraph is to have modules that matches the most commonly used contracts in open deployment contract and you can use them to be a tailor-made sub-graph that is dense that is effective for your application so let's have a look at what we can find in open zipline subgraph for each module you will find a schema of the entities so that's what we saw previously it's a schema of the database this is how you will be able to query your sub graph you have a data source template that you will be able to use so that you can very easily

create your subgraph manifest to decide what you want to index and you have the indexing logic that is written in assembly script we also have a pre-compiled schema with all the modules enabled if you don't want to rebuild your own because that can be a bit painful and the subgraphs can be assembled either manually or they can be created procedurally using this toolkit that i'm also providing it's external to open zeplin but it's working hand with hand we've opened the pin subgraph and note that i use the advice that i provided earlier meaning that all these modules that i providing only use events and as few function calls as as possible

so that i try to guarantee the best possible indexing logic when creating sub graphs so here is an example of of a subgraph and basically what we are going to do here is recreate a subgraph for the poap token and the poap token has a few features in it in particular it's a posable contract and it's on the rc721 and so from opendeployment subgraph you have a list of all the modules that are available so you can get just the erc721 template which is a yaml file and the same from posable and you have these two files with empty slots that you have to fill in like the network so you can put mainnet address the start block

and things like that and you will concatenate them and create a file like this which is your subgraph manifest and with this subgraph manifest you can just publish it to the graph and it will start indexing and you will be able to query it and the good thing is if you do it using the schema and just like other people have done you will have the same uh database language the structure so people will be able to create your subgraph and order subgraph transparently which i think it's a nice feature to have also what you can do and what i encourage you to try is use this tuning graph protocol utils which is even more powerful because what you can then do is just create a config.json file saying hey i'm on mainnet

these are five or six smart contracts i want to index these are their style blocks you can put zero here but if you put the right value it will index even faster and you're saying okay this is a known ability 721 this is a known able 1155 this is an access control rc20 this is a time lock controller so it's time lock plus access control and you could have as many module as you want for every address and then you just use this utility so graph compiler with this config.json so here i call it sample.json there is a mismatch but you understand it you include openzip in subgraph that's that's npm package and it will produce a schema that contains only the structure you need not nothing

more and it will also export the subgraph manifest and with these two files you can just send them to the graph either on the hosted service or under the graph net decentralized network and you will be able to start indexing and if you are on the decentralized network you can start creating an indexer will catch on that and your data will be available for everybody so yeah very quick word about open zipping in case you don't already know us we are a big part of this ecosystem we do audits but also we have a product branch obviously this is part of the product branch and this open dependent subgraph is part of open zipline contract but we also have defender and audits so basically whenever you are building a digital application you should be able to do basically any

step using or tooling at some point and so we will help you build your smart contract build your subgraph obviously we provide auditing services as well and we have defender which is another product we have that will help you manage your application when it's running you can trigger events like have notification whenever something happen on chains and hopefully very soon we will be able to detect events using this subgraph system that that is also available so if you're interested let us know because we also hire people to build amazing tools so yeah this is the links you want to go through we have documentation we have the forum we have the npm package and and thank you sam very much so

if you have any question i'm here for your erc721 subgraph for example so each erc721 contract is a separate data source in your subgraph and what if you want to add a new your c721 contract for example do you what's your process do you deploy a new version and then publish it to the network or do you wait for multiple your c721 to be on your subway to update it okay so this is a very good question and i

think it's a specificity of subgraphs is that data stores uh have an address field which is optional and so there are two things either you can have many data sources one per erc 721 or you could have one single entry uh with no address but that's going to under to index the entire blockchain so all 71 even the ones that don't interest you and the last option is in addition to data sources there is a template block so if you wanted to so i'm not really sure we support that but if you if you wanted to take this piece of code obviously the generating script will not generate templates but you could reuse these pieces of code

and this indexing logic uh for for for data sources and then you will have to build your own uh for example factory indexing mechanism that will get even from the factory and instantiate new data sources that's definitely possible to do i'm not sure my tooling is really adequate for that but that's a very good use case so i will investigate that to try improve your experience with that and yes if you if you don't use the templates if you just have like a few data sources if you want to add a new one uh you have to resynchronize the entire subgraph this is pretty painful that's why for cases like rc 721 i provide this meta subgraph that indexes the entire blockchain space i mean and at least mainnet that way if

the only thing you want is the rc721s and you don't care about access control posable and anything like that there is this subgraph here uh which does it already so if you have your c721 on mainnet uh currently or 1155s they're already indexed by these two subgraphs but in some cases people find it painful to target different subgraphs in the same application so that's where maybe for your application you want to build a custom design with only the data you want and nothing more uh hey um yeah i just wanted to add to that answer

um there's a compromise that we can do and that we did on karos curate which is to have a contract that emits an event and then whenever you just call a function then it's an event and then in the subgraph you can read that event to learn what's the address of the uh the contract that you want to to start tracking yeah yeah that would use a template mechanism and it works great except it will start indexing at the time the event is received so if you emit an event for a contract that already exists any past activity will be missed well i don't see any other questions so thank you very much and please try it [Applause]

out you