Enter the Portal Network

Transcript

[Applause] uh hi my name is coob ML and this is uh Nick we both work at the ethereum foundation researching and implementing execution and consensus layer light clients so let's enter the portal Network so ethereum's kind of broken and I know that's kind of like chilly but it's more of a it's more of a problem of how we access ethereum starting out ethereum was very access ible you could boot up a full node in maybe like 10 minutes and it would syn the full chain but now it takes days to sync the chain and what oh I'll just continue

oh crazy okay anyways as you see it's a little blop um so it takes days to syn the chain and because of that most people access ethereum through centralized providers such as infira and Alchemy uh this is kind of like crazy because I guess centralized sources was supposed to be the thing we're moving away from ethereum was supposed to be this accessible thing where you just ran this program on your computer and you were able to send money but that's not really how how people interact with ethereum anymore so I guess how do people will interact with ethereum decentralized Le

right now and that's through full nodes as you see on this image on the screen right now to run a full node you need around like one tab of storage and that's really not accessible and that's also not including the knowledge required to run a full node it's like if we want ethereum to be used you're you would want your grma to use it and that implies that you don't even need to know what a full node is in the first place so it's like what are we doing so earlier I said to participate in ethereum's core protocol you needed to store around a terabyte of data what if we took that terabite and divided it

into a th pieces and put it on a on a thousand nodes but you could still look it all up on our Network and so if everybody only needs to store 1 gab or so of data to access all of ethereum this lowers the requirement so you could maybe store it on a phone maybe you could access it from a Raspberry Pi maybe you could access it from your laptop what if you could access ethereum without running a full node with almost instant sync times and being able to access the entire Json RPC which is how people interface with ethereum to

today oops but that's not all what if you also had full archival access to ethereum so I know this is kind of ridiculous because if anybody has ever tried to run an archival note before with gu you know that instead of a terabyte of storage you would need 20 terabytes of storage which is kind of Bonkers but the idea is that well if we distribute all this data and you can look it all up something like this should be possible so how does this network of distributed data tie back in the full

notes how do we improve full notes well there's this ethereum Improvement proposal named 44s uh currently it's proposed as being deployed in two stages the first stage would be Genesis to the merge as this is more of a solved problem and then after that is deployed it would there would be I guess the second half which would be the merge to around 6 months uh how does this relate to Dev P top the current way blocks are gossiped on ethereum well advertising blocks on ethereum would be shut off so your only way to access historical blocks would be

through other methods such as the one we're building building or something like I guess bulk archival formats which something like Arrow Arrow one files which is just like a file format which stores uh headers transactions and receipts in around 8,000 block chunks how will this affect syncing instead of syncing from Genesis you'll sync from checkpoints so and how will this affect archival nodes will I still be able to run one and the answer is yes instead of syncing from De P to P you'll just sync from a bulk block formats like R1 as I said

before so I guess you have this I guess it's like why is 4 Force important though it's because instead of storing a I guess a terabyte of storage you can reduce that by around 750 GB so you only need to store around 250 to 300 GB of storage uh chain data takes up a lot of your storage and it's not really used for anything because everything what's useful quote unquote is already executed and in ethereum's global State So currently old block history is mostly for like researchers and people who want to sync the chain from Genesis for now at least so I'm going to let Nick go over

on how does this crazy thing work all right so now I'll go into a little more detail on how the sausage is made poor or Colby explained basically the overview and I'll go into the details at its base we use a distributed hash table the same kind of protocol you'd find in bit torrent um this is how we address our nodes and the content around the network and how it's basically the way portal Network or the foundation of the portal Network um portal Network uses this exct same dis V5 layer that ethereum runs on to find nodes on the network um we use the same talk wck messages but Within These talk wreck

messages we embed overlay protocols um and these are specific to the portal Network and the best way to think about this is we have three different overlay protocols at the moment we have history uh overlay which chain headers bodies receipts and then state which is State data and there are two separate overlay networks but together they serve the data you'd expect to get from any execution layer client and then we have the beacon network data which provides um the data or that we use to sync with the head of the chain so this is a portal wire protocol in a nutshell this is the set of messages we use to find and check

liveness of peers on the network it's also how we discover new nodes around the network then another set of messages to um directly send uh the content that you want um between peers and then the final set of messages is used to gossip and broadcast uh the content being stored around the network so that it Finds Its eventual home or node that's responsible for storing it um so brief or just like a general overview if you think of each overlay Network um can be thought of as like a 256bit key space

and each of these little cir circles represents a node participating with a certain overlay protocol now these circles all in theory have different sizes you can't really see it but we have a big circle down here with a big uh radius and so every node has a Associated data radius and this is determined by how much storage you're willing to allocate to the network so if you're willing to allocate 100 gigs um you'll have a fairly large data radius and be storing a lot of data if you're allocating one megabyte you'll have a tiny radius and probably won't store much at all um so your node ID um addresses your node around the

keyspace and every piece of data that we're storing inside the storage network is basically a key value pair and to to generate the content ID we take the hash of the content key which also evenly distributes the content around the network so no single node is responsible for an unfair amount of storage and then basically like as a single node you're responsible for storing all of the data that falls within your data radius um and then when a peer uh wants a specific piece of data they calculate the content ID and they Traverse the network until they find a node that's close enough to the content and then they simply exchange the

content so what we've described so far far is basically just a uh database a large database and an empty dat database is not very useful so you might wondering where does the data come from and the data comes from a special kind of node called a bridge node and from the perspective of the network uh Bridge nodes operate like any other node they participate in all of the wire protocol messages um the only difference is every bridge node on their back end is connected to uh an execution or consensus or both um client and then the bridge node is responsible for pulling data from the ethereum

client translating it into the portal format and then using that offer message before to uh broadcast the data into the network and this is where the data comes from and then once it's in the network it's available and we have different nuances to our protocol to make sure that it's always available all right so we have data but like how do we know this data is real um we do this by using the different overlay networks and depending on the specific piece of data um there will be nuances to exactly how it is validated but the most generic example is let's say you start with the state Network at the bottom and you fetch an

account balance um the account balance will have a proof uh the data you receive will have a proof um attaching it to the state route so then after you have the piece of State network data you'll go to the history Network you'll go look up the associated header validate the proof against the state rout in the header but now you have a header and you're not sure if that's real or not and so then every portal CL will be running the beacon Network and we use the beacon Network um to sync with the head of the chain and in this way there's various nuances and accumulators and uh the proofs we use in the data types that are passed around

the beacon network but this is how we validate headers Arch canonical and this is how we establish a chain of validation tradeoffs uh so what's the catch we've described something very cool here but it's not always so easy um like with most things in life there are tradeoffs and in the trade-off between storage requirements and speed we're leaning heavily into uh prioritizing minimizing storage requirements so a a full node will always be faster than the portal network but the portal network is compar if you're using infura you have Network request so it's for the average ethereum use case the latency of

network request is totally acceptable for users um if you're not sort of the power user of an ethereum node then the portal Network especially with its minimal sto uh resource requirements should be more than enough for uh your use cases uh the current status of the portal Network so we have our 44s history Network Live um it's live right now anyone here can spin up a Noe participate in the protocol and fetch any data from pre merge history Network and that's cool but we also have a test net which we just

launched uh yeah a couple weeks ago and on the test net we're running the state protocol and the beacon protocol and we're sort of going through the final phases of testing the beacon Network and the state network is still in the earlier stages but uh we have a live test testet for the or we have live State data on the testnet as well um throughout development we have developed a uh Suite of testing and monitoring tools GLaDOS is our main public facing network monitoring dashboard and we'll show that in a second and portal Hive is uh our version of ethereum Hive and we use that for

client interop client implementation um we have fluffy which is written in Nim we have Tren written in Rust which is what Colby and I are working on we have shisui written in go and ultral light is written in JavaScript so if any of you guys are interested in getting your hands dirty all of these repos have issues you can go check it out and start contributing um right now even all right what's the future of the portal Network um well our two priorities right now are getting Beacon Network pushed to production and state Network pushed to production um that is sort of our MVP

story that we're really targeting by Devcon to have live State data available and validated using those three overlay networks we also are working on a um vertical support first state data so we'll be ready for the vertical transition and as always there are plenty of optimizations to be done to reduce use the existing Network latency um and then after that we're going to be working on uh flushing out the archive or really just filling the state dat uh network with all the archive data so that we will have this Global archive node that anyone can use and you know eventually we want to be in everyone's

wallets we want all the wallets to use this decentralized Pro uh provider to access the ethereum protocol and to try and wean ourselves off of infura Etc all right we're gonna do a little live demo and we'll see how it goes um all right so sorry I don't know if you guys can read this um but basically all I'm doing here is in this window I'm just going to start a trend node locally um you'll see it's connect right now it's getting its uh external facing IP to generate the

enr and then after that it'll connect to the boot nodes you can see we're bonding with boot nodes there um some boot nodes are yeah two's not bad um okay so now I'm just going to run a python script and okay so the content that exists in the history network is addressed by block hash so I'm using infura I know I shouldn't but I'm using infura to just look up the block has for a random block from Genesis to merge and then um if you see on the second half I'm just using web 3 Pi just like a generic ethereum web 3 client to connect

to Tren and then we're sending the standard eth Json RPC request with uh the block hash and we'll try try and retrieve that from the network and hopefully we're able to retrieve it from the network um so there we got Block 14 million something and so yeah so here's the state route from the Block uh uncle's rot receipts route transactions route cool so I it's not the flashiest demo but you know it's it's cool anyone can spin up a client connect to the network and you have all of this data available to you based on like this distributed

data data storage model we have um all right so just a little bit more this is GLaDOS as I mentioned earlier currently we're displaying the 44 stats and uh yeah I mean 99.8 over the past we're we have maybe a couple literally a couple blocks left to fill out in this network but the vast majority of 44s data is in there in the top in the top right you can see the different clients that I mentioned earlier in their implementations um the ones that are active in the network this is a gra well I'm not going to explain everything it's cool you can

go look at it and play around with it um this is our audit dashboard so we're auditing this is just a long running process that's auditing all the expected content keys and making sure that they're available on the network um I I picked an audit earlier there are more fun audits to look at but I wasn't able to find one but basically you can see this is a visual representation of how a node traverses the DHT to try and find a Target piece of content we have the originating node here it sends out requests for the piece of content to peers um two peers responded not with the cont or one Pier responded not with the content but with a list of peers

closer to the content that itself this pier responded with the same thing and then eventually we ended up finding the content from a peer uh I mean two hops to find the content um uh we also have a I mean I'm just sort of showing things from GLaDOS this is a network Explorer so this is a 24-hour snapshot of all the nodes that were seen on the network and their active um status based on the pingpong message um and as you can see some nodes drop off some nodes come back online um but we also have majority of nodes are just constantly available and kby if you want to

describe State network uh sure this is our uh I guess our test nut instance of GLaDOS running on our test nut called angel food which is a kind of cake uh currently we're auditing I guess state routs so our state route strategy basically I guess starts out a state route and does then does a random walk of the of the tree to an account if you make it to an account that's a successful I guess audit and if you don't make it then there is a failure So currently what we're working on is executing all the blocks so we can I guess get the the the intermediate like State diff so that we can gossip it cuz when you when you basically in a Merkel

Patricia tree when you delete a node and it regenerates if it's a branch node with only one child it converts to an extension node and we need to I guess properly track what is changed so we can gossip down the network and then execute all the blocks and then I guess the state Network's like done there you go just like that yeah just like that so I just need to execute all those shouldn't be too hard um cool so thanks for listening we've been working on this project for a while and we're really excited that it's live and available um please run a client if you're so inclined especially if you have validator somewhere running it's very

trivial resource requirements to spin up a node alongside and you're really helping the network um we have our specs on GitHub but really the best place to start is our website um eth portal.net we also have the GLaDOS website you can find everything from the website eth portal.net the best place to contact us to get help or just ask questions about how it's working what's going on would be our Discord server which you can also find on our website so cool thanks a lot all right are there any questions thank you

yeah okay uh thank you very much awesome awesome this network um you mentioned the data r i mean the the data radius I think so each and every note in the network has a maybe different radius and you mentioned I think 100 megabytes so this is this is all the client has in the end or or how much data does it have it's it's up to the client to Define how much storage it wants to contribute towards the network work so if a client only has 100 megabytes available that the data radius is D dynamically set based upon the config basically oh okay and that broadcasts to the peers on the network what data you can expect to find

from that client yeah okay makes sense and the end Vision then is that we have this portal Network and then you mentioned of course in the beginning that there are light clients probably running on smartphones and these actually have not that much storage so just they are they also contributing then to the portal Network or just accessing it you don't have to contribute storage to the portal Network to access the data but our expectation is that defaults are more than sufficient um of course people could spin up many clients with zero storage but the lazy thing is most people will just run the client with the

defaults and we also expect that there will be I mean yeah we expect that there will be plenty of storage because cumulatively it's not a wild amount of storage required for each Network um okay thanks any other questions hey thanks for the talk um I'm curious could the clients be used for transaction tracing as well given that they have access to everything they would probably need to run the VM and stuff but yeah is that an option have you thought about that I believe it's possible it might take a fair amount of network requests

but yeah all the data is available on the network so uh this question is not going to be very smart um but is it possible to to build a wallet that uses the portal network instead of an RP provider that's a very smart question uh yeah of course that's that's that's the goal maybe I didn't stress it hard enough but that is the dream um yeah so if it's a dream what's the main blocker there or what what's the what do you have to figure out in order for it to be reality uh I guess I'll take this so I guess the first thing is being able to like

execute all the blocks which is what I'm currently trying to do once we have a network with all the archival tree well that is looking all that up is pretty slow because you would have to walk the tree on the network so after that is we're looking to I guess upgrade our history our not our our state network with I guess uh faster lookups and we'll call this like I guess head state so instead of doing like log n look lookups so instead of having to walk the tree to find your account on like the latest like uh I guess instance of the state you could uh do like you could do a look up directly on the account you're

interested in uh just another follow-up question so RPC nodes are used for uh quiring data but they're also used for comput uh so would the portal Network do anything for the compute side of things like when you interact with a smart contract or anything there's some compute that needs to be done uh is it also part of the scope or uh I guess the way to answer that is maybe you're referring to something like an eth call yeah yeah yeah so you could do an eth call and you would just like run an instance of the ethereum virtual machine on your local machine and then you could just like I guess walk the tree or it or

if you're doing head State hopefully we have like I guess o of one lookups at that point and yeah all right thank you so much guys [Applause] [Music]