Forgetting History / Lightclient

Transcript

thank you so much for that introduction uh before I get started can I just get a quick show of hands how many people are running a full node okay yeah we're doing pretty good actually surprisingly enough I think that the crowd here might be more uh tending towards running full nodes I think the general maybe users of ethereum probably less so so you guys who are running full notes probably don't need too much of an introduction of why we need to think about forgetting history but for those of you who aren't running full nodes on a day-to-day basis let me just give you a quick run through about why this is even a topic so today if you take a look at what your chain database in your computer looks

like for go ethereum we're looking at about 761 gigabytes this is on a freshly synced node from earlier this week and if you run this on maybe you're noted at home today that's been running for a little while I might be a little bit larger post merge we actually have to run two clients now so not only do you have to store all of the information for your execution client you need to store all the information for your consensus client and that adds up to another 80 to 100 gigabytes so in total just to run an ethereum now today you need to be keeping track of 800 gigabytes to a terabyte of data and this is kind of representative about how many nodes that we end up having on the actual Network and so all of the you that raise your hand you kind of make up

this amount the 8600 which is kind of an order of magnitude I think I saw Leo's talk earlier from codex and he was saying that with their analysis they've seen about 12 000 nodes in the network so we're looking at you know around 10 000 people who are running ethereum full nodes and we would like that number to be so much bigger because the number of people who are interacting with ethereum is in the hundreds of thousands if not Millions by now and we would love for all of them to to be able to run a full node and have this trust guarantee conferred upon them and we generally have this hypothesis that the hardware requires as we were able to bring the hardware requirements down we're able to increase the node count and so that's kind of like why we're talking about how to remove the

history from a client we want to take the hardware requirements down from maybe one or two terabyte solid state hard drive to one day maybe just your mobile phone to be a full node on the ethereum network and so that's where we get this question that was raised a few years ago like maybe we don't have to have every single person store all of this information maybe we can have different mechanisms for storing the information and just have some of the uh just have the users who are running full nodes validate blocks that are coming in and that's represented in the ethereum roadmap so this is a diagram that vitalik posted in December I think of last year which was an updated one of some road map diagrams he's made in the past and you can see now that actually

two of the sections in the in the roadmap diagram are looking at how do we improve this situation with the unbounded growth of the ethereum node and you can see in this first one the The Verge the goal of this is was talked about a lot in guillaume's talk earlier today on vertical trees but we want to make it as simple and easy as possible to validate incoming blocks for the ethereum network and there's a lot of different mechanisms that do it one of these things is Virgo trees is an import is a very important part and hopefully one day we can just have a single snark that you receive over the Internet run some mathematical computation on it and then accept the latest you know that latest block as the latest block of the ethereum blockchain without having to

store hundreds of gigabytes and you know a huge State and and do a lot of expensive computation this talk is more focused on The Purge which is the aspect of looking at how can we reduce and remove the technical debt that exists in clients and how can we try and bound how much uh how much data a client consumes on your local machine and so I've circled this this area EIP which is around eip44s and this is kind of this original proposal about how do we actually go about deleting the history that exists in ethereum so this is like the motivation statement then we need to think about like okay how do we actually go about doing this and to do that we need to change our

mindset about ethereum and I really like this analogy that I don't know if it was vitalik or if if Dan crowd was the one who kind of you know initially came up with this but we need to think about ethereum more as a bulletin board rather than some sort of long long-term storage mechanism and the idea is that instead of relying on this for storage we think about it as I publicly post some sort of announcement to this board and I can have a very high guarantee that everybody who cares about ethereum or cares about these announcements was able to see that announcement and that's the only guarantee that you get and you can see this represented in some of the new uh some of the new proposals that are happening on ethereum things like eip4844 if you look at what the

retention policy is for for blobs on 4844 they're actually specified to just expire after about three months so this is kind of where ethereum is going we're trying to think more about um trying to think more about being this bulletin board where you can post announcements you can post updates to your rollups and everybody can see them but we won't make the guarantee that you'll always just be able to download that information from from the ethereum network and so this kind of means that full validation starts to go away a little bit and this talk isn't going to be super focused on full validation it's an important part but because we moved to proof of stake last year we've sort of given up a little bit on full validation in the first place because we have this

new concept of weak subjectivity and I implore you to go read about weak subjectivity if you're not super familiar but basically you you need to come kind of come up with some recent checkpoints some recent trusted checkpoint kind of like a Genesis in in maybe the past three months or so to trust as the kind of the root of your chain and so this makes it so that okay you can audit that full chain if you want to and this is an important thing we always want to maintain but it's a lot less important and useful and the reality is is that most people don't want to spend one one and a half weeks fully syncing their node to audit all these chains the you know majority way that people come onto the network is to Simply check point sync their consensus clients and then snap sync

their execution clients to the latest state that exists and finally we need to bound the growth like this is an important goal as you saw on the roadmaps of generally all of the things that can grow linearly in ethereum we want to bound so that whenever you start your ethereum client you have a really good idea about how much computation resources it's going to take both for bandwidth you know in terms of network and in terms of your disk size so this is the way that we wanted to think about it going forward so before I talk about exactly uh the proposal for for four fours and how we want to think about removing history I want to make sure that we're on the same page about exactly what I talk about when I'm saying history because I know that when I talk about this to people

even though it seems like we're on the same page we end up kind of missing exactly what I mean by history and exactly how that compares to things like state so the pieces of the history that that I mean when I say we want to delete the stuff are specifically the consensus objects things like headers transactions receipts and omers these are the actual objects that are committed to within the ethereum protocol and these are the things that we were trying to delete well we're not trying to remove in this proposal are things related to the state and in the state you have your accounts that stores your balance and your nonce and you have your storage which stores all the information related to to your smart contracts and then you have your smart contracts and so these things part of the state these are not part of the

set of things we're looking to to delete in this proposal and then one kind of overlooked aspect of the disk usage of your client is also the indexes and in go ethereum we have this index which stores the transaction hash to the block number and this sounds like a relatively lightweight thing but in ethereum whenever you have hundreds of millions of transactions this 40 byte value ends up adding up relatively quickly and so today we've sort of created a bound where we by default Only Store the information of that transaction hash to block for the last around one year and so that's kind of bounded it bounded it to around 25 to 30 gigabytes and we can see that if we actually look directly into

if we actually look directly into this output this is an output by with of a Geth sub command called gethdb inspect so if you're ever curious about what is the actual underlying data that's in your database like why is it so big you can run this command and it takes 25 30 minutes depending on your hard drive and it will actually spit out exactly uh um it'll spit out exactly how much data every aspect is using so we can look at this for history history you I've highlighted is green this is exactly what we said headers and bodies and receipts and that in total is almost 500 gigabytes of data that's just storing historical aspects of ethereum then we have state States takes up

around 243 gigabytes that's your contract code your uh account storage and your contract storage and also the try node so we're storing this Merkle tree and we not only have to keep track of the values the leaves of the Merkle tree but we need to keep track of the intermediary nodes that is another uh it's hard to see here 160 gigabytes or so and then we have our indexes and so like I said this index for the transaction hash to block number this is around 20 25 28 gigabytes and that one we we are currently pruning and so it's bound around 30 gigabytes it's not going to increase and we would like to be able to do that for history as well we would like to be able to say that whenever you start go ethereum the history for go ethereum is only going to sit around uh

100 gigabytes or 50 gigabytes or something like this so this is what our goal is by the way the time says five minutes like do I only have five minutes or is it off okay um so for the state the state is something we can't delete right now we have proposals of improving the situation but today like the state you have to have this to validate the block we can't delete it this is extremely important and so there's nothing that could be done hopefully one day we can have vocal trees to make it so that you can easily execute stainlessly but today like you have to have the state history we want to delete it that's the goal unfortunately unfortunately uh we have this prerequisite and we want to have a robust way of retrieving that history

forever and this is because we made this commitment early on for ethereum that you would always be able to retrieve the history from your peers the no the peer network of the ethereum nodes and to kind of walk back that commitment this is not something that we take lightly and so we're thinking really deeply about how can we how can we move ethereum forward and create these new guarantees that we want where we are actually able to bound the amount of State without uh totally giving up on this commitment that we made that the history is going to be retrievable and we thought a bit about this and so we have a few Solutions one HTTP providers this is extremely centralized the simplest solution but if you spread it out across many different

providers you start to have some uh resilience against people going offline or people trying to withhold data so in the future hopefully that people like infuria and etherscan and universities and you know maybe the internet archive they all are able to serve this historical ethereum information uh whenever you desire it we also want to use BitTorrent BitTorrent is perfectly suited for sharing in information this way in this peer-to-peer fashion and so it's a good complement for having these HTTP providers that are known and you know centralized entities and then having a permissionless system where anybody who wants the data can go get it and share it in this peer-to-peer fashion and then the portal Network and the

portal network is a really exciting way of accessing historical information if you want to access it in a light manner if you're a single user and you you know don't want to spend a lot of computational resources to participate you can easily get a transaction or a historical block but if you want to access like large batches of data the portal network is not the best solution you would want to directly download it via BitTorrent or HTTP or uh or something else like that okay that's how we hope to provide the data then the question is exactly like how will this data look and so this is this data format that I've been working on a fair bit and I've also been working on it with yasik from the Nimbus consensus layer team and we've kind of

come up with these two formats one for the data that exists in ethereum before the merge and one for the data in ethereum that exists after the merge and the question is kind of like why why separate these two things and the main reason is is that post-emerge you can kind of think about the consensus layer consuming the execution layer and so it would be difficult to describe all of the information uh in the execution layer after the merge because it already exists within the consensus layer itself because in a consensus layer Block in the beacon block there's now the execution payload so that actually includes everything that we need to both have the historical information for the consensus layer the beacon chain and the

historical information for the execution layer also post merge the consensus layer has a really neat value in the beacon State called the historical roots and the historical Roots is a list of hashes and each of those hashes represent 8192 Beacon blocks and every time you reach that threshold of you've gone through 8192 Beacon blocks then you created a historical batch and you mercilized that and then you put it into the historical Roots list and the nice thing is that the error format follows along very closely with that there are 8192 blocks in this era format and given an error file and some Beacon state that

happens after the error file that you trust you can actually verify the Integrity of that error file so all of these things fit very nicely within the the beacon chain and the way that the data is laid out in the consensus layer so pre-merge we tried to follow along as like closely as we could so we've got this Arrow one format and it looks something like this and this Arrow one format uh instead of having the historical Roots that's Dynamic and constantly growing we have a static commitment and the reason is is that we're post-emerge there's not going to be any more proof of work blocks that we care about and so we can actually say these are all of the blocks that ever going to exist before uh um that are ever going to exist in in the pre-merge

world and so we can actually say one commitment to match to all of those and it gives us this nice property that we didn't have pre-merge before if you wanted to prove something about a pre-merge block you would actually have to create a linear proof via the parent hash so you would have some trusted block where you would say this block right here is maybe I got from the Block hash op code and so I trust its value I want to prove something about the block ten thousand ten thousand blocks ago you would have to provide 10 000 headers to recursively prove via the parent hash now with the the the the era one format and this like static commitment the static commitment is just the Merkle root over a list of blocks and you can create a logarithmic size proof into that to prove that some block was it had

you know existed in the history from that static commitment so these are the two data formats that we've been working on and now you're kind of like okay when when can I delete this historical data in my node uh my one terabyte hard drive is running short on memory and any day now I'm going to get kicked off the network because I cannot continue validating ethereum and so the progression so far has been 2021 we had eap44s initially proposed and 2022 we had yasik working with the Nimbus team on the initial era format now in 2023 we've kind of come with this era one format and together with both the era and the arrow one format we

finally have uh a full kind of specification of all the information that could exist in ethereum and we're trying now to think how can we prove out these retrieval mechanisms that we talked about and so that's kind of the work that's going on now is we want to make it so that we're confident that that the information is accessible by these retrieval mechanisms that we've proposed and so hopefully 2024 we can finally start experimenting with pruning and maybe the first thing that we do is is just prune the data that exists to pre-merge and so that would be a good first step where we get to alleviate 300 350 gigabytes of data that already exists in clients so I would upgrade to a two terabyte hard

drive we're not going to fit it within one terabyte right now but it's something that's actively being worked on uh that's it for my slides I want to say thanks a lot for the organizers of Youth Prague it's been a fantastic event so far and I'm so excited to be a part of it and get to speak to you today if you want to keep following along with the work on four fours you can just take a look at the discussion link if you go to the eaps website and click go to four fours you can read The ethereum Magician's discussion link that's kind of like where the latest information is being posted if you want to follow along what's happening on ethereum in general not just four fours then you can follow the ethereum PM repo they post notes after the all core devs call and the

consensus layer call so it's a good way to quickly see what's being proposed for ethereum and what kind of decisions are being made in terms of actually changing the core protocol and then if you're interested in actually getting involved you can join The Ether ID Discord we talk about this stuff in the history expiry Channel and if this talk was introducing to you and you're interested in core development in general I would take a look at the ethereum protocol Fellowship applications are due June 19th you can find this if you go to blog.ethereum.org or if you just Google Syrian protocol Fellowship it's run by Mario hovel I think he maybe gave a talk yesterday about it it's a fantastic program we have lots of core devs who are mentors in it and we're seeing a lot of people who are coming through it like actually making it into the core

development space yep thanks a lot guys thank you client for the insightful talk with slightly tight of time so uh apologies to the audience no room for question but feel free to connect with speaker of the stage thank you [Music]