At the Intersection of Data Availability Sampling and Sharded Mempools - Leo BG

Transcript

[Music] Thank you. Thank you very much for the introduction. Um so good morning everyone. Thank you for attending uh this talk. Um the talk is going to be um mostly a kind of educational talk. We're going to talk about um scaling Ethereum and um I don't know if this is working. Yeah. Okay. So um this is more or less the outline of the talk. First of all I'm going to give a little bit of the background on scaling Ethereum how we started doing this research. Um this is going to lead to um the changes we

have done on the on the recent year. So the denim half hard fork uh the pectra hard fork that just passed about a month ago. uh and then I'm going to talk a little bit of what are then the plans for uh the coming hard fork which is um Fusaka uh in especially I will talk about PRAAS and then I'm going to talk about what is the proposal that we want to do for the changes we want to see in the peer-to-peer network uh for scaling uh beyond um what we have today and then we are going to end with some conclusions so um as I said the background of this talk is at some point we wanted to scale uh in at the end of 2017, we realized that um the number of transactions per second that uh Ethereum makes is not

enough to you know englobe the whole uh digital economy on on the blockchain. So we wanted to change this. We wanted to go um much much u higher in terms of transactions per second. And so the mission that we set up at that time was um we want to scale Ethereum uh several orders of magnitude beyond what we have today. Uh but we want to make sure that when we do that uh we keep the system decentralized and secure. Okay. And this is kind of a trilema uh because it's very easy to scale if you give up on the centralization uh but if you want to keep the centralization then the problem becomes much harder. And when I mean decentralization means that anyone can run Ethereum node uh with a relatively

cheap hardware and a low network bandwidth. Uh and achieving this is is quite is quite complex. Uh so first in 2018 there was this charting road map uh in which we were supposed to have multiple charts that will communicate with a main chain and and so on and so forth. This idea was changed later on 2020 to a rollup ccentric u road map in which the scaling solution will be based on roll-ups, right? And so since then since 2020 we started focusing on on on rollups for scaling. Um and this is going to be the way we are going to be able to achieve more many transactions per second and much more bandwidth and

throughput in in the whole ecosystem. But the problem is um the rollups need a space. Roll-ups need some space to put their data. Okay. So this leads to the changes we have done in the recent years that I'm going to explain here. So first uh if you are familiar with the EVM or with Ethereum virtual machine, there is a uh thing called cold data where you can actually store data in the blockchain. Now this is nice uh but it has three main problems. The first problem is that the data that you store there is going to be there forever. Okay, whatever you write with

cold data is going to be stored on the trion blockchain forever. So that's that's that's one concern because actually for rollups we don't need to store this data forever. It's just a temporary storage. Uh the other problem is that you have limited space only a few megabytes of storage per block uh in cold data. So that's not enough to cover all the rollups uh that we want to use in order for in order to scale. Um and the other problem is that it's kind of expensive storage. So you have to pay something like 16 gas per bite uh with whole data. And so that's also kind of expensive. And the reason is expensive is because you are storing this data there forever. Um, so that's a problem.

So to solve this, we came up with an idea. We invented Tinder. I mean blobs. But you can think of blobs like the Tinder of Ethereum. Okay. So if you don't want this long-term commitment that you take this forever, you know, if you don't want that, if you just want as knack, I mean as narc, if you want to store as narc for your seeker rollup um for just a temporary time, then you can use blobs and you with this you can really really solve all these three issues. The first one is that uh the data that you store in blobs is not there forever. It's not going to it's only going to be there for 18 days. So that's fine. That's enough time for the rollups to guarantee security and that's

that's all what we want. Now the other thing is that you have more space uh you have tens of megabytes even hundreds of megabytes if we manage to do uh what we want to do which is what I'm going to present uh today and it's also uh cheaper because it's an independent market right the gas the the the data that you put on the blobs is a independent market than the than what you put on chain on the layer one okay so uh because of this uh different structure we managed to solve these three problems using blobs That's the difference between call data and blobs. So that's all great. Um and so uh in the previous um hard fork which was called Denb uh it was launched in March

13, 2024. So we have been running this for all about a year and it has been running quite well. Uh the figures that you see here on your left is how many blobs per block uh have been used. This picture was taken around the time of Defcon uh in Thailand last year. So about 6 months ago uh the blue color is zero. So that means about 40% of the blocks have no blobs. That means we are not using the maximum capacity uh of the space that could we could have using all the blobs. Uh the pink part is uh the ones that are storing uh six blobs per block which is the maximum uh that we have on Ethereum or that we had with the Denip hard fork. And so that shows a distribution of um how the blob market

is being used or it was being used about six months ago. And on the right side you see um more or less the latency of the propagation of blobs over the peer-to-peer network. This is important because um in Ethereum you have to produce a block every 12 seconds which means that you have to transmit the blobs within the first four seconds of the slot. If you don't manage to do that, maybe the block is not going to be taken into account because you cannot guarantee that the data is available. And what you see here is that um the blobs from 1 to six are being transmitted actually within the first two seconds of the slot. Okay, so that's more or less the time that it takes to propagate all the blobs in the

peer-to-peer network that we have in Ethereum. So that's great. Um that that works very well. And about about a month ago we uh did another hard fork uh which is was pectra uh or electra prag. Um and what I show in this picture is how the blob market has changed uh before pectra and after pectra. So this is count this figure shows 10 days before the hard fork and 10 days after. In blue is before and in yellow is after. And on the x-axis you see how many blobs we are using. So before pectra the maximum was six six. So that's why you see six is the peak on the blue curve and there is nothing after because that was the

maximum in pectra. We changed the maximum up to nine blobs and the target is six. So now we see that the uh the yellow curve is pushing a little bit toward the right side uh because now you can publish more blobs uh per block. Uh so that's great. Now we have more blobs. uh this makes so if you're each blob has about 128 kilobytes in size. So if you send about nine blobs in a block that means that you have to download something around 1 megabyte in 12 seconds which is relatively easy with the internet connections that we have today. So that's all okay. Now I want to talk about what is coming tomorrow. So

the next fork is foaka or fusaka and uh what we want to introduce there introduce there is something that we call prdas. So pdas is the idea is basically we have to cut the blob data and make the nodes in the system to store only part of the blobs and not the whole content. Okay, for two reasons. First of all, we don't want to store all the blocks because that's too much storage. But also we want to reduce the name the network bandwidth that we use for uh accuring the the blobs. Now what is the problem of that is that if you only download part of the blobs you are not really sure that the blob the entire the entirety of the blob is available.

Uh and that creates a problem that we call data availability sampling and that's what we are trying to do. So we want to make sure that the blobs are available even when you only download one part of the blob. So how this is how it works. First the blobs are disseminated over the peer-to-peer network in the execution layer. The execution layer nodes download the blobs by pulling the consensus layer nodes extend the rows with a ratio coding using root salomon encoding and then the data is divided into 128 columns. Okay. And then every node in the network has to download four at minimum four of those columns. If you want, if you run a big node because you have many validators or because you have a block explorer or whatever, you can actually

decide to download all the columns and basically you will have the entirety of the blobs in your note. That is up to the note operator to decide whether they want to download four or eight or or all the columns. Okay, but the minimum is four. Uh so this is what we have today. And then we do sampling through column sampling. So we we have to sample the columns. uh the entirety of the column is disseminated over the gossip sub um um layer. So this is what we plan for tomorrow. Now and this we have been working on this for at least one year if not more. Uh and this is more or less ready setup. We have already test nets devets that are running PRA with uh

multiple nodes and so this is more or less kind of already settled. Uh of course there is still a bit of work to do but this is working already. Now what I want to present today, the new thing that I want to present today are some ideas that we can implement in the peer-to-peer layer to improve over what we have uh for PRS. So what we want to do is to scale up to 256 blocks in the near future. That's about 32 megabytes of data every single block. And this you have to extend it with with a ratio coding. So the real data is only the blue part of this square. you are going to extend it with resolmon encoding horizontally and then you extend it vertically so that

you are multiplying the the size of the data by four. So instead of 32 megabytes that makes 128 megabytes. So that's a lot of data that you have to disseminate over the network. Okay. Um so what we want to make is a system in which the execution layer nodes do not need to have all the blobs in the blob meool. Um and this comp the consensus layer nodes do not have to uh custody all the blob data as well. So there is sharding data sharding that happens both in the execution layer meool and the consensus layer custody and we wanted this to be easy on the network bandwidth for homestakers. So I propose we proposed

two changes. The first one is uh blob meool uh sharding system in which the node decide which blobs to download according to the hash. So every blob has a hash and you take your own node ID you hash it and then if the last character of the last half bite of your hash correspond to the hash of the of the last bite of the hash of the blob then you don't load it. Okay, so by using this technique, what happens is that the nodes will only download 16 out of the 256 blocks that will come with every block. Okay, this accounts for about 2 megabytes of data to be downloaded every

12 seconds. So that's more or less double of what we have today in terms of bandwidth requirements. But we are multiplying the number of blobs by you know from 10 to 250. Okay, so we're multiplying the space by 25 and we only pay two times in network bandwidth. Okay, uh in a network with 8,000 nodes, which is more or less what we have today, um that means that every node will have sorry every blob will be stored in about 500 nodes. Okay, so that's quite a strong resilience out there. So there is a lot of replication going on even if we start uh in this way and we can start implementing this

today. We don't even need to change the specification of Ethereum. We can actually uh already implement this and let nodes decide whether they want to download everything uh on the execution layer meool or if they already want to chart it and only download part of the blocks. Uh so what you will have as a result is a network that looks more or less like this in which different nodes depending on their PR ID and the hash of their PR ID they will have different blobs and then between the whole network all the blob data will be there and it will be replicated enough time for the system to be robust even if you consider that a large portion of the nodes could be malicious. Um so the second um proposal

that we have is what we call partial column dissemination which is today what we have is you have to download all the blobs and then you send the entire column over the gossip gossip subop network. What we propose is to be able to send a partial column not the entirety of the column but only part of it. Why part of it? Because only you have part of the blobs. you only have some of the blobs and then you can just send the parts of the column that that you have but maybe you don't have the whole column but that's fine you just send the parts that you have and we can implement this today as well I mean it's not very complex to pass from sending an entire column to only send cells that's very minimal change in the in the

network uh implementation and so what that looks like is like okay you have some blobs only the ones that are corresponding to your hash to the hash of your node ID. You extend those using reolomon encoding and then you disseminate the columns that you have to disseminate on the gossip sub network uh but only part of the columns. Okay. But then if multiple nodes if all the nodes do that then creating the entire column with the um you know joining the pieces of multiple nodes is extremely easy and then everyone in the network will end up with the entire column even if no node

in the network is sending the entire column. Okay, so that is um that is extremely easy and then once you get the column then you can extend vertically with using resolomon encoding and then you go back to the two dimensional structure that I presented before and then you have different nodes that have different blobs but they also have different columns like this and they all disseminate on their corresponding gossip sub network and everything works fine. We retrieve exactly the same struct that I showed initially. Okay. And what is good about this is that again we managed to have a structure that is two dimensionally eraser coded.

Uh the execution layer nodes do not need to have all the blobs in their main pool. The consensus layer nodes do not have to cost to the all the columns but only parts of the column. And this is quite easy on the network bandwidth even for homestakers which was the purpose um that we had uh from the beginning. So that was quite fast. Um conclusions uh if we compare the two systems uh on the left you have more or less what we are proposing right now for PRAS and on the right is PR plus the two changes that I just presented um right now. So on the one side you have the execution layer nodes have to download all the

blobs. This is what happens today. Okay. Um on the other side by introducing the sharding in the mele uh the execution layer nodes only download part of the blobs. Okay. So you are reducing network bandwidth uh stress there. Now the other thing is that uh in the mechanism that we had uh planned for PR you have a block the block builder or the block proposer um we are going to assume is the same uh for simplicity uh has to disseminate everything to the network. So it has to create the blob, do the ratio coding horizontally, then vertically and disseminate all the columns and rows to the entire system.

That means that if you are a block proposer, you really need a really good network bandwidth to be able to upload all the data that you want every time you has to propose you want to propose a block. Okay. And that is quite a high demand for homestakers because that means you have to be able to download sorry to upload 128 megabytes of data you know in the first four seconds and send this to the entire network otherwise the other validators will not see the data in time and will consider your block come missed come missed as missed sorry. Uh so in the other proposal here uh we only have to send the block proposer only has to send the

execution payload and the dissemination of rows and columns happen in the entire peer-to-peer network. The block builder does doesn't need to send the rows and columns because it's already happening through the partial dissemination columns that I just presented. Uh sorry uh on the previous proposal uh CL nodes have to download rows and columns. In this proposal the consensus layer nodes do not need to download the rows because they are going to get that get them from the execution layer. Okay. So you have a code that is called get blobs and you just get the same blobs that your execution client got um previously and it's pretty much you have to constantly

the same rows and so you don't need to download that those over the network. that is already good because you are already reducing a lot of uh bandwidth that is unnecessary and um the other thing is that um the consensus layer uh only have to download some um some rows and disseminate the partial column. So also dissemination wise is also efficient because you are not sending the entire column you only send the little parts of the column that you have. Okay. Now there are several options that are important to uh take into account. Um for example uh we can consider that the block builder uh disseminates the whole rows and columns but only on a best

effort approach meaning that is not necessary for him to do this really really fast because the entire peer-to-peer layer is already taking care of the dissemination of data. So he only does this as a kind of plan B in case the dissemination on the peer-to-peer layer doesn't work as fast as we want. So it's it's not like it's critical. It's not in the critical path. It's just a plan B in case the peer-to-peer network is having issues. Okay. So that's one thing. Um another thing that you could add to this um that strategy that I presented today is mele tickets. Mele tickets is an idea uh in which you only can introduce a blob in

the mele if you get a ticket and this is to prevent deny of service attacks or if you want you can um to avoid somebody trying to flood your mele with blobs that are never going to be introduced in the chain. So this is something that can be done. Uh but honestly with this system um since we are horizontally charting um and not vertically charting the meool is not really necessary. Uh you could use it but it's not really um I think necessary at this point. Now another thing that we want we could add to this um um you know to this strategy is uh the use of it one messages. Okay.

So once you receive uh the blobs uh on the execution layer, you can disseminate I don't want messages to you know stop receiving um the blobs that you already have and this is a way to reduce bandwidth uh as well. Many of the pro implementations that we have today in the consensus layer already have I don't want messages implemented. So we can make use of those in order to reduce bandwidth over the network. Um another good uh aspect of this is that um since the dissemination of the blob data doesn't start from one single point which was the block builder but it starts from everywhere uh the entire

peer-to-peer network is disseminating blobs and then disseminating partial columns. Uh this is really good for anonymity. Uh and this is something that we have proposed um in the past. uh we call this a pushpull phase transition. So let me just very quickly uh try to explain the idea. uh what we wanted is to reduce network bandwidth because we noticed that uh when you disseminate something on the treum network after five hops pretty much like 98% of the network already received the the data right so what we wanted is to say okay once you have done five hops maybe you can say you can stop pushing and transition to a pullbased strategy in

which you don't send the entire blob you say I have this blob with this hash if you are interested let me know and I will send it to you. Right? So it's a transition between pushing which is here is the entire blob to I have this if you need it send which is gossip. Right? So this transition is nice. The problem of doing this is that um every time the block proposer sends this data and send this with this is the hop number zero then everybody knows that your node is the block proposer is the block builder. And so that means that you could deanonymize validators in the peer-to-peer network which we don't want. Um and so uh that was an issue uh and that's one of the reasons we were

not really sure if we could implement if we wanted to implement pushpull phase transition. But with this idea that I presented today with partial dissemination we avoid this problem because now the data is not being disseminated by the block proposer but is disseminated by everyone in the network. So even if you receive messages with hop equals zero which is that means you are the first one to receive it uh it doesn't matter because that doesn't say that you are the block proposer of the block builder that you say that you have some blocks that somebody sent to you and that's fine. Okay. So that's one of the advantages of of this strategy.

And um yeah, another thing that um that is important to take into account is uh somebody asked me okay but if you are the block uh proposer okay and you only downloaded part of the blobs um how do you make a block now what you have to propose a block. How do you make a block if you don't see all the blobs that are available in the in the main pool right? uh maybe the blobs that you have are not that you know nice in terms of economics right maybe you want to make sure that the blog that you propose is going to have the best blobs and you will take as much advantage economically speaking and get as many rewards as possible um and yeah this is true but then what you could do is

because in Ethereum you know that you are going to propose a block in advance like an epoch in advance so that's six minutes at least in advance And you can change the way your meool works temporarily. All right. So once you know that you have to propose a block in the next epoch, you can transition to a system in which you say, "Okay, temporarily I'm going to download all the blobs, not only the ones that fit my node ID. I download everything just until I propose my block." So I have the list of all the bloss. I send the block. I propose my block and then after my my proposal pass I can just switch and say okay now I go back to the shard pool strategy that I have before. So I can temporarily change

this in order to propose a block uh locally. Okay so that's another thing that we can have with this um with this strategy and I think with that uh that comes to the end of the presentation if you want to see the work that we are doing you can scan these QR codes and thank you very much for your attention. Thanks a lot Dr. Leonardo from my end as well. So anyone of you got any questions so feel free to ask. We have few minutes that was too technical.

No one. Okay. So maybe I will ask you a very generic question. So what's your road map for the next five years? What exactly are you planning to do? Good. So uh the plan is uh as I said for the next fork for full Osaka we plan to have PR dust implemented and running there. So that's already data sharding at the consensus layer. Uh we are also talking about um sharding the mempool in the next coming months. Uh so what you could expect by 2026 you already have a system which is sharded in the execution layer and the consensus layer. This is for next year what we expect to have and the idea will be to have around 64 blobs if

everything goes well uh by fulu. Okay 64 is much more than today we have nine. So 64 is already quite uh significant. And then the next step will be will be what we call full dust which is two-dimensionally erased coding uh and hopefully 128 and perhaps even 256 blobs and that you could hopefully expect this to happen maybe in 2027 or something like that. By then we would have enough space and throughput for all the rollups to put the data that required for the ecosystem to scale to a much much larger number of transactions per second and throughput. That's that's the plan.

Perfect. That sounds wonderful. I wish you a great success for your journey. Thank you. Thank you. Thank you. [Music]