119 - Dive into Danksharding | Vitalik, Dankrad, Protolambda, Tim Beiko

Transcript

hey bankless Nation Welcome to our special live stream this is going to be a panel Edition I think David this is going to be our dankest panel maybe the dankest episode we have ever recorded because we are talking about dank sharding today and I gotta be honest going to this episode I'm not entirely sure what it means like what is dank sharding but we'll ask uh some of the participants this and maybe David you could describe the setup of this panel and who's on it and how we're gonna handle this episode yeah of course you guys all know Tim Bako he's the guy on the bottom of the screen Tim manages the all core devs call and is coordinating the lead into the merge and Beyond and we've had Tim bakeel on before during he led a very technical eip1559 panel uh and ask the questions

that Ryan and I are just not smart enough to ask so this is one of those panels we're gonna get as technical as possible we have three fantastic panelists who are behind the scenes who got vitalik dank around in proto who are the minds behind Deng sharding and charting in general and other ethereum related Technologies and Tim is going to be able to ask questions that are the technical questions the smart questions but before we get there Tim we just wanted to cover some high level stuff what is dang sharding how to get its name and just like what does it mean for users right well first yeah thanks for having me on guys um so bank sharding and proto-bang sharding which we'll also get into our iterations over the sharding

design for ethereum um and we'll spend you know the bulk of this panel discussing what they are what the trade-offs are and whatnot but at a high level charting is a way for ethereum to have more data pass through the network and because Layer Two Solutions uh like ZK Roll-Ups and optimistic rollups and they produce a lot of data if we have a way for them to post that to the ethereum network more cheaply uh it immediately reduces how much they need to charge users for transactions so all of these kind of flavors of Chardonnay all have kind of the same end goal which is to create a cheap place for layer two solutions to post data on ethereum and the impact of that is that the transaction fees that

end users and the pay on layer twos is uh is lowered by a lot so this is all about getting transaction fees down particularly on layer twos gas fees down on layer twos and and Tim is this related you mentioned Proto dank sharding and dank sharding you guys will talk about all that in the panel but is this related to eip4844 because that's another EIP we've heard a lot about yes so eip4844 is proto-bank sharding basically and one way to think about it is like proto-bake charting is like maybe the first step we get towards Chardonnay and then big sharding is the simplification that we've had on the previous roadmap so like the the more prefixes we have

it's like the sooner we get them um yeah all right and this has been like you said an iterative cycle an iterative process to get to where we are today uh and dank sharding comes from dankrad who's on the panel and Proto dangsharding comes from Proto uh Proto Lambda who's also on the panel so these guys names have been baked into the the name of this EIP itself Tim is this the new eip1559 it's eip4844 going to be the new EIP that we focus about going into the future so there's a lot of stuff we're working on right now so you know it's definitely one of the big ones um and I think for end users it's one of the most impactful ones because it directly affects like the the gas price

they pay um hopefully it's much less contentious than 150 5559 was um but yeah it's definitely an EIP you're going to be sharing more about over the next few months and just one last question before we hand it over to the panelists and we bring the panelists online uh there's been a bunch of conversations about you know when eip1559 came and go people were like is this going to reduce gas fees and the answer is no uh and then people are like uh the ethereum merge is that going to reduce gas fees and the answer is no this EIP reduces gas fees this reduces transaction fees not for the layer one but for the layer two correct yes on air twos yes that's correct um and basically 4844 is a way for us to get some of the reductions of Chardonnay quicker and then the full dank charging

rollout gives us even more reductions uh but because 4844 AKA Proto dang Chardonnay is simple to implement uh we can just get that first fantastic all right I think that is all of my questions and I think that's all of Ryan's questions so with that I'm going to ask the panelists to come in from the shadows and turn on their their cameras and me and Ryan are going to actually duck out of here excuse me we're gonna dank out of here and let Tim take over this stream uh guys good luck guys welcome to the panel and Tim thank you for doing this and then absolutely just take it away awesome thanks um okay sweet uh it's just us now um so I guess yeah but before we get into it uh do you each want to just take a minute and kind of talk about like

what you work on and who you are um yeah vitalik we can start with you um yeah so hi um I'm vitalik I'm the co-founder of Bitcoin magazine um I write a blog I contribute to Specs once in a while uh thank God yep hi I'm Dan cart I'm an ethereum researcher since uh 2019 and um I'm working on uh among others starting and um would also work on proof of custody um statelessness there are some some projects on the on the roadmap of ethereum nice and Proto

hey hello I'm president um I used to work at ethereum foundation on Research there now I do the same thing but I'm not optimism I helped with shouting earlier on and now I'm contributing back to uh later on while working at optimism and Layer Two sweet um okay so just to kind of get into sharding generally um over like the past couple years what sharding means that ethereum has changed a lot um and I think like the biggest one is like this shift from Full execution charting the only data Chardonnay um Vitality kind of do you want to give us just an overview of like how that shift happened in their research roadmap

and why we've landed on just doing data chardinia yeah so I think there's been this ongoing simplification of the shorting road map that started released sometime in 2016. uh so for people who have been in the ethereum ecosystem for a long time you might remember some of these scaling docs that we published back in 2015 back in early 2016 some of the blog posts thinking that I came out in 2014 and the stuff that was there in those earlier periods was in a lot of ways really complicated uh right um like there were these ideas around hypercubes and uh Hub and spoke chains and uh in protocol supported uh

cross-charge transactions that would be routed between like one corner of a hypercube to another corner and where the protocol would help them move from like one short to another to a third to a fourth and there was uh even thinking about super quadratic shorting which is basically saying like instead of just having shards you have shards on top of shards and potentially like in infinite hierarchy of shorts instead of shorts inside of shards so actually the sort of stuff that the telegram ton project ended up by incorporating into their paper though I guess that it never really ended up coming close to going live unfortunately um but that was the kind of thinking that we had back because 2015 in 2016

and I think after that the progression has just abandoned this like big slow process of I guess increasing pragmatism um increasing appreciation of uh how complicated it is to develop and actually bring to production just about anything um like what feels like 10 lines of code actually becomes like hundreds of lines of code once so you add all of the complexities that clients have to inevitably have and deal with you know how some particular thing interacts with the syncing process how some particular thing interacts with the fourth choice how in some particular thing and interacts with the database um and the need to store it in optimized formats and that sort of stuff

so the process of simplification I think the big First Step was definitely the decision to not bother with anything beyond quadratic sharding and just say we're doing quadratic shorting right so not bother with uh with doing ever any kind of shards on top of shards and just saying we have to Beacon chain we have shards Shard headers are connected to the beacon chain and that's it like that's the only layer of shorting that ends up actually happening so that was the first step then the second step was the move from this uh concept of like chains that have regular commitment blocks in the beacon chain I

think there was another word for them I forget what what what what that word is across mics was the word all right you have short chains that cross when it gets in the beacon chain all right so move it from that to a system where you just have every Shard block directly getting quoted in the beacon shade so that was the second simplification I forget exactly when that happened I think that might have been around 2019 or 2020 or so and the big benefit of that simplification is that it meant that we didn't have to worry about short chains uh anymore uh then after that uh there was uh the uh idea that we're going to do data shorting first this was

when they started talking about the roll-up censors roadmap right basically instead of Shard blocks actually containing transactions that would be executed at the ethereum layer these short blocks would just contain big blobs of data and it would be the responsibility of layer 2 roll-up protocols to use that data space in order to create secure and more scalable experiences for their users right so ethereum the system would provide non-scalable computation and scalable data and what a rollup does is it converts scalable data and non-scalable computation into a scalable computation so we have a somewhat more performant layer one that has extra data

space and then we combine that with this layer 2 ecosystem and the layer 2 ecosystem ends up like really bringing all of the yes scalability to life so that's the role of centric roadmap and at the beginning I think the role of centric roadmap was phrased in this ambiguous way where it basically said well look the real obstetric roadmap realistically like data shorting is the obvious Prelude to full sharding anyway right like if we're going to implement full shorting with evm's little shorts it's just an obvious first step to have data shards first but it serves out the data shards uh are actually really good for Roll-Ups already uh and so we might as well run with that we might as well realize the Roll-Ups or best hope for short-term scalability and and just take

that direction and try to make the best of it right and that still leaves open the door for adding evm execution shorts in the future but it basically says well actually um you know ethereum will be fine even if we end up never actually completely doing that right so that's role of centric roadmaps it's another simplification basically saying we don't have to bother with execution on shards and then that also allowed some other simplifications like it made it even more possible for shards to not bother with Fortress rules for example and then the next simplification after that is dank sharding where basically said that there is actually this bullish proposal mechanism where there's only one

proposer that chooses all of the uh short blocks on all of the shards that appear within a particular beacon block and that simplifies things massively in a whole bunch of ways so it basically means we don't have to deal with like the whole short proposer bureaucracy which simplifies that consensus complexity a huge amount it simplifies some of the economic properties a huge amount it basically makes the system feel much more similar to like just what a non-scalable chain would look like except it's just more scalable right and that's extra scalability happens in the background and then Proto takes short and finally like that's not a simplification that's more a step on the way to full texture that gets us maybe

half the benefits of shorting but at a points that's maybe you know kind of like halfway along the timelines they're actually getting full sharding out there so we actually get some of the benefits sooner right so that was the General progression like basically more complexity and less complexity more um more of ethereum trying to do everything to less ethereum I'm trying to do everything and more willingness to work with Layer Two protocols um and those two things together and that's where we are now yeah thanks that was that was comprehensive um they grad like talking about thanks Charlie like can you kind of walk us through how like this idea that it's okay to assume that the block proposers or the block

builders have to track all of the shards because of the separation between proposers and builders that we've seen kind of over the past few years especially with the rise of Mev um so yeah just talk us through like why is it possible to do something like big sharding and not sacrifice the decentralization properties of the chain right um yeah so I guess I mean uh maybe like if if we go a bit into the the history of Mev like or maybe think about um how how it has been recently So like um it started with maybe like maybe some mining tools um doing some stuff like to exploit Mev

and like to like uh yeah to to make to get some more than just the transaction fees and um over over the course of time this has become more and more professionalized like nowadays most of them work with some other entity for example flash flashbots that um sells them complete bundles of uh or yeah maybe selling the wrong word here that buys complete slots for bundles to be included in blocks that exploit a certain amount of mov and then uh and then the block producers which are currently the miners they just get the payment for that they get like um so uh yeah so there's the the searches are now like lots of

independent entities that uh try to find the best Mev and um and Mining pools now or don't have to bother with us anymore and um and basically um it turns out that uh if we want to properly decentralize this which is something we really want to do like um so right now there's like maybe um a few mining pools like um tens of them or so and so like the way it works right now is that uh basically flashbots has a business relationship with each of them like they they basically have a trusted relationship and um if like uh one of one of the two sides did something naughty like for

example they um The Miner could start looking into the strategies and instead of um just executing them they could like exploit the strategies themselves so for example if you have a an Arbitrage transaction then like you can often uh do much worse things with the individual trades if you if you want to exploit them um or like they could try to figure out what the strategies are by by what they are being sent and so on um so um this this doesn't scale to a system where instead of like a few tens of mining pools we have uh probably tens of thousands or so of individual validators um because you can't have like a trust

relationship with each of them that's not gonna work um so the only way to translate this world of Mev into um into the future of proof stake is is by having some form of proposal Builder separation the way proposal Builder separation works is that instead of having uh traditionally like I guess like five years ago we all thought of it as the same thing if you propose a block you build that block right and with proposal Builder separation that's not true anymore what we do instead is uh someone a builder builds a block and the proposer just proposes oh yeah I'm gonna propose that

block that the sky built um so we separated the two roles and um and in that way we can have a professional role of block building building which is uh this role that will extract Mev or work with all the Searchers and so on and we can have the proposer and that is just a normal validator and the good nice thing is proposing is extremely simple and cheap because it's basically just selecting the highest bid and saying yep you get to build the block and uh whereas building is a complex process where you have to manage a lot of searches and they have to trust your system that their strategies won't get exploited and so on

um so that that is more suited for a complex and more uh capitalized entity and so um um it's it's good that not not everyone has to do it and uh basically this is uh this is the I guess this is how we are seeing the future of block building on ethereum there's not there's not really currently a viable alternative to this world um and um and what we also know uh is that um sort of building this uh building scalable system especially also building this massive data availability system uh

becomes a lot easier once you assume that there is someone who can handle these massive amounts of data so once you put that entity into the system and say well um there's someone who can compute this encoding who can distribute all this data and so on then many things become a lot simpler and so in the past um I guess we didn't really think about these designs because we were like well we want ethereum to be extremely decentralized um and now with the with this proposal building set builder separation coming into the design space due to Mev it's also become available um to think about for other things and

basically this is how we or like I first thought at the end of last year well let's use this let's use this idea where you have these uh entities that can handle for example large amount of data it's not really a problem if you are like running some large machines anyway it's not like an absolutely insane anymore it's not data center kind of amount it's just like large machines with a good internet connection kind of amount and um and uh ex yeah exploit these entities and uh and let them basically do this building and that allows us to uh get to a much uh simpler and more efficient charting design got it

um and so am I right and thinking like because it's basically very hard to build an optimal block that becomes a specialized industry but once you do have a block that is it's very easy to verify its validity right like so right you know finding the exact right block to build is hard and so you need you know tons of machines to do that but once you have found it then anyone can verify it it's almost analogous the proof of work in that way uh for like yeah yeah yeah that's absolutely correct like basically verifying I mean that's that's the that's the Crux of um data availability checks the this idea that um there there is a way to check that this amount of data is available that

needs much less work than actually downloading all the data so there's this asymmetry um where like there's someone's somewhere we need to do this work of encoding the data and distributing the data but then verifying it is much easier got it um okay and to get us all the way to Prolo day charting now um so recently the three of you have have written an EIP ip4844 that's now colloquially known as Proto Deng Chardon um which helps kind of lay the foundation for this full sharding design um without for cry without requiring this entire kind of Shard Data Network to to be live

um Proto do you want to walk us through like what are the things that 4844 does to get a steward sharding and also like how does that help layer twos like how do l2s then use 4844 to provide their users with lower transaction fees right so we just ended with our bank sharding basically introduced the data availability sampling and this other new Advanced Tech features to try and distribute this job across the network better but this comes with additional complexity will work take more time to properly task to introduce to ethereum so instead of waiting for the full

version of bank sharding we can reduce the feature sets and go with a amount of data in between these things we can offer additional data and we can look at layer twos like what kind of security properties they need and optimize products and they can already make a big vendor and then later with the additional features we can get to the filtering starting and so what we start with here is the changes to pay for the data also there too this type of transaction that introduces the data to the network and we need some changes to distribute the data across the network but a zombie as much data just charts so

it's manageable by the old Network to downloads and so we don't need sampling yet and we can have everyone download the data got it and how yeah how does the layer 2 then use that data like from say optimism's point of view how do you actually leverage that is it just changing where you post the data that's currently posted in normal transactions as call data right so if they're more involved you need to optimize for the layer 2 really needs you can take apart all the things to the layer 2 users one of those things is publishing the data making

sure that this homeless minority that protects the layer two is able to get the data in the first place and then there's this other property that the layer 2 uses right now of getting the data long term but these are very different data availability is this property that ensures actors are able to get the data and this can be for a more limited amount of time and so you do need to ensure that even with downtime and even with censorship and whatever whatever interesting circumstances is actors on the network on Layer Two are able to get the data but then after some amount of time this should be sufficient to guarantee the

security of the player too because you want the actors to be able to reconstruct the states because only we have a few States and you have to fill history they're able to challenge the operator or challenge the sequencer of the product got it um I think we've mentioned like a few times already and I want to make sure we kind of clarify for folks is this idea of data availability and I think this is something that at least to me was not like clear until I spent way more time looking at the in the sharding is like what when we say data availability what exactly do we mean and how is that different than like the data we store on like the ethereum blockchain today um I know vitalik they want to give a

quick overview sure um yeah I mean I think it's definitely a very important and the subtle topic like I think even the the really big point of comparison a lot of people have is like what's the difference between what we're doing and ipfs right um and uh this like ipfs is a platform where if you publish data um then that like presumably um you know if the incentives are right or if enough people care about the data the data gets broadcasted and then anyone who wants to download the data is able to download the data the difference between that and what ethereum is uh doing and going to be doing is that uh ethereum is and will be providing consensus on uh data

availability basically so you just you can always have a hard consensus on the question of whether or not a piece of data with a particular hash or a particular commitment actually is available and what we mean by available basically is did what did the data go through this publication process where it got broadcast on a public network and anyone who wanted to actually download the data actually did have a lot of time during which they could have downloaded the data uh so basically the the difference between that and something like ipfs has to do with the case of a malicious publisher right

um like if I'm I'm a malicious publisher then like on ipfs I could potentially do something where um you know I control some small number of servers and then I published a small number of servers and then a small number of servers might respond to and say data is available to some people but they might not respond to other people and so some people get the data other people don't get the data but you never actually like get this kind of very binary consensus on whether or not the data was actually published now the reason why this kind of concept of consensus on data availability is necessary has to do with a lot of these Layer Two protocols where those Layer Two protocols depend on data

being out there and use like not downloaded by Everyone by default but download a ball by anyone in the case that they want to download it for some of my security properties right so one very simple example is a ZK roll-up right an Ezekiel roll up you have a sequencer that sequencer accepts transactions that sequencer publishes these blocks that contain State Deltas they could get a proof and that's sequencer also contained like basically manages this kind of internal State like what is the state of like you know the balances contracts whatever inside of that ZK roll up but now the difference between Ezekiel roll-up and a validium right is that Ezekiel rollup has like

State Deltas or inputs on chain in a validium you only have the proofs on chain and you have everything else on chain from a pointer security points of view of like can they force invalid off to go into the system both Roll-Ups and validiums both protect against that right because of the ziki Stark prevents you from actually bringing anything invalid the place where they're different is what happens if the sequencer disappears right what happens if the sequencer becomes malicious and they just basically shut off from the network never talk to anyone again and the reason why they do this is basically that because they want to meet just make the make it not possible for someone else to like interact with that system

going forward and so if people have money inside the system that money gets stuck now in a validium this is actually a problem if the avolinium operator does this then they can't steal but they can't make people's money stuck um and so you know if they're really mean they could potentially like extort and they can basically say like hey you know if the whales don't like stanza 20 of their money to a ransom address then they're just going to make everyone's money stock forever in Ezekiel roll up on the other hand there's this guarantee that because the either the inputs or the state Deltas get published to the chain if the original sequencer disappears you can always have a new sequencer come in read the data from the chain and basically initialize the exact same

state that the original sequencer has and so be in the exact same position and have the exact same capability to then be able to continue providing ezekia roll-up blocks processing withdrawals and processing transactions right so because that data is on chain and so someone else can come in and reconstruct it and like basically swap themselves into the same role uh you don't have this same security problem that validiums have right so the difference between the two basically is data like in this like on chain or is it off chain now why does it matter if it's on chain because on chain is a very simple convenient medium where if you see the data is on chain even if you don't personally process it even if you

personally don't care about it you still know that if something terrible happens and you need it to recover then you will be able to like actually go Unchained and grab that data right um so what proto-dink shorting and like and that eventually folding schroding try to do is they basically try to like really zero in on providing a platform that provides exactly that capability right so the beacon chain would actually only contain hashes of data and so if you're a client then you would base you'd be just downloading the beacon chain and you would get hashes of everything but what I say has just here I mean like hashes of TCG commitments and you know blah blah complicated math but like think of them as hashes like actually yeah it is a kcg is a cryptographically valid hash function by

yeah you know definitions of collision resistance and pre-image resistance and so forth but uh basically yeah the actual full data would instead live in this like shorted system where it would be inside of shards and it will be inside of your sub networks um and the point of all of this Machinery around data availability sampling and that sort of stuff is to basically provide a way of uh kind of checking and guaranteeing that the data actually has been published through this mechanism where if in the future you need it you will be able to get it without actually requiring everyone to like direct way down model all of the data themselves right now the chain does not have to store the data or the shards

do not have to store that data forever right so like the plan is for them to delete that data after some period of time like it's uh you know numbers have been thrown around of like somewhere like could be 30 days could be a couple of months and then uh basically yeah but but the point is to give enough time that any mechanism or like anyone that wants to be able to download the data will be able to download the data and like for there to be enough time that for all of the people that would be making backups for related to your particular application to actually have the time to do that right so basically you can you know create the system that's like really optimized around this idea of like how can we just provide this exact guarantee of like data availability like proof that the

underlying data behind a particular hash has actually been published through this public notice board where if people wanted they um they can get it um so the roll up similar to is it can take advantage of that for scalability without incurring the complexity costs of like actually trying to Shard um like you know like full-on evm execution or whatever right right and so it's like the the guarantee of the ethereum L1 protocol is is quite tight it's like we guarantee that this data will have been published on the network for this amount of time beyond that obviously there's still ways to retrieve that data they're just not guaranteed by the ethereum protocol right and they it could very well be on

ipfs but that's not a guarantee that the protocol can make you know right exactly yeah um and and we've we've touched kind of on this a couple times already like this idea of data availability sampling and only verifying like having each Fair each uh validator verified parts of the data to make sure that like overall the entirety has been published bankrupt do you want to give us like a an overview of how this data availability availability sampling work um for calling it an intermediate audience you know not a photographer yeah yeah yeah so um so the idea between behind the data availability sampling is that

um you somehow uh want a scalable way um to to ensure that uh some amount of data is available like is basically and available means you could download it if you wanted to right and so the obvious obviously we know if we don't know it which is what we do now it's available that's simple right because if you could download it then you could download it okay but um uh but how do we make it scalable so scalable means we have the same amount a constant amount or maybe increasing a logarithmic amount but not like a linear amount of resources that we need to do this amount of work

um so we we need to find some way of doing this um in a more efficient way and the way data availability something does this is um what what nodes do is that they a sample they they pick these random parts of the data and they say I want this and this and this and and I'll try to get it and only if I can get all of these or maybe a vast majority of them then I will consider the data to be available and naively like if you just do this on blocks as they are now then it doesn't work and why because um if there's say just one piece of the block missing the probability that you

catch it is Tiny because you would have to request exactly that piece of the block and you want to only request a really small part of it so the probability that you catch it is more so that doesn't work because we know like in blockchains basically the thing with blockchains is like the bad things could happen anywhere like even one single missing transaction could screw up the whole system so you cannot allow any part of the data to be missing like just sampling directly doesn't work so what you have to do instead is you have to first encode the data and you encode it in such a way and that is this is called read Solomon codes you encode it in such a way that um

uh that any any fixed fraction for example you can pick 50 percent uh if any 50 of the data are available then uh then you can reconstruct the whole from that so you encode it in that way and now um now the scaling becomes different because now you don't have to ensure that all the data is available you don't have to know that all the samples are available but you have to know that 50 are available and that's a task that you can do statistically because um if you download uh 30 samples or well I mean the correct way is like saying like this if someone is trying to hide

this is the attack we're trying to defend against right someone is trying to hide the data somehow if they do that they have to make less than 50 percent of the samples available if they make less than fifty percent of the samples available and you download for example 30 then the probability that uh that you all of those will be available is 2 to the minus 30 which is one important in a billion and so it's really small and by downloading 10 more you decrease it by another factor of 10. so that that is a scalable way of ensuring data availability and that's the principle of hard works great great um and basically building this entire this

entire system uh is why shipping Bank charting is is going to take a while um Frodo can you walk us through like in the meantime in like the 4844 world how do we like sidestep that why why can we get away with not uh not having all this already live so for backgrounds here first of the merge the separate ethereum into a consonants layer an execution layer and we are not throwing more data at the execution layer but rather we continue to scale the consensus layer and even then they are only doing so with a limited amount of data so we're talking

about a month for maybe three months some amount of data that is retained after the periods we start to prune the data so we can ensure that it's available for layer twos versus for a sufficient amount of time for them to secure their Network but then at the same time it doesn't grow infinitely like it doesn't grow indefinitely where now we have a bounded amount of data to store and consensus nodes they can distribute this between the different speaker notes right and and I guess just to clarify this for the for the listeners um the amount of data that we make

available in front of dank Chardonnay is less than the amount that we make available in full length sharding correct um red so the Fielding starting the distributes the job of storing and propagating the data between all the nodes on the network between the different fertilizers whereas with Erp for it for fur they still require all of the consensus nodes to acquire of The Blob data but we limit this we don't make it curl indefinitely so we can increase the throughput got it and can one of you give me an estimate like you know how much do we lower the cost of storing beta uh for

day or twos with 4844 and then how much do we lower it further with a full sharding deployment roughly sure so current ethereum blocks are anywhere between like 50 maybe tops like 100 gigabytes it's very variable the like worse guys it could grow a lot larger but you are paying for call data data that is going through the evm and it's available forever this is very different type of data that the rollup really needs so instead you can try and optimize you can have this different type of data called blob compared to call data and we can grow it from this order of magnitude from like 50 gigabytes to maybe like a megabytes per block

and this is obviously this is already a huge increase that Roll-Ups could benefit from A reduced cost 12 and then the Fielding sharding it can go another order of magnitude larger because now we don't have to store all the data on one nodes that we can distribute it across 64 nodes so we could have a multiplier here like how we do the data apart got it so it's like we get an order of magnitude increase with just 4844 in terms of how much data we can have and then we get another order of magnitude with full big sharding um and then one thing that's that's been interesting to me to to learn as I've been spending time on this is the idea that like the demand for storing data in

these blobs or in the full sharding system is like independent or at least decorated from the demand to use ethereum gas right like there might be people who are willing to pay a lot to execute computation and people who are willing to pay a lot to store data but they don't necessarily overlap and it's like it creates two different markets um so can one of you kind of walk us through like how we're like designing these two different markets and and isolating them from each other to an extent well so we it starts with the transaction type where you add this additional fee parameter but um with this fee you create this different markets and so if you really want to leave good separate the transaction pool

and the capacity and the type of resource is very different and I think fetalek already wrote a post about a multi-dimensional EIP 1559 where we can try and think of all the different resources in ethereum as different markets and I believe bank cards already has a post while erp1559 could work for this type of blob data instead of regular gas right so it's like there's two auctions happening in parallel one is people bidding for like transaction computation and the other one is people bidding for storage and we can use kind of the same mechanism which we have already for for gas um and call data which is like weirdly

bundled uh to then uh to then separate it then have one 1559 that works for gas and 15.59 that works for for shard data um another thing uh we haven't touched on this a lot but um vitalik I think you you mentioned them earlier on that uh that the sharing design requires introduction of cases G commitments um and they're kind of like a hash but not really uh I know dank Rod you had like a great post about them do you want to explain again in sort of to non-cryptographers uh what these are and and you know how they resemble the the cryptography that's currently in ethereum and differ from it yeah um yeah I mean so I I mentioned earlier

at the end of explaining sampling um how the data has to be um encoded um in this way that we call uh read Solomon code um which is a way to ensure that um any 50 of the data can be used to reconstruct the whole data so I mean I guess data is a bit misleading here because there's like the original data which is the actual payload that we're talking about but then the code uh expands this data so it becomes twice as large in the process as well um and so um and so what is a read Solomon code so a read Solomon code is uh um well so yeah we we call it it's a

it's a polynomial so basically what what what it means is uh you you've learned about polynomials maybe in a mathematics classes it's a it's a certain type of function and uh and basically um this type of function has the property that uh that when you when you know it at a certain number of points which we call the degree of the polynomial then uh then you know the whole polynomial so basically uh we use that property um or that in order to put this polynomial function through the data and then if you if you have like a certain number of samples which is like um half the amount of the full encoding then you

can get all of them um and uh and uh the reason we need kcg commitments is this like when you just sample the data uh there's one thing that you can't decide from those samples and that is um whether the encoding is correct what if someone just like read solemn calls have a certain structure right they have a certain uh structure that allows us to reconstruct the whole thing but what if someone encoded it in a different way what if they just put garbage in it then every 50 of the samples would give you different data and of course that's not acceptable because the data has to be unique it has to be that all 50 of the

samples give you exactly the same data and um and the way we we do that I mean they're different approaches but basically over the years uh we have ended up here where we just found this amazing type of commitment uh carte or kcg commitment that you can bear basically see as a hash it's similar to a hash of data but with a property that instead of having to just data it hashes a polynomial so it's a way to Hash a polynomial function and and basically reveal any point on it and so that guarantees that this this correctness of the encoding and that's why we need

these kcg commitments got it and as I was reading about case of G Collectibles uh one of the first things you stumble on as a non-cryptographer is they require a trusted setup um and you know I I'd be curious just for the viewers to walk through like what what actually what are the trust assumptions in a trusted setup you know like what are the things that like we are trusting in that setup and um as we uh as we make one to to kind of enable this on on ethereum is there things that like end users can do uh so that they can have kind of a higher assurance that the

setup was performed correctly and like minimize the trusted substance they are making individually right yes um yeah so basically The Trusted setup is like basically what what we have to do is we have to uh generate uh this elliptic curve points that have a certain relation that's that's like one of the fundamental inputs um of the of the kcg commitment scheme and The Trusted setup is basically a way and okay and in addition to that property that they have a certain relation is nobody is allowed to know the actual relation between them so this has to be a secret and that's why it has to be the structured setup and it's called trusted setup because like one of

the ways to do a trusted setup is just to say hey we all trust him Tim you do it and you give us the outputs and then it's done and you throw it away and everything will be fine but the problem is of course that's not really sufficient for uh the ethereum community people would be like well what what if um so instead we have this way of Distributing this trust and saying like we let many many people participate in this trusted setup and and we can design trusted setups in a way so that if even a single one of these uh people that participate in it did it according to the protocol and the protocol means that

you you execute this whole thing you run those programs send your output and then you destroy your data like you destroy the secret that you've used to do it you don't you don't keep it and if even a single person out of the potentially thousands that are going to set to participate uh did this properly then the setup is completely safe so even like let's say 1 000 people do this 999 colluded and they all kept their secrets and they come together and try to reconstruct it but one person did properly and they don't have it even in this case these 999 people know absolutely nothing that helps them to break it um so that's the security guarantee

which we call like n minus one so even if n minus one call you they can't get anything on that last person that participated properly and yes I mean obviously this property has this has a nice property that if you are really really worried about this and I like oh my God how can I trust these people um then you can just participate so like one obvious way is if you participated and you know you did things properly then you don't need to trust anyone because you're part of it and um and yeah it's it's done so like you you can consider that secure obviously there are also all the parts of like making sure there's all the software works properly and so on that we're all very familiar with in our blockchain

systems like all of this we're relying for the function on ethereum as well but so obviously it has all to be very well audited and uh we need several implementations of this as well but yeah so that's that's I guess like the other trust assumption that we have to make sure that all the software is safe and works properly right so there's like kind of different levels it's like either you know you don't care at all and you just trust that somebody somewhere has been honest um and maybe you just it's not that you don't care but you learn about ethereum 10 years from now which is basically what you have to do because you can't participate if you're if you're part of the ethereum community today there's going to be an opportunity for people to participate individually so then as long as you're confident that like your

participation was was correct then then the the whole output should be correct and then if you're even kind of more concerned uh there's a specification from which we can write different implementations so I assume you you know if you didn't trust the existing ones you could write your own and also produce an output um or at the verities kind of review the code of the different ones and make sure that they they match up um and yeah you would kind of get a high level of certainty that things are correct I'm okay and I guess the the we're we're coming up on time here uh the last thing I did want to talk about is like how do we actually get this deployed

um which is the part I usually get involved in um I know Proto you started prototyping 4844 uh along with with some other folks uh can you give just kind of a quick summary of like what was done so far in terms of prototyping and and what do you think are like the next steps that the community can expect right so earlier this year it started with an initial variety of what it could be like then during the hackathon in Denver it transformed into this software where we have an actual implementation of The Proposal and over time we have been improving that and testing that and what we need to go and do from here is

there are two these two different branches right where they want to further develop the client software to be able to make a test Network and we want to continue the development on this trusted setup so that they do have the cryptography everything there on that side all set as well for when we do want to deploy this and then once we have both ready you can make larger and larger testnets and then eventually included as an Erp vid awkward apps process into uh etheria minute got it got it so we have some initial prototypes we want to product like grow them make them more robust make sure the

trusted setup is working according to plan and then once we have that it kind of becomes a normal EIP we need to Shepherd through the process [Music] um and then one thing that that's worth noting about 4844 is what's very neat about it is from the execution layer point of view so like the the kind of smart contract and end user transaction generation point of view um sharding is basically done then like uh there's no more changes that we'll do uh for people to interact with this blob data what what will happen is then we need to deploy kind of this entire Bank sharding uh infrastructure on the consensus layer um but from applications point of view that that kind of just happens in the

background um but at the consensus layer dank Rod like what are what are the steps to get this deployed like how many how many hard Forks do we need to get there uh you talked a lot about like proposal Builder separation before so like what do you see is like The Logical set of Stepping Stones to get us to the full full sharding on on a consensus layer yeah I mean I I hope it's uh it's two but uh I don't know yet I think like um so clearly like I mean this is the reason why we chose this uh stepping stone of proto-denk sharding that it's a it's something that gets uh substantially closer to the full implementation so it will become simpler it will like the interface for example

will stay the same the execution layer changes will be minimal once that's implemented um so that's why it's really nice if we can get that done um in the Shanghai hard Fork um and then um I mean I think it's very unlike that will be the next hard Fork after that um but hopefully relatively soon um we will get uh yeah um we'll get this I uh yeah I don't know I I it's currently still uh there are still like some things we definitely need to work on like there's there's a lot of work on the network working to be done to have uh yeah full um sharding rolled out um

I I currently have no exact estimate but I would hope that that can be done in one hard Fork okay I sympathize with not being able to give direct estimates about complex projects so I won't push you on there um and maybe yeah to to kind of close this off like the public like if people want to contribute from like the from a researcher like engineering point of view like what are like the big open questions in sharding land that they should spend their brain Cycles on um I think one problem is definitely like figuring out the networking of data availability sampling like there are

designs that we have um that work in theory right like there's uh doing it based on subnets uh there is the uh approach of trying to make a DHT much faster there's a couple of other techniques in the middle um but like really taking those ideas and from an engineering perspectively just trying to optimize it really hard like how do you actually make a basically it's like a specialized uh scalable DHT but we are publishing and downloading uh can have been extremely quickly um so that's one problem and I think in general like in the ethereum ecosystem the networking side is um one of the

sides that's talked about the least I mean possibly just um I think a bit of an accident if you're Serene that the ethereum core research Community just happens to like never really have people who like spent a lot of time thinking about networking stuff it's generally like they always spend a lot of time thinking about I'm in a cryptography and kids dentists economics but it's still a really important problem and it's a problem where I think it would be amazing if we can have more work very active networking expertise in ethereum so that's the short term or that's like a very clear problem another problem is that with folding sharding there's this issue of like how to combine it well with uh proposal Builder separation and

there there's some like economic challenges there's still the challenge of like well how do you actually make a good proposal Builder separation protocol how do you add like censorship resistance lists so you can bypass uh censoring Builders and once again there like we have ideas on each of those things but there's still the question of digging into the details combining a PBS design with the design of ethereum's future proof of stake which is uh something that at some points will probably start um having to kind of talk and think more about right like like if there's been this uh increasing effort within the ethereum research and the protocol community of basically thinking through

what would a better uh proof-of-stage design look like in the long term like you know do we want to have singles a lot finality how do we achieve single slide finality what other benefits can we achieve um how can we offer more in protocol of what Lido offers to people extra protocol uh to try to reduce taking full centralization incentives um and just generally increasing Simplicity right so I've written a bunch on that like if you just you know Google for a single so what finality you can probably find it um but the do it that in the user section of that and purpose Builder separation and the intersection of that and uh shorting is um you know going to be another research area going

forward um and then the other one is uh also adjacent to dig shortening stuff but and also adjacent store 4444 stuff which is uh very critical to um prototype shooting actually being viable and just generally ethereum scaling well is like creating the like as decentralized as possible and as robust as possible uh systems to give people the same guarantees that the Asia come to expect out of uh ethereum in terms of History retention but without requiring participants in the core ethereum consensus protocol to like all be retaining blocks forever right so uh

there's a team working on Portal there's things like the graph like there's this big long list of projects right um and uh trying to figure out you know how to make those better or even create better alternatives to those I think also an important area um yeah also um one last one I really wanted to throw in this is important um the switch to Layer Two um is um I think one where it's very important for the ecosystem to try to maintain and even improve it's a decentralization going through that switch and so you know we need white clients that can poke into optimism and poke into arbitrum and poke into a stark net um and uh that's something that there's been a bit of like theoretical thinking work on

um but it's the sort of thing that I think there is a lot of room for people to select themselves in and uh like really try to improve that ecosystem a lot um right like basically try to like really think through if everyone's really going to migrate over to L2 over the next that's well to 24 months um especially as prototank shorting uh comes alive and the roll-up costs or go down even more like how do we really make sure that transition goes well and that it preserves all of the decentralization properties it even improves on those properties um that we've come to expect of if ethereum yeah that's a really good list um we have like five minutes left so I wanted to leave a bit of time for the three of you if there's anything you

wanted to share that you think is important about Chardonnay in your 4844 or ethereum generally that like we haven't talked about um before is yours whatever you want to rant about or get people to pay attention to um I mean the last week has been kind of chaotic to say the least uh I think this is like the type of Market where if you feel a little bit dumb like maybe read a post try and get involved with me some projects let's start the not the bear Market it's the Builder markets read the the specs for the EIP there's the site called Erp 4844.com so I hope that you

started and then there are sample diagrams and all the way down to like very elaborate posts about the crypto cryptography involved and uh yeah just reach out and get building stay optimistic by the way out of it they grad vitalik anything else you want to share no stay optimistic but in the long term hopefully I'll stay zero knowledge wow [Music] anything out here on that crowd uh yeah I mean like I mean I think I think the week has shown that um

uh that that you need solid designs and that we need to like build things that can actually last and that are yeah built to last and um uh won't go away and that that's what we're trying to do here and um yeah so I I'm optimistic on this but yeah I am clearly also pessimistic about many other things that are happening in the ecosystem so we just have to be better and build better love it um yeah then that's basically a wrap the Bagless guys did ask me to end with a disclaimer so uh here it goes uh and I'm reading from the screen now risk and

disclaimers crypto is risky you could lose what you put in but we're headed west this is the frontier it's not for everyone but we're glad you're with us on the bankless journey thanks a lot um and yeah thanks a tan Prado vitalik bankrad for coming at a bunch of different hours across your respective time zones um I think this has been really helpful to explain the entire sharp robot to people difference hey we hope you enjoyed the video if you did head over to bankless HQ right now to develop your crypto investing skills and learn how to free yourself from Banks and gain your financial Independence we recommend joining our daily newsletter podcast and Community

as a bankless premium subscriber to get the most out of your bank list experience you'll get access to our market analysis our Alpha leaks and exclusive content and even the bank list token for airdrops Raffles and unlocks if you're interested in crypto the bankless community is where you want to be click the link in the description to become a bankless premium subscriber today also don't forget to subscribe to the channel for in-depth interviews with industry leaders ask me any things and weekly Roll-Ups where we summarize the week in crypto and other fantastic content thanks everyone for watching and being on the journey as we build out the bankless nation