zkSummit - Sharding Progress Report with Fredrik Harrysson

Transcript

sort of have Forks alright yeah Rob is sick unfortunately so he couldn't make it I put some slides together a bit last minute here and we'll talk about how someone else will sauger or scaling problem before you know I could not agree more with Adrian we it's everyone's problem you shouldn't just be sitting back and waiting for someone else to do something like we have a lot of companies who now have a lot of money and a lot of stuff at stake and basically can't launch their apps because aetherium is not good enough and I cannot agree with that or that like it's it's everyone sort of

responsibility to build toward this common good of trying to make this thing work this talk is sort of progress updates so I just got back from Taiwan yesterday both mean rob of it was a workshop between clients implementers and etherium research and sort of all the people involved in trying to define what a scaling solution is for etherium and so before we dig in I kind of recap a little bit of what Adrian was saying as well but there's a lot of discussion around what scaling actually is and what it should be and the first one that's been talked a lot about over the past

couple of years is raiding and lightning and what's important to know here is sort of that it's almost like a point-to-point system if I like Alice wants to send a transaction it's about they open us a channel with each other they can transact however much they want off chain and then then like settle their final balances to the chain there's various ways to make this point-to-point system a little bit more distributed but it is in essence that so you could send a transaction through a third party but it's still sort of point two points between all these different players plasma plasma cash is sort of aiming to be a general off chain

aggregation method and you take out latest podcast where I talked to Carl floss and Phil Dyan on this topic and we dig into everything with the security model and what floss it has your really like you don't have data availability like Adrian was saying if everyone wants to exit it's a problem it has its applications but it's not a general scaling solution either what a theorem and the same foundation a theorem researchers and clients implementers are trying to build is this sort of real layer one scaling solution and I'll talk a little bit here about what that entails so we have a theorem the naive

way to scale it is to make it bigger right this means larger block sizes it means larger state larger history it's essentially something that we can't keep doing we can't keep up with it it's not realistic so the naive approach well bit less may approach but what we've seen in the past is sort of view for caching you create a new chain you do bunch of transactions on there and so essentially you start up all these other chains and then try to somehow swap tokens in between them and with bridges or atomic swaps or whatever and the problem here is that you're splitting up

security like if you started off by the theorems you would have five times the scale but how do you do this without splitting up security so I'm trying to illustrate that here by basically this chain has six miners and so this is secured by those miners but really we've split our entire security pool across these five different chains so we now have one fifth the security on each chain and this is obviously not acceptable so the question is can we build a system in which we have five chains but they're all secured to by the same pool of security and the answer is yes we can't have that that's what's

being planned and what's being built in this system you go from having my nerves to having it proposers co-leaders and executors what's important to note here is that this all assumes a proof of stake system this cannot work on a proof of work system so you have to have proof of stake you have to have the ability to slash someone's tokens for basically crypto economic security so I'll dig into a little bit what phase one is but it's if you actually want to read all the details there's a great post on the research forum I made a little tiny link to this as well so you

can just go to tiny dot CC slice sharding and it'll get you to this forum post it digs into kind of everything and how phase one is supposed to play out but essentially phase one is intended to like literally be useless but it's a proof of concept of the proposer and co-leader of structure so there will be a hundred charts in the first version of this which means that we have a hundred times the scalability this transaction throughputs and there will be no cross shard communication so each of these chains can propagate data up to the main chain

but they can't talk to each other and that's a that comes in a later phase so we have this proposer kool aid or executor structure in phase one there will be no executor so the proposers proposed completely random data like it's just some random data to be able to agree on the co-leaders take this random data by the proposers and they commit that to an on chain contracts so there's this sharding shark manager contracts that the etherium foundation is writing that would be deploy to the main chain and then we have these two different players in this game where

so it's the proposers and the co-leaders and the co-leaders collectively secure all of the chains so it's the same collator pool and these co-leaders stakes on the Mount of ether to become a co-leader it's free to be a proposer so anyone can be a proposer essentially what the proposer does is assess here's a block of transactions that I proposed to you Co leaders to include on the main chain and the co leaders are picked free to pick whatever proposal they want and they sign off on it and sort of committed the Merkel tree roots of that collation to the main chain and then

sort of that's how this shark progresses and every collation that is committed we get one step further along the shark chain the initial proposal I think is to have every have a collation every 5 or 7 blocks or something on the main chain so we're not like overloading the main chain with these collation transactions the main chain is completely unchanged from a protocol perspective in this scenario it's only this on chain smart contract that manages the coalition's and there's no cross chain communication no executors so it's a very simple

system this is something that we talked about like roadmap and everyone is kind of starting to work on this now and should be done you know in three to six months something like that it's it's a pretty easy thing compared to the full system which will take several years to build phase two I'm sort of copy pasted this from the forum post it's full nodes only so there's a separate conversation that I'll go into a little bit about like what different types of nodes we can support in the system so the overall goal obviously and this is to increase throughput with

increasing the amount of resources necessary to secure this or to run and no it on this so you should really only ever be responsible for running only the main chain or only one shard there are there is a concept like a super full node that runs all the charts and the main chain but those would you know require and like N squared amount of resources compared to how many charts you have and it's not reasonable that normal people should do that so the goal here is still that you can run a client for the shard that you're on on your phone like that's that's the ideal

that's what the ideal have been for ethereum always and will continue to me so we in phase two there will be cross shard communication this has cross contract here but but it will be asynchronous so essentially there's some idea or some plans to make the evm asynchronous as well so you have like a callback model which means that you can send a message to a chart and then wait for that chart to like a block on that chart to be mined and then the transaction goes back calls function on your original shard and that's sort of how communication will happen and this is a bit of a problem so there's this

problem called the training hotel problem which is essentially that I want to book a train and a hotel but they're from different service providers so if these two different service providers live on two different shards I need some way to like say that I booked a hotel here and only if this booking is successful do I want to book the train over here and with only a synchronous calls it's not necessarily possible I kind of abstraction I don't necessarily have to dig into all that much a notable point is that ewaz them is the target VM for shards so it seems at this point that'll be

unlikely that he was and will ever exist on the main chain and if you want to run awesome contracts you have to run it on a shard somewhere so the shards will all be homogeneous they'll all run the same VM and look the exact same way and will probably run awesome and this is sort of the reason we're not talking about this at all until Phase two is because we're not proposing any transactions or anything we're just proposing random data in phase one so in phase two is when we introduce a VM and propose actual transactions that are run on that VM accumulators this whole thing that was like a whole day of

the chef to talk about at various accumulators but say like our Merkel tree is the type of accumulator that lets you prove something about the data that you've accumulated into like a hash Storage Francis another thing that will probably be necessary I don't know if it's necessary for phase 2 but I'll dig into that a little bit as well and in a bit so the the rest all these different phases is very much unplanned like your unmapped outs all of these contain areas of very active research with lots of unanswered questions

stateless clients is essentially a type of structure where instead of every node maintaining a full copy of the state database you just maintain a copy of like the data that you own or that you're interested in and then when you make a transaction you submit that transaction along with a witness of the Merkel tree that lets you prove that yes I this is like a valid transaction and then you know miners can include these transactions into a block because they've seen the witness of it and no one has to manage State really but then you have huge questions of data

availability and and a bunch of other problems that are sort of unsolved phase fours crossed our transactions which is like to solve the the train and hotel problem you need like internally synchronous and zones within these charts so that you can actually like in a synchronous atomic fashion do transactions across charts face five is sort of working out all of these data availability proofs the Casper integration for free sharding is a very kind of controversial topic polite things that there should be four full charts and is very undefined and the

question here like there's some other like layer of abstraction that we could introduce here like having a manager chart so essentially replacing the main chain with a chart so that we can have like recursive charting and it's not locked in to be a hundred charts or anything so it basically have a manager manager chart that has a hundred charts and then that one can report up to the main chain so you can have like yeah super Quadra Rd charting which is basics and then the question is how do you do load balancing between these if I have if I deploy a contract to a shard it becomes super popular maybe I need to move that to another shard or other contracts on that chart need to move to not be bogged down by that super popular

contract and yeah like providing a developer abstraction on top of all of this is super important as well so in a load balancing model you really don't want developers to have to care where their contract is deployed like on which shark that sort of goes into the account subtraction bits as well and I will take questions before I dig into this stuff I think because they're they're interesting topics that we all talked about and kind of cover that this workshop as well but they're semi unrelated to two sharding any questions yeah so there's there was a couple

different models proposed on how a shard is picked it could be that yeah there's a fixed number so you know take your a node ID modulo 100 and that's what your shard becomes or your address module 100 something and that's one model another is you actually get to choose yourself because you may actually have knowledge about like I my contract you will need to talk to this one or like this shard is less BC than that one and so I want to be there gas prices and gas price markets will be a thing in this model because gas prices may be cheaper in one chart than another so then how do you actually work out where to deploy stuff

that's part of the load balancing questions as well initially it'll either either be that you just pick it or that the the you node ID or your addresses picks it for you I think what's important to note as well and I didn't really cover was the co-leaders there's a pool of co-leaders that secure all the charge and you have in an honest misunderstanding like 400 or 100 some number of them randomly to be co-leaders for any particular shard and you swap this often exactly how often is undefined but then you have like just statistical

significance like a statistical property that your shard is secure under like 2/3 majority if the honest majority assumption holds for all the co-leaders you could yes it sort of depends a little bit on the how do you pick a chart and what the yeah what the model is there but there's really no reason that you couldn't add charts but probably what's more likely is that we'll run with a hundred until like

super quadratic charting or like the the whole managers hard thing is introduced so that you basically add when you run out of space you add another manager chart and get another hundred that would have to be left up to like a governance structure that is undefined yeah no randomness actually would

probably work quite well to distribute the load across the charts especially I mean to be honest we don't actually have that much deployed on ethereum there's not that much running it and because there's not that much that can run so once we get like a hundred charts we have a hundred fold increase in capacity and we'll like we're pretty good for a while so really feels unlikely that we'll need any really advanced load balancing mechanism for a long time would it be possible to split one

contract across charts no that's not part of the model at all so the word is basically if you have a very popular contract that would saturate the transaction throughput of one chart you need to find a way to split that contract up yourself so you might have one contract that does user management on one chart and another that does you know Twitter postings on another chart and then you would have to rely on cross shard communication to actually like make all this work and yeah it's pretty complex question of how do you do that and how do you make the UX of that good quite a lot in a synchronous sort of

zone and constructs you don't necessarily like you'd have to block latency I guess but in the asynchronous model you could happen like your worst case is actually quite bad like you'd have to wait for a number of collisions and then get things back to the main chain and then all that has to go back out to the shard and so it's actually quite a lot of blocks that might have to go to pass in between [Applause] since this is proof of state you would

have transaction final finality like it would all be instant finality stuff basically and the reason like I don't really understand the reason Vlad doesn't like for freeness but in his so flood has his own model for charting unexpectedly sort of binary charting structure where you use merge blocks instead of reporting anything to a main chain so you have like one chain or like two chains at the top that have merge blocks that they talk between and then there's like so imagine a binary tree and at each level

you have merge blocks so you can kind of merge up to one chain in that model you you could actually run each of those chains on proof-of-work or something else and you just need to make sure that these merge blocks happen in some way and in that system you have completely different like you kind of gets cross shard communication out of the box but you have completely different latency expectations and how you traverse this tree with various transactions and yeah so larding is is a thing in its early stages sorry yeah yeah basically the assumption

of this model of charting is that it is proof sake I yeah otherwise like you're saying like there's no real guarantee to provide for freeness otherwise yeah and if so yeah it's a whole thing of light you could under certain security assumptions allow both Forks and reorg sand everything like that but we arts can actually lead to a situation where you need to have cascading reorg so if one chain reorg then you need to like reorg everything else and it becomes a nightmare and actually might break networking assumptions and everything else I'll only have five minutes left

but I want to talk about storage trends so story trends what's most a big thing also on the latest podcast this Phil diamond and he talks a little bit about strength this is sort of his thing that he's talking about now and I think it's a super important thing to highlight there is a misaligned incentive right now in etherium between miners and full nodes so when you deploy something or when you use storage on the chain you pay a fixed fee once to put that source there and the miner is the one that gets this fee but the miner is only one player in this market that actually pays the cost of this storage every other

full node and the network also pays the cost of this storage and they get nothing for it so there's a very misaligned incentives the miners don't even necessarily have to care about storage they only need the latest box so they could technically not pay the cost even and still get the get the reward for it furthermore it's obviously unsustainable to have the chain keep growing forever so we need to fix that somehow and storage friends the proposed solution where instead of paying a fixed fee for storage you pay a you know price per unit of time for storage so essentially anything that you deploy to the chain would have time to live a TTL and depending on what this TTL is like you may choose to use the

contract or nots and like if you see an app and it has a one-week TTL you probably wouldn't trust that app whereas if you see it has a hundred years TTL then you might feel free to use it the biggest question around all of this is what's a good UX and how do we actually migrate people from having had free storage essentially free storage forever to having to pay for it and there's a bunch of other questions like by when we talk about storage friends and what happens when you run out of rights is that it gets removed from States but it doesn't get removed from history so now you kind of talk and

you're in a world where you're explicitly separating state and history history being like all the block parties and all the actual blocks in the chain so in that model you could actually say paid to resurrect a contract bring it back in two states it would cost the node operators some amount of money to like go through the history to find that data and bring it back but if you start separating this thing like do then have to pay rent for history as well there's a lot of talk about like history is also unsustainable to keep around forever so when do we start talking about actually deleting history and yeah it's it's a very complex topic but it's something

that I feel everyone should start thinking about now so that we have a good answer in like five years when it becomes unsustainable with the current model any questions on on that tough for or anything else alright then thank you very much [Applause] you