L5 - WTF is Data Availability | by Robin | BlocSoc Lecture Series

Transcript

okay yeah so hello everyone um welcome to this lecture I know that uh it's quite late in the night probably for some for most of you here um but yeah glad to be here um today I'll be talking about um data availability and what it is um yep so to begin with um what is data availability right so order to begin with that question we need to understand the data availability problem and then we'll uh you know slowly go into how we how the blockchain space is tackling the da problem and how

how they're solving it um there are multiple approaches to solving this problem and uh we will Deep dive into that a bit more in in the session so um essentially understanding the da problem right um so the basic question is how can PS in the blockchain network be sure that all the data in a block is actually available so the why is that important is because how do we know that uh there are malous transactions within a block uh if we don't have the data itself to check it right uh it's pretty self-explanatory because blockchain so if you think about

it the only way a blockchain can actually work is if you have that data of transactions to check against the current state of the chain right only then can you basically reconstruct the state of the chain or you can uh identify if there's any malicious transaction in it right so it is very critical to the security of the entire system and uh you know there are different ways in which the da problem is solved and it is very core to the uh blockch chain stack like when you're talking about blockchains Da comes the da problem comes in built with it but uh you might have heard the da term being

pro uh coming up in the past couple of years maybe in the past year more frequently than before because of basically the modular movement right because people are talking about decoupling different uh layers of the blockchain stack that is um we'll get into that a bit more but essentially we've been so far been involved in monolithic blockchains which means all of the execution all of the uh settlement and the ordering and the data availability was handled by the chain itself but we've Now quickly realized that it's much easier and much more scalable if we can decouple these systems and uh kind

of uh have each layer focus on itself rather than having a coupled system that that focuses on everything at once right so we'll go into that a bit more so uh that is where the da layer comes in a DA layer is a blockchain or some system that is hyperfocused on data availability and allows rollups and app chains to submit data off chain in a cheap and secure fashion ensuring that the data is made available so um how let let's just see how data availability Works uh in the real world right so basically the main use cases of Da comes in and usually seen in Z rups and

optimistic rups or basically rollups in general um basically in Z rollups if you think about it right uh you need to access the transaction data in order to recreate Advance the state so how zbs work is you have some users coming in sending some data uh or probably the sequence sequence is the data it is being executed the proof is being generated the proof is being sent to uh ethereum for example right but the thing that you have to understand is that the proof alone does not uh mean anything you need to have that transaction data of the State dfts uh differentials to be able to make sense of that proof right um even for

optimistic rollups optimistic rollups is the most uh I think the easiest thing to understand here like you have some sequencer that is collecting all of these data does uh this the transactions actually and uh it sequences it and it batches it together and sends it uh it updates the state and sends it to ethereum and without the D transaction data itself you are not able to under identify if there is uh some malicious activity going on if there is any dishonest uh State being proposed right so um yeah this is basically what I was talking about of the ZK rollup um suppose Alice is operator of a ZK rollup and she submits a ZK proof on ethereum which is verified the proof is verified

obviously but if she does not submit the transaction data on ethereum although the proof proves that the trans the state transitions are valid um you are still not able to get clarity on the actual State itself uh because well it's just think of this right so you can create a ZK proof that basically verifies that some computation was done correctly but you don't really get information on what computation was done correctly right so um that is the nature of ZK and that is why we need the state uh you know the states to be sent the or basically transaction State diffs to be

sent to the L1 layer or a DA layer um if you say so right so if you can see in the diagram as well the users are sending transactions the trans transactions are being back batched the proof is being sent to the L1 as well as the batch is being sent right um pretty straightforward here same thing for the optimistic rup as well so um it's being batched if there is a fraud proof you need to create this fraud proof in an optimistic rollup you need the transaction data right otherwise you're not able to create the fraud proof at all so um I hope you guys have understood what the da problem is if you have any questions you can obviously uh you know type it in the chat and I can

get to it as I move along so uh this is a very quick simplified view so you have a rollup that is creating all these batches is executing these transactions and all of that and um it sends the proof of execution all of that to uh some settlement layer and a DA layer and the da layer basically ensures that the data is available for the execution uh or the settlement to take place right um so that was the problem so we kind of understood already that we need data we need the transaction data we need the state differential data for rollups in order to uh you

know uh in order in order for these WPS to work properly in order for blockchains to work properly and um basically if you hide this data uh there is no security guarantee that the chain works or the data the the executions that was done the computation that was done is correct right so how do we solve this so there are basically four approaches that we can see right now in the space one is the monolithic approach that is obviously execution by full nodes uh that is um basically all of the execution the data availability the ordering everything is done on one chain so this is what ethereum this is what Solana does this is what probably sui

does right if you want if you Interac with these chains they are basically just monolithic chains uh by itself uh if uh and they take care of all of the activities there right the full nodes get all of the uh user the user transactions it executes it and uh uh the block producer produces the block the data is being put into the block and the block is being propagated throughout the network right and and uh if the data is not available or the straight transitions are not valid the full node because it has all the data it can reject the block right so um this is very crucial because the full nodes are able to do both the

execution and the data availability in there uh they're able to reject uh straight transitions that are not valid and only accept uh you know correct blocks right so this is how a normal blockchain works that's why I said that in a monolithic architecture this is hidden away because it's just code to how a blockchain would work you need the data to be available right but now where things start to change is when uh ethereum had started to uh not being able to scale it was not being it was not scaling properly right it could not handle the uh the amount of demand it had and people started creating pull-ups rollups essentially just offchain execution you are

basically doing execution of transactions somewhere else you don't have to know where it is happening um and whatever uh at the end of the of chain execution whatever state that uh execution has achieved you want uh to prove that it is done properly and that state is valid and uh you and you're sending it back to ethereum right so that is what rollups do so many rollups today uh use this approach called as uh so they use onchain da so they uh they basically use something as call data or blobs um when they do their offchain execution uh basically they send their

data back to ethereum um through call datab so I think in March the there was a denen upgrade where they implemented EIP 4844 which introduced a concept called as blobs um but previously rabs used to send data through call data um which is very cheap to do but it's still blo the network and gas fees was pretty much very high always because of that and there was a lot of demand and the E 4844 introduced us a new type of transaction called as blobs and um basically what it did was it was introduced into the consensus client of the ethere of ethereum whereas call data

meant that call data was uh on the execution client so those are different clients ium has multiple clients there's a consens client there an execution client um basically what it achieved was it reduced the amount of fees for sending data to ethereum so rollups could now use blobs they didn't have to fill up the transaction the uh the transaction pool through call data they can now use these blobs to send a data in a much cheaper uh fee um right and this introductional blog was called call as prot charting um and what is Proto sharding it is the precursor to what ethereum calls Dan sharding which is their uh rollup Centric road map

uh uh which is one of their end goals in the rollup Centric road map which basically allows uh more space for transactions uh and also to introduce uh a much more matured blob ecosystem basically uh instead of providing more space for transactions ethereum sharding provides uh more space for Blobs of data which the them protocol itself does not attempt to interpret this is because is like like I said you just need to submit data it doesn't have the ethereum protocol itself doesn't have to interpret what that data is doing because uh at the end of the day um this data is being inferred by smart cont and stuff etherum doesn't have to do any execution or do any state changes U for

the data itself right uh and this is uh this will help support L2 rollups uh for high throughput transactions so the main thing that D sharting will eventually so it's still to it's basically it's still three or four years away from us D Proto Dan sharding was just implemented Dan sharding full D sharding is probably years away from us but the main thing that the main feature that it kind of brings is the merg free market which means that um you know instead of there being a fixed number of shards on the network that have uh that are you know creating their own blocks and all of that stuff you have one proposer that is choosing all the transactions and data that is going into that particular slot at a

time uh this is done through uh something called as proposal Builder separation uh and block Builders carry out so basically proposal Builder separation basically means that the you have two kind of participants in that Network right now you have a block Builder and a block proposer the block proposer proposes the transactions uh and the builders build the block uh and they both work hand hand hand in hand and the validators just have to verify that the blocks are valid right uh and they can do this through dataa b sampling and we'll go into that a bit more uh essentially this is just a diagram of what that means right you know basically you have the blobs blobs are being sent to the ethereum beacon chain the consensus uh part of the network uh nodes can download the entire

block data or if they don't want to they can do something called as data availability sampling to verify that the data is uh available and the uh and it has been you know properly sent to the network right so the problem with this approach is that previously before we had the Proto tank sharting before we had blobs 90% of our rollups cost was sending data to ethereum so if you use optimism arbitrum z z kvms of any sort what uh what you may have what you may have known or may not have known is 90% of the of that particular rollups operational cost was sending data this transaction data back to eum called and this is why blobs were introduced right

but the problem is that if the demands of rollups keep scaling as was uh in 2023 which was you know 16 times since the beginning of 2023 um then we will soon reach pre uh danin or pre-pro sharting levels of fees so a one-time increase or a decrease of fees is not going to be helpful so what is the obvious next approach that they going to do one approach is obviously offchain da via a data availability committee and a data availability committee is basically just a centralized subset of validators or nodes who uh basically say that hey I'm going to keep this data I will make sure that this data is available um and I'll

keep it on my centralized storage I'll keep it on my side and when you want this data if you or if you want to check if I have the data you can simply ask and I'll show you so basically this is a very centralized form of uh of chain data availability so the difference here is data is being made available on like we're using data blobs you making sure the data is being made available on chain whereas in uh via data availability committee you're keeping this data offchain uh and if if you want to use it or if you want to check this data you need to ask the person who is holding this data to give you the data to you whereas on ethereum you don't have to ask anyone there on chain you

can uh use it by yourself right um basically you're trusting a group of nodes who have some monetary or reputational value at stake so these players might have some uh tokens at stake or they might just be uh you know big organizations or big validator groups that have reputation at stake right um this is the cheapest form of Da uh uh but obviously cheaper does not mean it is safer uh there's obviously a risk of security here and so what is uh the next best approach next best approach is obviously a decentralized blockchain ga layer which uses data PG sampling it it may or may not use it so The Best of Both Worlds is

that we have some kind of an offchain da layer that can give you cryptographic and cryptoeconomic da guarantees um so da layers are usually blockchains uh that are hyper focused on enabling data availability they have these specific features and modifications made on their protocol level layers that allow it to provide da guarantees for example aail da uh uses validity proof technology using kcg commitments and light line data availability sampling to ensure that the data stays available on the network so when I talk about data availability sampling here right what does that mean what I've been using the term data availability sampling and data availability so data availability is

obviously I I've explained what data availability is so what is sampling what is the sampling that is being appended here so basically uh imagine you have a client um but a light client basically what it does is in a conventional sense it just takes um block headers it does not download the block data itself so how can you ensure that you know that block header is valid or not so basically data sampling works by probability basically you have a light line uh that says that okay I'm going to sample or it doesn't even have to be a light it can be a full node as well uh or any node for that matter basically data sampling means that you're going to

patch random small chunks of data you you're not going to download the entire block you just going to ask maybe 10 bytes of data or uh maybe a few KBS of data or a few bytes of data here at random points in the block and if the node is able to give it back to you at each and every single time your confidence increases that node has made the data available so if you look at this picture I think it explains it really well so if you have a box of uh you know multiple different uh balls right it has a a couple of different balls in there and you want to ensure that there are uh for example

only um you know valid okay you want to ensure that only valid blocks are there so each time you're picking data from this box right what happens is so the you either have it or you don't have it so the probability for you to have it is 0.5 like if I have no information the probability is 0.5 so the first time I uh pick so randomly sample some data there a 0.5 chance but then I again sample one more time this is 0.5 into 0.5% chance right and if you the more times you do it the higher probability that that uh node or that entire block data is available right if you fail even

once it means that entire block can get rejected so it's a kind of a tricky concept to kind of wrap your head around but once you get it it's uh pretty cool to see it's basically mathematical probability around which uh you can scale blockchains um and you can with high uh confidence say that you know this block data is available and this is if that's only one light client or one node the sampling imagine there's multiple nodes or multiple like line sampling at the same time you are you know going to be 99 or 100% sure that you know that data is available right uh and this scales well because as block size or data size increases you don't have to increase the number of chunks that need to be sampled hence making the

resources needed for LC is very small right um yeah for example in the case of veale a veale has a P2P network of light clients randomly sampling data and verifying that the data is uh you know available by checking against uh the kg commitments in the header and we'll go into all of that pretty soon um you don't have the the advantage of this is that you don't have to trust validator set so uh if you have a light client doing data availability sampling you don't have to trust a valid data set for the availability of data once enough V clients have sample data so in the case of aail aail maintains a distributed hash table that kind uh that kind uh so the more you sample data it is being stored into a distributed hash table

which is being maintained by a P2P network of light lines so imagine there is like a million light lines in that Network and they're all managing this distributed hash table and if you if you're sampling different small chunks of this dat or data in a block imagine if there's a millions of light lines sampling the same block there will be a point of time where almost the ENT the entire block will be sampled at one point of time and because of that the the distributor hash table is able to create a block rep replica so even if the entire Network or the RPC Network goes down for a veale you are not losing your da guarantees up to that point because you can you can always fetch

that block again the block replica is uh able to be fetched by the uh P2P network of the light cines right so this is something go that ail provides um so what makes a DA blockchain different from a normal monolithic blockchain right um essentially um a DA layer is focused on da and does not have a custom execution layer so if you talk about ethereum or polygon or uh you have any other rollup or any solanas elements like Solana or sui what they have is they have an execution layer they have an evm they have a Solana VM they have a move VM okay they have a zkm

or whatever they have some kind of execution engine that allows you to code smart contracts and deploy the smart contracts and uh you know have some kind of function happen because of the co functions that are because of the code that has been and being ex executed by these virtual machines on this chain but on a DA layer a DA layer doesn't necessarily need to have an execution layer it can so this is the reason because of this is essentially you can have any execution engine be built on top of a BDA or any other D for that matter and basically whatever execution is being done you can

send the data to aail or to that deer and you know ensure that you know the data is available so it is not have to be opinionated on any kind of VM it chooses it can so a ZK VM can uh use a deer um wasm uh uh can use a d and evm can use a DV a d layer right it so because it does not infer the data that is being sent to it it can remain a d layer can remain neutral it can be credibly neutral and unopinionated right um so how does uh an offchain da

layer uh how how is the da problem being solved by an offchain da right because data unavailability is not a uniquely attributable fault this means that participants cannot prove to other players that a particular chunk of data is missing okay um let's so basically what that means is if for example right let's just consider there is this uh this D this is basically the da problem all over again right so you have Bob who has some kind of uh the block BL block data and Alice is requesting this block data from Bob so Bob might have done some something malicious on that he

might have added some data that is not supposed to be there or he he or she might be hi wants to hide some data away from Alice so what Bob does is blob Bob uh hides that date particular am data and sends the remaining block to Alice so since Bob is sending this data with which has some missing data uh on it um sorry since Bob is sending this block with some missing data Alice uh can you know reach out to Charlie saying that hey I think there's some data missing I want to create a uh you know I want to create a complaint or something like that uh but Bob can essentially uh send this missing data to

Charlie right for example it can make so for example let's just see say that um Alice requests a a block from Bob but the data is missing when Alice receives that block Alice complains to Charlie saying that hey uh I think you know there is something going on here and if Charlie requests the same block from Bob since Bob does not want to get in trouble Bob can add this missing data and just send the complete data to Charlie making Alice seem like the bad bad actor here instead of Bob even though Bob is a bad actor right and this is a problem uh because we do not want the you know the the

bad actor to win we want you know the not bad actor to win basically right so how do we solve this uh with an offchain dealer basically let's think about it uh from scratch right so full nodes syn up on the entire block right they spend resources to check transaction validity and stores the Block in uh light notes for example uh they just receive the headers without cheing on the block data there's no way to verify the data is available uh the only way they can uh know how to reject a block header is that if the full node send a fraud proof saying that hey this has some something going on uh this is this is the malicious Block Please reject this header right but light know face a problem if the

block producer does not reveal the entire data behind the block the full nodes will not be able to create um a fraud proof because it does not have all of the data required to create a fraud proof so L lines have uh so the light client thinks it has the correct header and it will accept that malicious header right and basically it will accept that malicious block so to counter this we need a mechanism for like clients to verify data availability so an example would be suppose Alice constructs a Block B with transactions one to n and let us suppose transaction one is malicious transaction and it is broadcasted right uh the any full node can identify that it is a malicious

transaction and reject it but if Al wants to hide transaction one and reveals the header and all the transaction data except transaction one the full node cannot uh basically create that fraud proof telling the light light node that hey this is wrong to not accept this header so hence um you know we have no way of basically conveying to the light node saying that hey this is uh basically mous so um one way to solve this you would think is lifelines because we just discussed about data sampling a light C can just simply sample random chunks of data to ensure availability right simple that's a simple solution the problem is if there

are end transactions uh in a set there is a one byn probability of that un unavailable data to be found because if there's only so for example one light client if you running a light client on your device for example um there is a one in end chance for you to find that to find out that this data is not available or this block is not valid right uh so you might only sample five positions five chunks of data for example so what if I sample this small chunk this small chunk this small chunk this small chunk and the small chunk I'll think that the entire block is valid um but this data is unavailable

here right so this red part is for examp manag is unavailable so how we can solve for this is basically redundancy this is done through eraser coding which is a mathematical way of basically kind of ensuring redundancy in data um basically if you have some data of end size you can create two chunks of data uh in such a way that you just need n chunks of the 2 N to reconstruct the original piece of data you can you just need access basically to any so if you are basically eras your coding here right in this box uh in this blocks here basically imagine this data is uh

eraser coded this uh this previous block is eraser coded uh to 2 N so if there are n transactions here you have now two n chunks here and what you uh you just need n any n basically you can have any end of the total two end chunks or transactions to reconstr conru the entire block and that is the magic of eraser coding and the samech basically and it's not like a new technology it's been used for so long uh like for years uh I mean if you've used CDs for example CDs use eraser coding the on their disc that is why like you might have seen like if there is a small scratch on your disc on your CD or dvd it still works

when you put it in the DVD player because there is a redundancy of data even if you lose some of the data through scratches on that CD it is you can still get almost all of that data uh uh without losing the original data right um so this is uh what eraser coding is and the the beauty of it is that because you you just need N Out of 2 N uh data points to get back the entire block if someone wants to hide the data they have to hide N Plus one data chunks they have to hide N plus one so they need to hide a block a complete block

and one more transaction in order to completely ensure that some some data is uh you know not available right and this is this is hard to do this is hard to do because now if you're randomly sampling data there is a high probability that you'll understand that some data is not available right um so basically imagine you have you know some transactions which are basically imagine you have a metrix of data or metric of transactions U and you are basically eraser or ex eraser coding it or extending it uh like this um so in the case of aail or any da layer they are creating they create some kind of a commitment and add it to the block

header uh in order to um ensure data availability um basically you eras your code data to ensure redundancy of data and this helps in making it difficult difficult for the block producer to hide data or any node to hide data from you uh because they will need to hide N plus one data uh chunks in order to uh you know ensure that the the block data is not completely reconstructable right um and this ensures that light client has very high confidence in the underlying data is available um and this diagram basically

shows that entire process um and we'll go we'll come back to this real soon real soon um essentially does this solve the da problem does doing data availability sampling and eraser coding solve the da problem for an off screen D of course um because if you are obviously having a monolithic chain none of most of this doesn't matter because all of it works already um but for an offchain dealer does this make sense already have we solved it no we have not eraser coding can be done incorrectly intentionally or not is another question but eraser coding can

be done incorrectly and if eral coding has been done incorrectly then there is a problem again right so although the simple trick uh makes the job of hiding data more difficult it is still possible that block producer just intentionally performs AAL coding in a wrong manner hence making sure that you know whatever you've uh guarantees that you think you have is wrong so uh a full node can obviously again verify this eras coding was incorrectly or Not by re-executing this from scratch uh re-executing the encoding again and ensuring that you know all of it is done properly and if it is not done properly you can create another fraud proof just like in the case of malicious transactions before

right so interestingly um in the case of Da though you just need one honest full node to be certain that if a block is malicious then it will receive a fraud proof because if there is one full node that is honest uh and it does the encoding correctly and it finds that the encoding is wrongly distributed it can create a fraud proof and uh send it to uh all of the other uh all of the other full not as well as a like clients right so there is a fraud proof uh guarantee there as long as there is one single honest full right um D like Celestia use this fraud proof approach

but like we have always mentioned already fraud proofs can be inefficient it does not give you a succinct or proper guarantee that you know it is uh that you know that data is available or if the encoding has been done properly and all of all this right because what if there is no one single honest validator or one single honest no or whatever right um it has always a point of doubt that comes in there and this is why validity proofs are much better and uh even in the fraud Pro and in the fraud proof case light lines always need to trust the full node uh to

be honest right right and honest nodes need to be incentivized to keep being honest right and this is the problem of being a fraud proof Centric approach and a veil solves this using validity proof technology so a validity proof based solution enables applications to use the system without having to wait for broad proofs right uh aail uses uh aail is the only da layer that uses kg commitments uh and avda is able to to commit to a polinomial using a single ELC curve group element it is the same commitment scheme that ethereum plans to use for the darding road map a block so basically what it does is uh I and now I'll go to the previous diagram

that I was about to show that a block can be encoded into a encoded into a extended metrix of n to M rows rows and columns obviously and each row of this extended metrix generates a polom commitment so basically in each row you're creating a pol normal commitment and send these commitments as part of a block letter um and I'll send you I I'll just show you the schematic representation of this and light clients can now query any cell of this metric to open it against the kg commitment and check that okay has uh because of the kg commitments now you're sure that you know the uh the eras coding has been done properly and when you're doing the light line data sampling uh you're able

to fetch the data check it against the kg cment B to ensure that everything has been done properly in a validity proven method right and if I go to this previous diagram here you can see that happen here so basically you create an a metrix of the data or whatever chunks of data in cells basically uh and then you extend the data you basically eras code the data and create the create this extend metrix of n into M size and in a rowwise manner you create a commitment saying uh commitment for this row uh so for for this row of Sals you create a commitment and for this row of Sals you create a commitment and so on

so forth for the entire metrics you create a commitment and make it part of the block header so because of this now you're able to verify when you're doing data will sampling that hey because I have data from for example D2 2 and two I can check against this commitment and open this commitment and check that this uh this data is valid and all of this is correctly done right and that is a magic of valy proven technology uh and kg kg commitments that Avail users right um um yep so yeah this is how that entire life cycle would look like right so you have the original data you have the raal coding and extended data you

send it into you put it into the header so and now the entire block would look like uh the header with the data and the extended data all together and uh you propagate the BL to validators who verify the block and then uh you have the blockchain here right and we reach consensus on the Block order and all that U and then what happens is now that you have the blocks the lightline network can do its job of doing late availity sampling and uh ensuring that the data is made available without trusting the full nodes you don't have to trust a fraud proof coming from a full node because now you can the light client that mean which can run on your phone or your Smartwatch or your browser even can directly look at the blocks

verify that the data is available without relying on a full Lo right um and this is where you know magic happens um so we are able to minimize trust and increase decentralization uh in a way that no other approach could uh do right and that is what the data will be solv say we want to be able to uh have cheap and secure da without uh you know compromising decentralization and security that a monolithic chain would provide right or submitting data

directly on chain to ethereum could provide but even though that is secure it is costly and costs also come into the factor for all these rollups and how can you have cheap da and secure da uh that is what these da allow you to do um this is again another diagram that kind of shows which the data extended data the commment pin being sent to the header uh and and this da layer is uh basically creating blocks uh which is being used by applications and the da layer uh is basically underneath all of the execution part of it um this diagram again just shows through all of the different kind of constructions you can

have um mon chain ethereum rollup without any other D ler like call data or blobs or if you use a d like aail uh for data availability you can create a validum or Sovereign rollup and stuff like that and the last part of this lecture I think I'll quickly close off um so we have the da uh you know guarantees on this offchain da but what if you want to attest to this da guarantees on ethereum on on chain right so to ensure that the data was available uh made available L2 State validity bridges are created on the settlement layer itself and it provides a proof saying that okay this data was sent and

this is the proof of the inclusion and the uh and you know this ensures that uh prove that the data was made available at somewhere of ch right um so should L2 publish data directly to uh to ethereum in forms of blobs or call data this is inherently satisfied because the data is being posted on chain but if it is using an external da provider like a w da or any other offchain da provider then the data commitment needs to be posted to ethereum uh in this case ethereum must be convinced that the full data was indeed made available and uh this this so-called D brid is attesting to that fact right um uh basically if you look at this

diagram which I took from l2b recently released da risk framework um basically you have a bunch of users sending transactions to a sequencer and uh this the layer two has two approaches here one they can send data directly on chain uh which is so for example ethereum for example um it can send directly the data to ethereum who is also acting as the data availity provider here and E is also acting as a settlement layer uh so this data on chain can be sent as call data on the execution layer or as blobs on the consensus layer alternatively uh the sequencer can send this data off chain to a dedicated blockchain like a ve or an offchain committee like a data committee that we discussed right and

this and the attestation of this data being sent can be sent uh created and sent to the D bridge and it is sold on the settled layer like a the right so this is how uh for example um an optimistic rollup would look like with a deer the transactions are being sent to a sequencer uh the transaction data is the sequencer does the ordering um the transaction data is being sent to a veil or any D or for that matter uh the block reference or is being sent to the sequencer sequencer sends the block reference to ethereum uh if it wants and then the attestation is sent um by the attestation bridge to ethereum so here because you just have

the block reference and a route and a data route the cost of sending this route or this basically proof or whatever you want to call it is very small because it's a very succinct amount of data it's not a huge amount of data so you don't have to so if you're sending this transaction data that might be MBS of data uh to ethereum it is going to be very costly but here you're just sending some KBS or even much lesser amount of data to ethereum so it it reduces your cost uh all the while ensuring that data is available and in for a ZK rollup again how this would look like is the settlement is done eum the proof is probably being sent to ethereum here but the transaction data the state diffs are being sent to a DA

layer and the attestation of that inclusion is being sent to ethereum um I have added some reading materials here that you might want to check out um as well and yeah if there are any questions I'm happy to answer them I hope that uh this was I know it was a lot of information at at once but like yeah if any questions are there just ask me and I'll help you out as much as I can um yeah any questions yeah so a question I have is that these validators are the part of the da layer that are actually uh holding this data do they hold this data indefinitely Or do they hold this for a particular time

period uh depends on how you want it um so you don't have to the validators so for example aail has validator in aail for example the validators have the option to prune data after uh I think 18 days or something because the reason for having this random number 18 days is like because most optimistic rollups uh they their their challenge period is like seven days maximum um and you know you don't really and for ZK rollups for example after 18 days like you know you can basically create a proof saying that okay all of this this all

made available and it was been proven and you can just create a suing proof again aggregate that proof or something like that and you can submit it again or whatever you want to do it it's is basically based on how that olup wants to um have the security guarantees and all that but a deer can choose to keep that data or prune it away uh most layers will choose to prune it away because after a point of time maybe like data from six months ago or like one year ago is not going to be relevant anymore it will probably be pruned away and won't be kept uh on chain because the the the entire point is not to keep it as a data storage mechanism it's a data availability mechanism so you just need it for a short span of time and once a particular amount of time is

passed you don't need that data anymore uh if that makes sense okay uh so like if you opt to prune the data after like a sufficient period of time what if I want to access that data uh so like how would you ensure that if you if the nodes are pruning the data then it the data is not lost um pruning means you lose the data right so uh if you want to you know at some point of time you want to keep that data you so every network has this called thing called as archive nodes that kind of keep the entire set of the blockchain right um there are archival node Solutions if you are a rollup for

example suppose you are an polygon for example right if you're polygon and you have a rollup or a validum running with data being sent to a deer and you as a customer of that daer or you as a roll up on that daer don't want to lose the data you can choose to run that arive node or you can choose to uh hold on to that data for a longer period of time by storing their data somewhere else right um but as a deer a deer's job is to just keep that data for a small period of time to ensure that the whatever dat has been submitted you you have enough time to check against whatever execution has been done and so like maybe one year like I mean obviously 6 months later one

year later whatever transaction happened one year ago is not immediately it's not going to reorg anymore right that data whatever transactions happened one year ago is not going to reorg today right uh so if if it does reorg it is going to be a big problem right uh that is another discussion to be had so um that is the uh basically that's the gist of it basically the project can if anyone can store archival node if they want to but the like over time that cost of storage is going to be you know GBS and TBS of data will eventually over years it will accumulate so you'll want to eventually prove some data away which is not needed yeah okay that makes sense that

you might not want to keep data for a long time because you have the chances of reor decreases exponentially but uh like you might want the historical data for some data analysis or stuff like that and as you said that you anyone who wants to access data can query these archive nodes but it is obvious that these archive nodes they have huge storage requirements so there would be like far fewer archive nodes than there are these validators and full nodes and light lines so uh do you think that this can create a sort of decentralizing problem that there are only a few archive nodes have the entire chain state from Genesis and that could create maybe uh issues that you have to depend on such a small set of uh data

storage nodes uh I mean that's a good question but the thing is again what is the centralization risk here in the sense for data analysis I mean what is the centralization risk in a data analysis perspective right uh the the guarantee you need is that when there execution happened um you know nothing malicious happened like data analysis may like what is exactly the guarantee that you need like centralization decentralization like doesn't really matter at that point uh to at least to me uh but that is a valid question and it can be solved by solved by

decentralized storage networks like you can use ipfs uh to probably send data for long-term uh storage like file coin you can use systems like these to kind of work with the D layers to kind of create like a long-term block archival system and a whale is thinking of about Solutions like that if I I already know about a veale I don't know about other ders but aail is working on solutions for long-term a decentralized long-term blockchain archival um in a way that you know you don't have to trust the centralized parties obviously so I mean that if that helps yeah that's really uh exciting and

interesting to know uh another question I had was uh that uh these da layers they're basically making it very cheap to make data available right uh and they're able to do so because uh you're doing erer encoding and you're basically sharding the sharding all the data so like each node in your validator set does not have to hold the entire data which currently happens on eum like like which happens on ethereum before the blobs upgrade and now in the blobs upgrade there's a different mechanism so like my question is uh is that after Danang sharding happens on ethereum where sharding is an inherent feature on ethereum itself and there's also another

EIP I read about like I think it's a triple 44 where there's a proposal to also prune data on ethereum as well so like if these changes are made to ethereum then what would be like the difference in cost and what were the economics of ethereum and data availability layers yep uh that's a great question so one thing about so aail so uh to just give you a bit of this thing on it so aail aail and I think also celest as well they don't Shard the data among validators um all of the validators hold the entire block data uh so you get that

guarantee like uh the full noes at least they hold the entire block data the light clients can randomly sample data to ensure the data availability and they also maintain a DHD for a distributed has in case of AA the light clients maintain uh a peer top Network that can also uh maintain the state of the network that can replicate the state of the network as well um now when it comes to tank sharding and any other future improvements that comes to ethereum it is so the basically the question is what is there today versus 5 years later right aail for example provides everything that

ethereum ethereum dang sharding wants to provide today not 5 years later so um that is basically first mover Advantage kind of thing um secondly obviously ethereum would be the go-to for many chair many rollups or many projects there but they'll have to wait for 5 years and once it does come um there is a lot more improvements that will still need to be done even you see it's not like a onetime solution like introducing blobs everyone things is like a onetime solution and it's done and we won't have any blob like gas fees issues anymore because no blob blob fees is keeping on increasing day by day like as demand increases as more rollups come now

imagine uh the the end game here is or the end Vision here is that rollups are eventually going to be more application specific and not only General uh uh General uh chains for example or generalized rollups you'll have application specific chains and just think of it like a microservices architecture wherein you have specific rollups that do specific things and if that is going to be the case you are not going to have like just one or two rollups you're going to have thousands or if not hundreds of thousands of rollups being built for specific applications that's going to be tackling that right and that

introduces the rollup fragmentation concept also but this there's going to be a huge rollup demand and ethereum alone will not be able to handle it there will obviously be need for external providers like a way and also there is going to be a need for solutions to basically uh interoperate basically connect all of these rollups in a way that you abstract away the rollup layer right you abstract away what Ro up you're using which ecosystem you're using are you using a layer are you using op stack op stack super chains are you using um some other random chain are you using this application specific

chain nobody wants to know that right when you're using Amazon on your uh browser for example you're not knowing you're you're switching from Amazon to razor pay to pay or do your payment and then you're coming back to Amazon the requests uh between these microservices these apis are not being known to you it's happening like in a snap and eventually we need to reach uh get this ecosystem to a space to a place where that happens and ethereum alone cannot do that aail has aail Nexus that's coming da is not the only thing that a is working on aale has uh Nexus happening it has Fusion happening so Nexus is a way of coordinating multiple rollups is a coordination Hub uh which

uh Aggregates proofs of of State transition functions of different rollups and allows asynchronous compossibility and atomic transactions between them allowing them to interoperate in a way that is never possible before so um it's not only about daas at the end of the day it is what it allows you to achieve uh obviously ethereum will have that uh like when tank Shing happens it'll have users rollups being built on top of that but there'll always be rollups being built on top of other external providers as well and also there will be Solutions like Nexus from a whale that is going to be able to interoperate and allow the unification of all ecosystems even ethereum rollups

even uh you know aail rollups and other ecosystem rollups like even if you have celesti based rollups they can integrate Nexus and they can inter they can interoperate among all of the other rups in the ecosystem right so that's the kind of the end game here so it's not just about da it's about what it can enable I'd say like that's just my opinion there but yeah yeah makes sense uh also another question is that if Danga if like aail while is innovate Sals are not doing dang sharting then where is the cost cutting exactly coming from because like yeah when you're submitting the State rout on ethereum that is like it's it's the difference between a single state

route or the entire transaction call data but the validators of aail itself they need to be paid rewards in order to store all all of this data in order to incentivize them to do this so like what is the economics of that uh I mean so every layer one has its own tokenomics in terms of like how their what are being distributed how how the validators are going to be incentivised I think I think that is um going to be there for every protocol um so I mean it's it's not only about the cost like I said right so there's a lot more to it um it is also about security it is also

about uh you know what can be done Beyond da and for validators themselves obviously I mean for any da for for that matter even if it is uh Celestia or I or near da or a vda they have they're going to have to incentivize validators in some sort of manner um that is probably going to be through a token or through other reward rewarding mechanisms so um in Avail for example the Avail token will uh be the one the token that is going to be uh going to be used at least now to secure the consensus of that Network but we will also bring in what I said so we ail da there's Nexus and

there's fusion and by Fusion like I mentioned the end game is to unify all ecosystems and if we want to unify all of these ecosystems aail is also going to be able to enable other ecosystem tokens to be staked on top of uh the da layer the Avail da consensus layer that means you can imagine you can imagine that you can stake your Bitcoin you can stake your ether you can stake any other token or any other rollups tokens on the consensus layer to add to the cryptoeconomic security of that consensus layer so it's not only about um re like you know rewards in terms of the the native token of that layer it can it's

also um other ecosystem tokens other ecosystems uh pulling in and providing their cryptoeconomic security on this consensus layer which will be unifying like everything together which is something that other uh I'd say no other layer is doing right now uh but yeah yeah makes sense makes sense yeah thank you for answering our questions and are there any other questions from the chat all right in that case I think it's a wrap yeah thank you Robin for taking this lecture yep I'm happy to be here uh

thank you for for having me actually