Technological and economic solutions for data availability / Viktor Tron

Transcript

foreign [Music] thanks for coming to my talk can I start so so this talk is about data availability or is it really in recent years this term was used to refer to the problem of persistence and retrievability guarantees of such in transaction data in particular and most

relevantly data transactions submitted to ZK relapse which state validators of Road apps need to reconstruct the state but my talk please but my talk wants to broaden this context admittedly and maybe provocatively hijacking this consensual narrow meaning somewhat when the founders of ethereum re-recognized the fury concrete blockchain as a layer of global secure computation they started imagining what a complete computation environment would look like an infrastructure underlying the newly decentralized internet then they turned web3 who remembers the word computer here

components of this architecture if this infrastructure were conceptualized using the analogy of computer hardware the ethereum blockchain being the CPU of the word computer they envisioned the hard disk for data storage codes one and messaging layer called whisper referring to this as the Holy Trinity of components [Laughter] for various reasons manifesting this greater agenda proved a much harder not to crack and subsequently fell out of scope for this hearing Foundation nonetheless research and development in this direction never stopped so the ambition is to realize the serverness internet

and the data layer Services purpose [Laughter] in this context a complete future proof base layer infrastructure data layer serving the ethereum ecosystem does not only have a wider scope of uses on its agenda than the data data available requirements under this narrow sense of this availability these aspects include

representation the missions and persistence the first constant concerns what form is the data accessible pertaining to units of data units internal structuring linking below jobs to each other and categories of content in more familiar terms it is concerned with chunking addressing mutability and searchability of data the second aspect is confirmed its previously aspect who can access the data and who knows who can access it in more familiar terms is concerned with encryption Access Control censorship resistance and possible deniability

the third aspect is concerned with how long can we access the data and how certain can we be that the data can be accessed now let's look at some of the major use cases such an ecosystem by generating Italia and formulate requirements based on the needs of these use cases starting with what the SE availabilities in the narrow senses concerned with support for blockchain related data secondly the problem hosting application code and Associated static assets of course don't get confused this includes JavaScript

well assembly code that is diving Dynamic pages of the application device and thirdly personal and application data however we must not stop here we can generate data layer with the objective to power battery we do not only include storage of data we need to take care of Distributing transferring fearing and discovering data if you go the hallway and ambition the implement the serverless internet I mean 10 minutes to tackle the functionalities I carried out on the server namely carried out by server hosted databases server-side processes and server mediated communication and notification

subsystems this is no small feat let's zoom into these use cases and formulate some requirements the important data layer first of all let's look at the Block generated stuff we can regard the blockchain state snapshot for example as a baseline since it has no special requirements it is public it is only auxiliary data to help synchronization or a quick boot sapping and therefore has no requirements on data availability this contrasts with with the transaction data for rollap support which is strictly needed for better State validators and therefore have strong availability as well as low latency retriever among its necessary

requirements Main Street State support is an interesting case because it raises the possibility that the native representation of ethereum State namely in the form of Patricia Trice Maps directly to to the distributed so it's representation namely blocks of data corresponds to serializations of the of the Patricia train Woods and the content address links are kept uh and therefore it opens the possibility of querying the ethereum state by traversal in the in the distributed storage system and

therefore potentially providing decentralized alternatives to to centralized providers of endpoints like inferior Anatomy or support for Life client requests or even decent times blockchain history now hosting that code is a whole different concerns about usability first of all it requires as we talk about browsing UI it requires uria-based addressing from this data this can be described as a minimum reference to a collection and the path based reference to an asset within the collection before Miranda requires mutability and can be solved by ens resolution on the blockchain and the letter requires representation

of a collection and index of graphic support unfortunately application code raises previous sequences ideally accessing a web free UI should not leak the user's identity and therefore requires request or anonymity and also really in the spirit of true permissionlessness making a available available UI must not be leaking the identity of the originator just think don't need a cashier for a second thirdly serverless data support should include searchable structured data necessitating sub-bn s mutability because of quickly changing data sets

optimized updates optimize the structures such as optimized for for updates of search and data indexes potential historical persistence and of course infinite scalability it's worth it but most importantly even if you also want to encompass communication and notification subsystems these are harder not to crack and it's not not there's not often that the current the existing Solutions offer a solution for this this programs namely communication and notifications so let's go ahead about

about the commonalities between back please let's talk about the commonalities between existing active projects that try to [Laughter] or something similar to that they were shared the idea that the the network that that this emergent behavior of of this data layer is is ultimately implemented by a peer-to-peer Network where where the players use some sort of underlay connection pattern but also you should use an overlay

connectivity and topology to make routing efficient is a common choice because it already serves as an Integrity solution that is usually combined with the model where where the where you can place the participating nodes in the same address space as the chunks themselves and therefore you can use the overlay topology to allow efficient routing to nodes that store information about chunks that are close to their own address they differ however in their storage model simple stewardship based as in the fast sharing type of Paradigm where you open

up your hard drive and register whatever content you have on there and that leeches becomes Cedars or the direct storage model in which my check will address the storage and the neighborhoods of a particular proximity [Laughter] chunks that that are close to them so they they do not always a location information about those chunks which are close to them and says as foreign in terms of

persistence guarantees we can mention three types of solutions incentives is concerned with with you can means of this replication resilient against the percentage there's a particular percentage of data loss almost efficiently however in a distributed way using error correction codes that is information theoretically the optimized stewardship here refers to a few things

yeah I refer to stewardship also in the sense that for example the fast sharing type of systems him have the origination of content based on stewardship since since they make the entire semantic content like the entire content that semantically relevant to uses in full from from there node you know sure replication is is also done by its consumers and therefore it leaks the information about the originator and

therefore kind of this is also true if if insurance is is based on a kind of marketplace because there was so it leaks the the if the insurance is provided that is by the insurer if the storage is provided by the insurer it exposes them and makes me them to Target the attacks no incentives positive incentives means that not storing the content gets rewarded upon

showing evidence of entitlement so so they they proved the retention of data and with the recency and relevance but they have no punitive measure Associated or implied if any particular piece of content gets lost this contrasts with negative incentives which essentially is insurance or particular content in case of negative incentives insurers sign up on a collection and in case any chance gets lost or goes missing the insurer can be challenged and stand to lose their deposit their Stakes what's this

the storage instances are primarily in the form of marketplace providers was the collection and the originator and they sure all like the identity and therefore provide an attack service and raises previously questions we argue for a solution by the primary mode of storage incentives it's just positively positive incentives what about the incentives please please [Laughter]

so now I'm gonna arrive at the tonight arrived at the most important part of the talk which yeah is that we do decentralized Storage Solutions are quite quite similar in some ways so they share a lot of communities which will lead you to think that any further

specific details about architecture is just an implementation of detail I'm gonna argue otherwise and I'm gonna argue that whoever architectural choices have major consequences on on the requirements and tibia my beer not really sufficient but even necessary requirements to satisfy some of the requirements that I identified in the first half so give me a second to read so USB storage

this recipe stories as I said means that the chunks are directly stored in the place with nodes with the dress is close to their content this is this implies that they need to be a protocol the content is uploaded and chunks need to be routed so the neighborhood that they belong and this already shows the distribution provides a proper cloud service and not only presents like the fashion solution and if and this allows for users to basically upload and disappear so it was it was a sorry solution it's genuine genuine

cloud service most importantly it contrasts with stewardship realist approach where where storing the file it's kind of if you're looking at your local storage it leaks information about the originator and therefore it is not censorship resistance provides local redundancy right away from the start and and most importantly incidentally actually not most important incidentally

it comes with the bonus that's a constant upload of chunks yeah provide a cover traffic for address-based messaging what is this messaging and this is messaging is when you send a message to another address and this is typically useful scenarios but you have the established a communication Channel with someone as a recipient you don't know it's basically the solution for spamming versus Anonymous messaging so this primary Insight here is that USB messaging things there

that messaging is secondary to storage on the first book it looks like it's just inquiring uploads you can hide the messages which are first for any third party and distinguishable from an encrypted chunk and therefore can only be opened by the recipient themselves on top of this on top of this it has water properties so some storage of chunks namely intentimization asynchronous delivery

and yeah and a couple of traffic to mitigate against traffic analysis attacks [Laughter] now this is the most important side probably so we mentioned that peer-to-peer Networks under a transport protocol in the corresponding content connection type and then over later foreign if one choose is a keeper life connection it has an immediate benefit that you don't have to establish a new secure channel for every request and appears that you select you can be sure they are online most importantly this enables

request forwarding as a type of writing which is in contrast with iterative routing iterative routing is when you iteratively ask your peace to give you crucial and closer piece to the to the destination address but you directly contact them these leaks the identity of the requester I have a relaying one request for money leaves the requester in the U.S and therefore allows private browsing and sales assistant uploads also due to the fact that in English no the Chang data

travel back Travelers back along the same path as it was formatted as the request was forwarded it gives the opportunity to forward us to to opportunistically cash the champ content which is by the way incentivized then with incentives and this help miraculously takes care of automatically scaling popular content so you don't need see this become richer type of stuff the same but that is kind of a very similar effect so the most important innovation that I

want to introduce is singular chunks so we talked about content that there's chunks so far which their integrity to the irreversible nature of the hash function because they are content to dress is based on hashing the content Implement their integrity by the attestation of their singular owner and this allows for construct where you have a bit predetermined address where you can put arbitrary content into and therefore uh someone can follow an address

which is by convention point for someone to leave a message this allows for for genuine communication channels to be established and if you Implement sequences of a conformations about sequences of predefined addresses it can Implement feeds which good for solving sabians mutability as well as uh as well as version checking in with the rest of it [Laughter] finally it is recording if cleverly defined once more

relatively small chunks then any reason to be suggested if it's downloaded and it does not always provide the benefit that is normally Associated which is increased data availability and higher certainty of persistence but it also allow us for masking I'm asking you the network condition errors of of a percentage of chunks or even condemn their connectivity gaps in your routing table and therefore

it puts an upper bound one retriever later see surprising this is this one and I will conclude by saying that what I talked about the takes away messages that if you want to implement a generic data layer that was there that is Enlightenment to the original ambitious of the ethereum project namely Italia underpinning the reason centralized serverless World by them and then we need to meet certain diverse requirements I argue that some of these learnable choices I I

not even sequential they actually have far-reaching consequences in terms of the satisfiability of these requirements and effects I argue that swarm has the features that allows it to comply with these requirements and in fact some of the features at least my claim is that these features are not only sufficient but in fact necessary this strong cream of course it needs a proof which is was well beyond this this hope of this talk anyway so and if you read the message that the the idea of the word computer lives on and if this channel data layer gets in production with two features and secure operation and proper

ly incentives then we can prepare for the next big thing where where the original Dream of implementing decentralized eBay decentralized all you can eat Etc can be finally implemented thank you for your attention and if you have to ask questions [Applause] thank you very much can you

all right thank you very much Victor um we have time for a few questions from the audience so if you have any questions my colleagues will come up with the microphone oh there's a question up there hi first of all thank you so much for the talk I had so much fun looking at those images that I could barely Focus they were so pretty um my second question is if you imagined a world where data access and the interaction with the blockchain type of application could be in the

hands of the many in the hands of every one of us what would that look like in Practical terms and I know it's it would be an oversimplified answer much appreciated thank you for your time foreign [Laughter] [Laughter] I was like who are the four F words

can be mitigated before efforts are basically various aspects in this freedom and fairness is as an obstacle this efforts are Force fallacy fraud and friction and with with basically secure secure layer for The Neighbors free trade for data economy at least of course it can never claim to be relevant for your local Baker but things that can be censorship resistant proper which is ever more proportion of the real economy

if it implements base in which the image the barrier of entry and the availability of productivity very different Vehicles decreases and the availability of productivity tools and tools that you need for connecting business and finding finding business relationships it's kind of trivially available and cheap then you can expect all the benefits of free trade which is the obviation of the network for State control and in general it's interesting or or any kind of stupid regulations and secondly as a consequence of the

resulting closely need Social Network on planetary scale it's a conduit for World Peace no less thank you are there any more questions all right I'll hand this over to the random C all right thank you very much again Victor put your hands together for Victor [Applause] [Music] foreign