From Free to Fast: Product Tiers of Decentralised Storage - Viktor Tron

Transcript

Thanks a lot guys. Hi everyone. H I'm Victor Victor Tron from the S Foundation and over the next 30 minutes I would like to share a way of seeing decentralized storage you know in in a kind of manner that I think is missing from the conversation. basically not not as a single uh product but as a spectrum from free all the way to fast but only because these are the two uh favorite four letter um fwords that I wanted to put in the presentation title but it's actually all the way like the various layers and and and for properties so what are these h

so Yeah. So what we have is basically storage is not one thing. That's the premise. It's like at least at least these two we we know about that like at least these four different products oids. And I'm going to argue that even even if you conceive of decentralized storage as being the basically part of the back end of web three and even if you think about just one DAP or one application you you're going to realize soon that maybe even within one DEP you typically need more than one of these and therefore your favorite platform that that you want to use may have to cater for for all these uh for all these. So what I'm going to talk about

is basically how swarm can cater for all these uh product needs and what we're going to do is is a is a despite all the variety say every behavior uh in in decentralized storage we reduce to these two primitives. One of them is propagation and the other is persistence. Let's call it that way. They're sufficiently sophisticated words that they can serve as technical grounding. They actually about just moving data and the other one is to keep data and everyone else in everything else in this talk is just built on top of these and no extra layers might be need maybe a third one is going to be just added but that's a kind of economic primitive really. All right. So what the let's let's map these primitives on to

uh the the resources that they actually consume or they they depend on. So what the users view is that bandwidth is needed to push and retrieve content basically to upload to the cloud cloud decentralized cloud storage and download it and and for the storage they have to specify something not to be forgotten. The operator's view is that they relay relay packets or relay particular messages in the protocol and and that's that's that's how they use the bandit for and they also kind of reserve some this space for for uh for um uh storing stuff and as as you see the quality the quality expectation of

the two are very different as well. One is kind of just maybe low. Yeah. One need to be good quality for the propagation to be low latency, high throughput and well connected. And for the storage you need to be uh with large capacity and some redundancy and probably durability. Yeah. So in another way we can conceive of these two resources also in a physical physical way or maybe in also borrowing some sort of finance finance leies. One of them is flu and the other is stock. And as you see the nature kind of reflects this.

There's time wise the propagation is instantaneous. So better to be treated as almost real time. Persistence is extended in time. Uh the scope of propagation is local. You can locally decide where to propagate certain things and persistence only globally evaluated and you don't even know where things are stored necessarily. And there's also this this notion of dual basically which will be maybe important in later of this talk when we talk about how to compensate for for the other primitive basically by frequent frequent retriever you can compensate for persistence and with dispersed uh uh storage you can also compensate for retriever but let's let's let's jump into a bit more about

the incentives. So these two these two uh resources are accounted and and settled in very different ways. So as as you see in decentralized storage you want to think of incentives which are basis for self sustainability in the network. So as you see bend this is essentially instant gratification which allows it for be accounted and settled in a tit for tat manner between peers and that's swarm uses a kind of check exchange for this which is kind of a microp payment system we don't need to go into the details and on any kind of mis misounting you can just use this connect and that's how you punish things and the unit that you use is basically

bz per chunk per per proximity or the bin which is per per academia bin basically and storage on the other hand is premisary in nature that's why it needs an escro payment and and it's we use postage stamps so postage stamps is basically you you buy the right of of issuing stamps uh on the blockchain it's a batch of stamps and then you can issue them you can link to them to you can stamp one one chunk at a time and this kind of uh provides some sort of revenue to this contract and that revenue is basically distribut distributed redistributed to the to those that uh play the game basically node operators and if they don't then either they slash

to frozen or if they kind of misbehave and and don't store the stuff don't do all the stuff or don't synchronize or if if they kind of storing stuff that's not supposed to be stored etc etc they might be punished and the unit here is like busy per chunk per block obviously like how how how the unit is calculated So as you see there are different units, different time scales and very different enforcement mechanisms and so basically two two different markets two separate markets and and uh so this is this is where the stand the system stands today basically bandwidth incentives are done quite uniquely in swarm in a way that's I don't see in other systems done that's

basically you you you account and settle the cost where it's actually created And on the other hand, storage is based on this redistribution game. You're redistributing content to those that contribute to storage in a in a fair way. So let's let's let's consider some some of the some of these tiers like what what these primitives will allow you to have. So let's consider the free tier first. Uh at the moment you have time based settlement which is uh essentially conceives of bandwidth as a as a re as a renewable resource that kind of replenishes over time and if you if you want a free bandwidth then basically you have to rate limit this the flow rate

limit the it's kind of a throttled uh experience and I think it's a bit bit messed up. I think it's really bad UX that especially for new users who want to use the free T just for trials and maybe maybe some form demos of things and and introduced to the to the the system. they really too difficult. You don't want to show a kind of a really rate limited and and you you really don't want to show them side that's kind of worse quality than than what they would have because you want to convince them to to have some retention of the network. You want to convince them to stay. Yeah. On the storage side, the postage stamps have a have also a

kind of price oracle which are acting acting on supply and demand in neighborhood. So that's how the postage stamps are are priced and there's no no equivalent on the on the on the bandwidth side to this at the moment. It's pretty much fixed price and these are not good. And also what I'm going to what I'm going to present or what I'm going to propose is that after after a re reframing of this these problems what you're going to have is basically this this this change here uh what you're going to have is is a proper true uh true through true uh free chair in the in the storage that is insertion to cache. Maybe that's a good good question to ask in the question and answer

section like why why this is definitely needed and and um and you also have free and the propagation but it's a it's that free is kind of just provided as as an X layer in fact what we do have is is abolish the the the genuine freess of this of of provocation and this is because even if persistence was free and where bandwidth then when you upload stuff you would have the immediate uh spamability problem. you could just spam spam the network and basically punch the entire cache with the with whatever whatever spam you you you're pushing to the network and therefore so either

either the bandwidth or the postage has to be nonfree in a way but we can still provide the freess with limited credit basically lending construct and it's basically the same primitives as you see but a much simpler model and much cleaner UI UX and the rest of the talk is about how to make it work in practice and what kind of other constructs we can we can kind of uh mix from these really. Um so continuity services what is continuity? So continuity is basically uh well let's let's let's first focus on on the two storage types that's that kind of emerges from this uh from these two

primitives dichotomy. So cache is related to propagation. Basically the nodes it's it's it's in the interest of the nodes. It's it's basically the best strategy to cache the content cached content that goes through them. If stuff is retrieved uh from like basically backwarded as from the forwarded requests they backwarded in the chunks in the in the kind of retriever protocol. you' catch them and and uh therefore you can you can serve them to to uh to other other requesters later and so it's it's in your in your interest and you also the other other way to to get into the cache is of course to to let let the post sense expire and when from the node operator's

perspective if the if the post is sent expires then you put put the chunk in the cache on the other hand reserve is basically a persistence commitment so you dedicate your this space to to to to to you reserve it to a particular set of chunks which are are postage stamped and you you keep them until the postage stamp expires. Now the so the so basically you have a you have a from the user's perspective the cache is something that's that's defines the stampless upload. So they basically it's it's the target of the free free tier and the reserve is something where you have to pay not to be forgotten. stay retrievable and and um so let's see what the what what the

pain points of the UX are which kind of might break the adoption really. It's not really the the the core protocol that might be might be lacking some feature but maybe maybe some sort of UX pain point that still prevail. So what's happening here? There's may there's essentially three three uh UX pain points here that I mentioned. When it's posted, it's anxiety. What does it mean? It means that the post systems as we said the the price or is basically changing the price as we speak like basically based on supply and demand or supply per demand and therefore therefore it can it can change the price and what's what's currently uh uh deduced as the price of

the storage rent on every kind of round of this redistribution game might change. And since the expiry is quite naturally defined as a expiry happens when when when you when you deplete the funds on your contract or sorry on your on your on your batch on your stamps basically then basically TTL or time to live or expired date is basically going to be uncertain. It's quite annoying because like people obviously want to want to know like where where the stuff is expired. It's also not very compatible with this upload and disappear scheme that we always say. Upload and disappear by the way doesn't mean that what you upload disappears but you can disappear after you upload stuff. Yeah. So uh upload and disappear is basically not not really compatible

with like anxiety that you always have to look whether whether you post this stamp expired. So you won't really want to either certain TT or some sort of subscription based idea that that that we are our our favorite centralized providers also have usually the other one is the checkbook friction. what checkbook fiction means that in terms of bandwidth I mean you have two options if you if you want to pay right now because like remember there's no no free bandwidth anymore so there's a kind of earn earn to spend uh idea so you basically you have to work your way to the so first like if you if you if you don't have any any cash or any any business have to work first and uh create some some some income and therefore that's how monetize and deploy

your contract and then you can spend from or uh but it's basically it's bit annoying that you don't have a zero cache entry where where you would um where where you could just even even you can on board on board the user and at the same time use your use your bandwidth as and to to retrieve stuff and the third one is like basically the underlying underlying reason for for for one is that you have a the busy price moves and the users want to kind of predictable pricing really and they don't want to and also there's also kind of another annoying fact. It's also the token price is volatile but that's solving that is a little bit different problem but but if even if even if the

token price is kind of fixed or considered that a non problem a nonissue because you have your BZ even then even the price of unit price of storage might change and you don't really want to be exposed to that kind of commodity price volatility. Yeah. So um so protocol is fine. The user experience is what brings adoption and these and and basically what what we have to which okay so we have to provide a continuity service that's what I call it continuity uh which which basically guarantees some sort of outcome for for

for the user and helps with the bootstrapping and and the smoothing of the volatility and also with the zero cash entry in the in the case of bandwidths. Basically, this gives gives zero cache entry and no with a non- throttling experience and also kind of continuous access regardless of whe situation you're in and on storage you also can to solve the upload and disappear problem. Basically make make sure that TTL the time to leave is is certain and people know no I'm not anxious anymore there's no this post anxiety. Yeah. So basically then as the known exposure to to the commodity price volatility and so this the and and when we think about it there's a genuine

genuine user need and on the other hand there's a genuine hordless need to have some sort of productive options for your capital and and luckily this meet basically if you reframe it as a as a capital availability or capitalization problem then you can you can have a solution which provides some sort of some sort of restaking or basically tie up your liquid stake your liquid stake state VCC into into this subscription contract and and ultimately kind of guaranteed some sort of profit which is basically the premium that that absorbs all this risk.

It's basically more like a insurance insurance arrangement and you can you can in fact characterize this this scheme with with basically have three roles have subscribers underwriters and executors under that are the kind of holders who would hold the capital anyway and they want to seek like active yield on their on their holdings and and they they they essentially happy to happy to take on the boundary risk and have almost guaranteed profits. and and the subscribers that are the users which you want to control on ramp want and predictable pricing etc. And you want the executors which are execut are essentially like smart contracts

themselves or things they just have to act deterministically even if they have some offchain component but they don't have to have any set of offchain trust and most importantly they they are not assumed to be node operators. Yes. Because in fact what's the ultimate interesting thing is that is is is a decentralized network is mostly aligned in their incentive is the node operators pay the pay the stake and and um and take the take the profit from whatever the ultimately the the network is making profit from. But but in in fact the if it needs some sort of high stakes or if you need to provide capital it's most of the time we run into the problem that those that hold the capital and and

would be able to pay stake they're not the same demographic as the ones that would be geeking around and and then operate nodes. So actually an interesting problem and if if if you want to directly solve this might not be might not be easy because the stakeholders if I mean it's like basically more hazard. Yeah. there a bit. And on the other hand, if if you if if stays this pure end and basically all the all the contribution that this continuity service providers do and not doesn't require online presence in the in in this particular network just just the presence and everything is executed by the smart contracts then what happens is this s suddenly become

very very useful for the for for the users. Okay. So basically continuity services where where hlers and users needs meet. Yeah. All right. So the what I have to convince you is that both bandwidth and and also storage the the the the risk is bound. It's kind of all kind of juggling around with this phase of of how how the how the how the risk is bound. It's it's not necessarily is obviously but but uh but if you if you basically cap the credit and you make some assumptions about the rate of people who would default the rate of people would not have kind of this incentive to be retained in the network

then then basically loss can be modeled as some sort of uh with to be limited and also with postage stamps topups it's you can I mean you can you can imagine that during the limited time of of a few runs where you can guarantee the price the the price cannot move the price so much that it's it's not going to be remain bounded and if the if the risk is bounded and of course of course every loss of risk or every loss every type of risk in terms of the potential loss can be factored into into the into the premium that you pay basically can be a risk adjusted uh

profits. Okay. So, um what so this what this is what I just said basically and and if if you want to take away basically the the the single econ economic takeaway message from this talk basically this should be this should be the uh this should be the page for it. um um so um as I said like boundary risk basically let's let's say storage so first first u let's consider like first the cash let's revise again this there's cash that's um also in in the interest

of the nodes to to to keep and that's why it's it's it's free and and what what this guarantees is that you can earn earn cash basically earn the bandwidth incentives from serving these these things which are kind of not not necessarily at the place where they would be normally stored in terms of the stores storage model of swarm. Yeah. So storage model of swarm works in a way that the in in the same place that you would normally find uh information about who hosts the data in in in normal DHD like in bit torent IPFS etc. you'd find the actual data. So and uh so basically the it's it's very strict storage model and and also kind of what makes it

possible to have a balanced load balancing between the nodes is actually by the the property of of addresses which are kind of hashes content to the rest chunk store content chunk store and that's what makes it possible to have a load balanced that's possible but what enforces it is actually is actually the postage stamp the batches which are kind of global in a way that if you postage stamp is is a is a is a volume of stamps that you can issue but if you issue some some stamps to this part of the network then you also have to issue that that part of the network. Yeah. This is basically as if a book of check book of stamps in real life letters would be

something like if I send to two letters to my mom in Budapest I have to send two letters to my to my uncle in Miami. Yeah. Same time or or lose it but then in that case it's actually the skew is kind of exponentially costly. Yeah. So uh basically hash and and reserve the p storage that's so the reserve is the paid storage as I said and basically what it what it guarantees is it guarantees some sort of hot availability and and basic basically an arford local replication. Now what we what we want to uh also kind of uh include is u is is code storage but let's let's let's like what we call pinning. So what pinning pinning pinning is like cached.

So there's cached content reserved content and pin content. Pin content is what you kind of pin to your local node or print and any node operator pins to that node and and and serves it from there. So u so what what you need for pin content to be served is a modification for the for the retriever protocol which which currently is when when you request a chunk then that chunk's address of course is taken as the as as the ultimate destination that the request has to be uh reaching and and it's it's served by anyone on the way from the cache or ultimately the one uh that's that's close to the close to the close to the chunk and the chunk is stored wherever the to to whichever node

is closest to it and and and and and that's that's in in its reserve. Yeah. And and the pin content just doesn't is exactly use the same routing. It just use a different uh different source or input for for what the what the what the routing is basically host let's call it host based uh routing. Yeah. So uh the pinning comes basically in three flavors and and so in well let's keep this and maybe maybe I can return to this in the in the question section if you really want that this this kind of routing issue but the pinning pinning basically comes in comes in two in in in three flavors. So if you if you think of it as as as also free tier in a way that if

you if you if you operate a DAP you may be storing some content for your users and and so so you just pin it and and and therefore like storage of it you can rely on your own infrastructure in a way you trust yourself and it's a different trust model but it's the same same uh same retrieval mechanism and the same kind of caching above it. So basically if you if you think about it's not it's a little bit better than like just a web two solution of a of a single server serving it. You can also make it a bit better by by still having a private distributed cluster which kind of maybe maybe the best is that you you actually distribute your content in in a in a balanced way

and and you assign some some to the servers to maybe to eight eight part like eightway partitioning of the address space and you run an eight eight clust eight cluster or an eight eight node cluster and basically serve the content something that is still private and distributed but it's still kind of a simple private simple simple trusted model but what you what you can also have is a decentralized service network which I will call code storage is a public public pinning network of swarm so basically what you can have is a is a smart contract that allows you to to maintain a balanced set of nodes serving a particular each serving a particular

partition of the of the address space. So basically as as you see for example in this in this particular well that would be 60 16 nodes partitioning all the all the all the different exhaustive list of four four bit uh prefixes of of the address space and anything that's that starts with these these prefixes would be stored by an particular uh node in in this balance network. So and and and the actual one. Yeah. Okay. Or two maybe. And and so but until three until 30. Yeah.

>> Until 30 and then 30 minutes. Yeah. >> No no no. >> I'm come on. You took 15 minutes to set it up. So >> yeah, I really want to have at least 10 minutes for a question. So yeah. And I started when it was 25. So I'm 25. Can I have like maybe five 10 more minutes? Yeah. Thank you. Okay. So I quickly go quickly run through this. So basically what what you gain with this extra pinning service is basically that this can act as a as a code storage in in in general but also as a code storage back end that will allow you to to to seriously reduce the cost for node operators. How do you reduce the cost of cost of node operators? Well, you

actually increase the storage that links to one node. But you cannot really extend it forever because that because the it depends on how how much requests they get. So depending on how swarm is used whether it's kind of always used as a just archival style of storage where where where things are stored but not really retrieved like for example encrypted user backups or or it's used like for video videos live streaming which is kind of almost almost no storage is needed everything through through cache but it's actually lot of lot of retrie lot of retrie is is happening. the two types of use and and the kind of and the the the the ratio of these kind of use cases are not

very not very clear what's what what's going to happen. This might even change over time. Anyhow the the the other ways to to reduce cost is obviously reduce the local redundancy. So as as if even if you even if you want to current with the current system if you have four forward redundancy no matter whether you can reach the the the the economy of scale of like an S3 S3 cluster like purple Amazon Amazon cluster then we can't really offer better price because of the four replication obviously it's like huge huge extra extra cost but As we know we have eraser coding and with eraser

coding it's becomes possible to actually not even have at all any availability any one particular neighborhood at the same time. So it allows for a for a gradual reduction of the of the redundancy factor local redundancy in one network and actually the code storage can be used as as kind of guarantee the retention for the single primary neighborhood. So primary address neighborhoods and it's very important to have primary and secondary addresses. uh maybe maybe a question section I would also like to maybe answer this if someone asked like why why it's needed to be separate but the but code storage like kind of as as a fallback recovery mechanism to to to retrieve to retrieve the the chunks by their secondary

address. Yeah. All right. So uh just just a few slides and and then I go to questions. Yeah. So basically uh yes this I can skip definitely and just just a few words about the the arbitrage pricing because I mentioned that it's very important to have to have a proper pricing because like sw has this idea that that so actually the decentralized storage and and in fact this web three back ends should not be hand should not really be uh served as a kind from a kind of as as a kind of marketplace for storage really because it's a it's a

it's a market but but but the pricing should be should be centrally uh centrally kind of set in in a way but cent for for for the whole network and and and therefore you you don't have to bootstrap the markets but also but on the other hand you have to have some sort of very clever pricing mechanism that's in in the in the case of storage is very clever to supply and demand but in the terms of in the case of What's what what's the what's the way to set the the the the price of bandwidth for example by anyway. So what what we have is basically a system where you can where can the markets decide in a way and no so no governance no committee and no

discretionary ways to like no basically no voting on prices and stuff and and the way it would work is that operators arbitrage. So the supply side and demand side arbitrage. So supply side arbitrage operators would would would have these options to dedicate that this to to cash in which case you they they earn from the from the bandwidth or so that's these are the kind of forwarding nodes that that just act as act as a act as a yeah forwarders of content and and they just they just serve whatever they just reserve whatever data it's in the it's in the cache and it's the reserve nodes which play the redistribution game and

there's pinning spinning pinning stores which whatever whatever you want to pin and serve from there local this or or have the code storage mechanism and also there's there's demand side arbitrage which in which case you can divert to the cache to the reserve or pin depending on your on your actual storage need or depending on this kind of dual so what what's so this self-regulating and and sorry self-regulating nature of this whole thing. So basically if if the storage gets expensive then people just uh people just uh lean on the cache and and and kind of arbitrarily maybe regularly re retrieve that content even even if they even not necessarily need

it and therefore the the operations operators rebalance it because because that's will the storage price down and the other way is like if the bandits gets expensive then they can higher the level of erasia coding which makes the dispersion happen and quite interestingly if you if you disperse the content more in sparm it means that you pay less for the bandwidth on average uh by by by the unit of of chunk. So it's actually very very interesting how how this rebalancing can can actually provide waste for for both users and and operators to to influence the price if they if they think it's kind of mispriced. All right. And in this my conclusion is that you have these

basically three three commodities. I mean two two of them are kind of primitives of you know how you move data how you keep data and the continuity is basically this it's an economic primitive of how you how you create an underwriting service from from basically holdless capital and helping helping some of the some of these uh some of these outcomes to to happen some of these guarantees and basically the full composability happens this way that the subscribers by continuity the f price underwriters supply the capital and operators basically supply the resources and and and and and take take the revenue from whatever the subscribers pay for ultimately. So hopefully it's

it's lower friction and and then hopefully it will give some higher adoption for for the network and and uh what we what we basically ultimately conclude is that this will this will give a different sort of a different definition of swarm basically is a market for bandwidth storage and continuity and what I want to conclude you with and if you want any have any questions quickly I Just a little bit.