Storage Incentives in the Swarm Network / Daniel A. Nagy

Transcript

foreign [Music] so I'm going to talk about the storage incentives and the Swarm Network that's not the only incentive system there's also a bandwidth sharing incentive system but I'm not going to talk about that one so now I'm only going to discuss how the Swarm Network instant devices nodes to store what they're supposed to store and how it incentivizes users to only upload what they really want to be stored so what is swarm

so there are three things that are called swarm which is a decentralized storage solution which kind of encompasses everything it's a network appear to peer Network and it's also a Content address namespace so it's in that in that way it's similar to ipfs but it's dissimilar in that it also is a storage Network so it actually stores data not only indexes where everything is stored uh so here's a quick uh overview of how swarm works very very superficial introduction so all data is broken up into small four

kilobyte chunks they are redundantly distributed among the nodes uh it's all stored in a DHT meaning that node addresses and the chunk addresses come from the same namespace and nodes actually store what is closest to them so the content is divided up based upon the first few bits of their content address and those bits determine which set of nodes is going to store it redundantly and these groups of nodes that are supposed to store the same content we call them neighborhoods so when I'm talking about neighborhoods of nodes

this is what I mean why do we need incentives well first and foremost is to protect against spam so before the times of the blockchain uh there there were several attempts at creating uh distributed content address the storage Networks and basically all of them were swamped with frivolous content that nobody really cared about but they were just overwhelmed by uploads so and they did not reward the participation in any meaningful way so basically what happened that it was that the amount of data that needed to

be stored in them just grew and participation became more and more expensive and then basically participants gave up on them so we need an incentive system in order to motivate participation of stores and also we need an incentive system in order to make the allocation of storage resources efficient so basically to store as much relevant content as possible and here the emphasis is unrelevant because we actually want those who upload to prioritize what they actually really want to upload and for

how long they want everybody to store it and we also want to provide adequate redundancy meaning that there's we expect some churn in the network so nodes come and go and we want to make sure that content is not lost because some nodes go offline so how do we implement it uh it's implemented as a set of smart contracts on an evm compatible blockchain currently we're using gnosis uh originally we wanted to do it on the ethereum main chain but unfortunately it has become too expensive and we're not married to gnosis so

basically any uh any layer to solution that fits the bill is a good candidate for running this incentive system and we have an ER C20 token that is issued and redeemed on the mainnet on ethereum Main on ethereum mainnet and we chose gnosis because it has good air seat 20 bridges that are sufficiently trustless and reliable and the way the whole thing is implemented is a shelling game so this is how you make these smart contracts aware of what is happening of

chain namely in the in the storage itself so shelling games are a standard way of making smart contracts aware of some state of the world which is outside of the blockchain so basically in two sentences what happens is that uploaders pay and stores get paid so there's a there's a way of redistributing these payments to the stores and I'm this presentation is basically about how upload their spam then about how stores get to

receive this payment so store upload their spay by fixing so-called postage stamps to each of these four kilobyte chunks postage stamps are purchased in batches by on-chain transactions by registering a public key and the batch size with a smart contract and once that public key is registered you can attach a little bit of metadata to each of this of these chunks and seal it with the digital signature that can be verified with this public key and this way uh each uploaded chunk is marked in a

reliable way that it has been paid for so uh once the network receives a chunk it knows whether this chunk has been paid for or not and it only needs to look into the uh into the state of this contract to see whether the payment of this chunk is sufficient because all these postage stamp batches have a balance associated with them which gets regularly debited so there's a rent that uploaders need to pay in order to keep data on the blockchain on

the sorry in swarm and so this rent is paid in this token BCC per chunk per day let's say so basically it's proportional to the amount of data that you store and the duration of storage the rent is continual is going to be continually adjusted according to supply and demand if there's time I'm going to talk about it if not then please ask me and I will answer questions about it but the short story is that the more saturated the network is the higher the ram becomes and if the network gets

if there's less data to store then the rent goes down and batches with zero balance are of course invalidated and the corresponding content is garbage collected uh so now we have a mechanism of collecting payment from uploaders and marking data that needs to be stored in a unambiguous fashion and we need to redistribute These funds among the store nodes so how do we do it so it's a randomized game where participation is tied to storing stamped content so if a node is storing what

they're supposed to store then they are going to be rewarded from time to time uh and because we have these neighborhoods of course they're not going to be storing exactly the same data because there are some latencies involved in how data is propagated so there's no uh no exact consensus on what needs to be stored but there's a rough consensus so all the postage stamp chunks whose content address begins with the same bits with which they address the overlay address of the store node begins must be

stored by all the store nodes that have this property so essentially all the nodes in a neighborhood agree on what needs to be stored although it's not an exact match so we cannot we cannot verify exact match exact equivalence of what they're storing but they still uh can verify each other whether they're storing the same thing and in order to be severe resistance so in order to prevent uh

like fake nodes that use the same storage but just pretend to be more knowledge than they are notes need to put down a stake in order to participate uh and then we have this redistribution game that happens regularly in rounds so first we have an on-chain oracle that we picks a winning neighborhood this is done with uniform probability uh independently of how much stake is in each neighborhood so basically each neighborhood has an equal chance of being selected then nodes in that neighborhood commit to and then reveal

the Merkel hash of a deterministically chosen random sample so the reason we're doing this random sampling instead of building a Merkel hash of everything is precisely in order to have a little bit of tolerance to not storing exactly the same so if the random sample the random sample has a large chance of being the same even if there's a few chunks of difference between what they're storing because the chunks haven't arrived to some of the nodes in neighborhoods yet but they are expected to agree on that random sample and then the correct sample is chosen

with a probability that is proportional to Stakes so an alternative could have been that the majority stake the majority stake vote is taken as the source of truth but we decided against this because it would have meant that with the large enough stake you can reliably take over a neighborhood and without any risk basically sensor and with this choice that the correct the this source of Truth is chosen in proportion to the with the probability proportional to Stakes we have the

property that usually the majority most of the time the majority stake will get its way but even the majority stake is not safe from having a different opinion from what the truth is if they play a dishonestly so this is an additional incentive to to play to play honestly because otherwise even if you're in a majority you risk to have your steak Frozen So currently if you're not revealing the same uh the same Miracle hash that the source of Truth revealed then you get your Stakes Frozen uh maybe in

the future we're going to introduce slashing currently we're only freezing because we don't want to punish too harshly the participants because you know our software can have bugs in it so there there can be there can be reasons for losing the game even if there's no bad intent on your part so currently we're just freezing steaks but that might change in the future and then from those who did reveal the uh correct marker root again with the probability which is proportional to the stake the winner is chosen and the winner gets the reward

uh the smart contract is structured in such a way that the winner is the one who executes most of the housekeeping and therefore pays the gas price for it so these three uh of these three kinds of transactions that are submitted by participants the claim payout transaction is by far the most expensive and the commit and reveal transactions are actually cheap and there's also some technicalities like when can you top up your Stakes we're also uh uh thinking about Stakes to be withdrawable

after some grace period so but these are really technicalities so this is the basic mechanics of redistribution and just to reiterate the only safe strategy is the shelling point which is to play honestly and since everything every reward is proportional to Stakes it's never it's never worth operating sock puppets because you can achieve the Same by just fooling the stakes and operating one node honestly so basically this is the incentive system and

I would also like to discuss what so how swarm given this incentive system can possibly be used for roll-up data availability because superficially it seems like a perfect fit that here here's a network where you can cheaply store data for a potentially short period of time so you don't have to store it indefinitely you don't have to pay for it to be stored forever you can actually store data for a given period of time there are some nodes that are in some device to store it uh

so it looks like uh it's it looks like a good match uh the problem is that we cannot provide swarm cannot provide data availability guarantees of the kind that for example dank Chardon can that you know given certain assumptions and this regular sampling you can give some number some number really close to one which is the probability that you're going to get your data back and also there are these more philosophical questions that data availability can be subjective so something that is available for me may

not be available for you uh so what we came up with is that we can assume that layer one data so what's on the blockchain is available to all but it's very expensive so basically you want to use that only as a last resort and in order to provide some kind of guarantee of availability instead of the statistical availability you can statist statistical guarantee you can provide a economic guarantee that there's some State guarantor which

can be the sequencer of the roll up themselves so they have a stake and they have an Unchained commitment to the Merkle root of the data of the transaction data in this case and the duration for which they are making it available and what they do is they upload the data to swarm and they monitor the smart contract because the rent might go up and down and if it goes up then they might need to top the batch price up so basically they need to keep the data available and pay for it

and there's the smart contract to which they committed and one particular transaction type with which you can communicate with the smart contract is an unavailability challenge that you can claim as a user that hey this specific transaction which is the number whatever in this Merkle root in this Merkel tree is not available I claim that this is not available I could not download it from swarm and at this point if challenged the guarantor has basically two options

either they respond to this challenge and they upload the whole transaction to the uh to layer one and in this case it was very expensive for both of them so both the Challenger and the guarantor had made a layer 1 transaction and they paid the gas cost of it but the data ended up on layer one where we assume it to be available to all or if it's really not available then the guarantor will lose this amount

of money and maybe this model is actually workable for Roll-Ups to a store transaction data in swarm because uh most of the time these challenge responses will not happen because if the data is really available you're not going to do the challenge because it costs you and you gain nothing so it's basically just a griefing attack but it's it's costly so it's not it's not free and it's relatively cheap for the uh responder so it's it's a griefing attack

but not a not a very not not a devastating one so if the data is really available there's no reason to do that so it's not going to happen so it's it's it's only there as a deterrent and the storage incentives that I have uh that I have introduced in the first half of my presentation actually incentivize the participants of the network to store them and there's a separate set of incentives built into swarm which are the bandwidth incentives which incentivize the nodes to actually

make data that they do store also available so there's an incentivized Network that is incentivized to store and make the data available on one hand and on the other hand we have the state guarantor that is going to lose funds if he doesn't keep this data on this network but if something happens like something extraordinary happens then of course we still have the fallback to resolve the dispute on layer one and in the worst case the sequencer will have to upload part of the transaction data to to layer 1.

uh so thank you that was my presentation I see that I still have more than five minutes for questions so please go ahead and ask me very much Daniel well done very informative and yes we would have time for one or two questions from the audience if there are some thank you Daniel so my question is that the four kilobytes looks like a small choice to me because because just a digital signature has some signs and I'm sure there is a lot of metadata potentially for for every chunk so how was that how was that chosen that

the chunk size is four kilobytes so the four kilobytes seems to be a kind of a popular choice in chunking up data uh it's so it used to be the basic unit in which ssds operate that's not the case anymore but when this Choice was made it was still the case and also there are other systems so one example is a SQL database called sqlite very popular all of you have an sqlite database in your pocket because probably that's how your mobile phone stores the address book and many other things

and one of the things that you can do with sqlite is to uh so sqlite transforms SQL queries into uh file operations on one single file which you can also transform into uh into query into range queries HTTP range queries which are compatible with the Swarm API so what you can do is you can store a huge sqlite database on Swarm and have a client-side sqlite so there's a sqlite implementation in JavaScript on

the client side and then you have the full power of SQL queries and you can very efficiently access swarms so this four kilobytes comes from the fact that many other systems have this kind of Chunk size but it's not dictated by any and I think in particular okay cool I have one more question but maybe go ahead while you're there okay cool so my other question is uh so this is basically kind of similar to atarounds staking I just also need a hard disk right so in some sense your your competition is a terrible may not

sticking right if they are paying let's say five percent or I don't know three percent was the current percentage then to use my money and make my money work on Swarm I also have to buy a hard disk and do a little bit more than ethereums taking simple main Mainline terms staking so my question is that how how much free space do you have in the calculation that how much the the interest on the mean at can increase before it's totally not uh possible anymore to use one because everybody will just go to stake at around or may not

so as I said we have we have a mechanism that adjusts the rent and if nodes leave the Swarm Network because it's not worth taking by the way I disagree that operating swarm involves more work than validating ethereum validating ethereum is also hard work uh it requires different resources but anyway if nodes are leaving the Swarm Network it creates a shortage of supply when we still have a lot of demand meaning that this press adjustment mechanism will increase the rent and will keep increasing the rent until this situation persists which will increase the kind of interest that you

earn on on your stake in swarm so there's a market mechan there's going to be a market mechanism that adjusts the rent if nodes are leaving the network only until it's reasonable right so yeah of course an AWS is cheaper so how much Levy do you have compared to AWS or something like that uh so I would say that AWS is not a direct competitor so unlike some other decentralized Solutions and here for example storage shape being a prime example so they want to they want to compete head-on with AWS I think swarm is not a competition to AWS in that it

like there are many use cases when AWS is the preferable choice swarm is a decentralized censorship resistant Network which means that it serves the needs of those who need this kind of decentralization and censorship resistance which AWS obviously doesn't provide and it might have a very different price from AWS okay we have one minute and 20 seconds was there one more question the lady here on the side uh yes uh can you elaborate a bit more about the rent is adjusted to the available estoras do you mean uh also

the available storage in a certain neighborhood and if this thread is adjusted how does it work when you have a disrupted application that is taking a lot of space and what happened with the previous store data right so uh so the incentives are laid out in such a way that we can kind of count on nodes being uniformly distributed across neighborhoods so all the neighborhoods have roughly the same redundancy and same roughly because it's never equal never exact and this uh rate adjustment mechanism

works as part of this iterated redistribution game and indeed each round only looks at one neighborhood so if in one neighborhood we have a shortage then the rent is notched up a little bit and if we have an abundance then the rent is notched down a little bit and it only looks at one neighborhood but in each round we're looking at a different neighborhood meaning that if this is just a statistical fluke that due to a random chance there's one neighborhood that is underserved it will not have a dramatic effect on the rant and everything that happens in the rest of the neighborhoods

will kind of equalize it out but if there's a systemic shortage like there's a lot of data uploaded to swarm it means that because of the uniform distribution of the hash function and the uniform distribution of nodes we can expect that we're going to have a shortage in the majority of the neighborhoods meaning that in each round of the redistribution the signal will be the same to hey increase the rent increase the rent increase around okay thank you very much we've come to the conclusion of this presentation Daniel thank you very much very interesting I assume you'll be available afterwards yeah sure for some more questions thank you a round of applause

for Daniel thank you [Applause] [Music]