IPFS

Transcript

alright shall we begin let's do it so my name is Juan one of the people behind ipfs and today I'm gonna talk to you about data and data structures so the talk is structured this way I'll give you a brief introduction to ipfs actually how many people here know already what I think if sis Wow awesome that's a great distribution so that'll go really quick we'll cover the IP LD data model and then we'll talk about use cases and examples of ipfs and data structures and so on alright so ipfs another name for it is the distributed web or the permanent web

or the merkel web in honor of ralph merkle and in a sense it's this kind of evolution of how the NRI on the web were structured and I think everyone here both coming from blockchain history 2 through aetherium and a whole bunch of other interesting technologies that are happening right now sort of see what's going on or what has been going on for the last three years as this this total reinvention of how to do distributed computing and distributed networks and in general and so this is this of course layers on top of the old internet and so on but what's really happening is that we're taking the web applications of the you know web 1.0 or web 2.0 world and we are separating them

from their physical location in servers and so on and distributed distributing them across different hosts so ipfs is a protocol to upgrade the web because the web has some problems I won't go over these there it's I think most of you are familiar with them it's an efficient doesn't work offline links break central points of failure and so on and the gist is to try and creates what the web look like if you took a Merkel tree or a Merkel chain and put it underneath the web itself so what would happen if you gained immutability for distributing content on the web I you can think of data structures like gits

and the amazing power of distributed computing that you get out of that or just version control the tribute version control you can think about aetherium and the amazing powers of contracts and running programs you know the tribute of sense and of course before the etherium bitcoin and so on all of these are Merkel trees and what ipfs really is is this massive Merkel forest that's kind of like the the intuitive description of IPF assets saying let's take let's write one transport protocol to layer underneath the web and other protocols as well and build this distributed data structure transport that is all Merkel addressed cool so imagine being able to

link across different aetherium chains and being able to link across Bitcoin and aetherium and being able to link to get repos and be able to link to BitTorrent torrents or being able to link directly into specific commits inside github so you could create a contract and say whoever you know patches this commit or something successfully get some money or something and in the ipfs world which is a world on its own there's a whole bunch of different protocols and we have this this view of the world that looks a little bit like this with a whole bunch of different sections for these protocols kind of on top of the on the lower level Network and this is of course all sits

on top of IP although it doesn't need to on top of transports on top of different routing different ways of doing routing we want to layer what we're calling IPL D which is the Merkel dog or the Merkel web as this common data structure that we can all use for expressing our data structures it's kind of the same idea as the Semantic Web but instead of just having an object definition imagine if the object definition linked between objects with hashes that that's the the heart of ipfs that's like the core what happens when instead of of having a URL that points from one object to another using a location what happens when you point using hashes and

this vide has all that's been around for a while but this talk will be mostly about what we're sort of loosely describing as the Internet of data or Internet of data structures to write on top of IP LD and so you've probably seen something like this where the ipfs model is you you take the original HTTP world of mapping a dns name to an IP address which looks kind of like this where you're resolving through the network to a specific host and you instead say hey let's just take the hash of the content itself and use that to address the files and it looks instead of that you get something like this where anybody in the network that has the file can serve it to you right this is all super intro

ipfs stuff and in a sense what's really going on underneath the hood is that that hash is a pointer to a directory you can retrieve the directory because you can check you can ask the network for the directory people can return the directory to you you can check the content make sure it matches the hash so that way you don't have to trust anyone and within that you look at look for foo and what that means is that inside that object who is a link that is pointing to another hash and you're using you use that to resolve for the next step and so on down right this is taking the idea of file system with inodes but replace an inode with a hash and of course this is merkel linking and you get full path

reversals down down this kind of graph so if you imagine a blockchain layered like that you would have a massive path going like parent parent parent Peron Peron Peron and someone down right but you imagine you you'll be able to index directly into a specific block and be able to go down into the transactions and so on and so the the IPS link sphere I guess is that you first have content addressed links where they start with slash ipfs then you have a hash and then the rest of that stuff and by the way those are shortened hashes I haven't yet found a way to compress information magically that's what that's suppose to be I shot 256 but it's turned so it fits here and yeah of course we have the scheme

identifier if you want to use it but we recommend that you don't use the scheme identifier because we the way we look at the world the regular w3c based scheme identifier broke compatibility with Unix so you can't we're making everything so you can layer it directly on top of any politics machine so you can mount it directly so you can you can mount ipfs and you can fetch data and write data to the network with standard old-style applications you don't have to teach them to speak over some other protocol and on top of on top of the merkel linking which which is sort of like a URI and I say URI with quotes because we do prefer people not to use the schema

identifier we'll have everything working and resolving with just straight-up paths on top of that we layer what we're calling also Meszaros links this is for David missouri's who author of Kadhim Leah sort of stellar author of SF s way before that and he came up with a great idea in nessip s and one go through the whole thing but the basic notion here is that the way you achieve mutability in ipfs is that you use the hashes of public keys and use those to resolve for pointers that have been signed with a private key oh it's explained better or somewhere in the web but there's more interesting stuff here but it's a very subtle idea that gives you a way to achieve naming and this is naming that it's not human

readable but it's a way to achieve naming that's cryptographically addressed or cryptographically mediated instead of through some central authority right and on top of that mutable link then you can layer names and so here's the old style DNA world we work exactly with DNS so you can add a text record into your domain and now you can resolve through IP NS or directly to my PFS and of course you can have etherium names the same way right and so you could you can imagine resolving if your names directly into ipfs content the exact same way what's amazingly interesting about these kinds of names is that you no longer have to have any machine running yourself you can create the content and add it to the network you can create the

names and the key names and sign contents and distribute to the network and then you can register some name and point it to the house of a public key and now you can just disappear just add content to the network and walk away and if the network will continue serving your stuff and that's you know an asterisk there everything will continue working right and that's something that's very different you don't have to maintain your own infrastructure so this is often non-obvious so I'll kind of go through the ifs content and replication model this this is kind of like what most people don't expect it here they're like what but it makes sense to go through so there are three important points that bear mentioning here other

than a whole bunch of others that you probably can assume but number one is that we want I P FS to be able to go as fast as the underlying network so we need to not we need to allow users to dial between efficiency and privacy so as you know systems like tour and ITP and so on that are focused around privacy have to do a whole bunch of work routing packets and moving them around and you spend a bunch of bandwidth hiding hiding the traces of people's accesses in the network and this is how you achieve privacy but that's not fast right so you're trading off the speed of transfer there to gain some privacy but

the users of ipfs are very some people really want that use case and layer ipfs directly over to our ITP and some users don't care at all about that they're using IP FS inside data centers or they're using it in some other way and they really want it to be as efficient as possible right like do you need to route through tour to go from like your computer your laptop to your mobile phone in the same room maybe not or maybe you do it depends on what kind of home you have but you know and another point here is that ipfs only downloads explicitly requested data you do not download other people's stuff and this is an important piece from a legality perspective right so there's been a whole bunch of problems around peer-to-peer systems where they download

other data as part of the peer-to-peer algorithm and the end of downloading stuff that's illegal this prevents the adoption of these systems in companies because companies just cannot afford to do that so n ipfs those are policies that are expressed and that can be added on top right so if you want to organize a cluster that moves around and shares data and so on you can do that but that's a policy that becomes that is layered above ipfs so there's a strong commitment there of being able to say look you can use ipfs and not have to worry about other people who are other people are adding to the network and yeah so I think that's this is able to follow what the user programs and so these policies that will be we haven't

done as much work here will be doing some interesting stuff there to be able to express how you might trust other keys and trust the content from other keys and being able to give people capabilities to add stuff to your note so yeah on this on this only downloads explicitly requested data so this is a huge surprise for most people because they're like wait what you have this distributed file system it's supposed to be pure appear and like why we're like if I add content how do I back it up right like that's what if I disappear from the network it pulse it falls off and the idea here is that when you add content to the network you're explicitly publishing something so the rest of the network has an option either they spend energy maintaining this online or they

don't and if they do it's because they care about this thing so if they don't care about what you have to say because they just what you say it's not that interesting then you can just pay them to care and this is where something like Falcone comes in so HTTP maps to ipfs and AWS s3 maps to Pal coined or swarm or really AWS and Windows Azure and so on right so there's that how you maintain stuff in the network it's a separate part different from how you move stuff in the network so IP office is strictly about how you address and move content in the network so and this and so these protocols around incentivized organized

distributed storage those layer on top of IP FS and so an interesting thought is a lot of people say well how could you ever hope to compete with something as big as you know Amazon and Google and Microsoft and so on which have these massive infrastructures and then I just kind of point to them point this out right so this is the hash rate of the Bitcoin network and like the massive spike and this is what happens when you get a an economic model coupled with a very easy to enter market and so somewhere along this graph way before the knee the hash rate of bitcoins surpassed the equivalent of the world's most powerful supercomputer and then the world's 500 supercomputers put together

before the knee of the curve so that gives you a sense of just how insanely powerful the the hash rate of that network is sadly all that is going to waste so hopefully we'll get proved mistake working correctly so we don't have to like waste this but when you when you think about this model and you apply it to doing something useful like storing content you can get an enormous amount of storage wired up into the network so you'll steal so mark my words in the next year and two years you will see enormous distributed systems with storage popping up into the network and they're already getting started but oh come and so you have something that looks like this which are the Bitcoin mines but for storage all right so now

that I've kind of made a claim about the future let's have a dose of like web reality today to like cleanse our minds right so so the web here and these are strictly kind of like web 1.0 web 2.0 companies or web sites in general that I've put up there these applications kind of run your life right so everything that you do nowadays revolves in some way around these these applications right so your personal communications your work how you learn how you get stuff how do you buy food everything is now mediated not only through the internet but through specifically web applications and this means that all of your capabilities are

tied to specific organizations maintaining specific infrastructures online it's got scary right because those could be this wouldn't disappear at any moment in time or those could just stop wanting to serve you or blah blah right but these I like pointing these out especially in this crowd because this is kind of the competition right so if we want to push this very different model of computation very different model of storage and actually upgrade how the web works we have to work equally well at least and ideally better than these systems in fact not only do we have to work equally well ideally better we must be way more weight we have to have much

higher performance or none of these people will adopt this stuff right and so this is what I mean by like a dose of reality and they're like when you think about the data on the web today you have like these massive infrastructure providers that are there have huge high performance databases and and sort of the model is that people have these business logic services and they have to maintain them and this is where perhaps these systems can come in and change the model where the people don't actually have to maintain infrastructure themselves it becomes a lot cheaper if they don't have to do that but you know this is the competition right like this is something it's an diagram from Amazon itself it has like think about how many machines here are involved in just like

one application right this is kind of like if you if you're a legit application this is what you do you have like a DNS resolution machine connected to like a CDN connected to like a bunch of static content buckets connected to a whole bunch of web servers that all the scale connected to like routers which hit your application servers and then going all the way to the databases like it's madness right and this is all up all for like optimizing really fast accesses on the network like when I user hits your web page they have to get the stuff like within a second or like it so game over right like they'll leave and they'll go to somewhere else and so this is like the beasts lying in the center of the backbone like to just be able to manage that but and we have this enormous

explosion of like databases right and so the constraints are around like being able to do billions of transactions per second handling billions of users millions or hundreds of millions of simultaneous users were the big ones sub 100 millisecond latency like from the backbone to the user or ideally just from the user to the application and we're talking about terabytes petabytes and exabytes scale so terabyte scales if you're a small company exabytes scales if you're you know basically Facebook and so on and of course so when I hear like blotching technology right like how is it gonna deal with this right like how are we what but here's the magic right but but here's the magic right so

the thing is that it's not about matching the capabilities today it's about proving that those capabilities can be achieved but complementing them with a whole bunch of capabilities that were in there before right so stuff like like the idea be kind of blocking is being able to do multi-party computation but it's verifiable and trustless you don't get that at all today and web 2.0 being able to do trustless trusses Ledger's like being able to account that things are actually happening time stamping being able to like run contracts that are verifiable but like you just write the source code out there put it up there and it just runs on its own and you don't have to like maintain it and nobody has to do it being able to like write decentralized and distributed application is things where you write them once you sign them

drop them into the network and disappear and the people can carry them on you could disappear as an organization in a company and people could continue to use this application it's like these are real capabilities that give people that you cannot take away and that's extremely interesting that's where like there's a completely unfair advantage compared to the web web 2.0 right and so we're talking about building taking data and code and execution in a completely different verifiable model and of course being able to secure an encryption in a way that the web 2.0 people can't because a lot of web 2.0 relies in advertising and so on which means that they can't really do end to end encryption so that's a huge advantage right because we don't have to

deal with that we can just do end to an encryption for users we don't need users data at all and this is a strong reason to be able to move move in this direction cool let me check timing because sweet so that was like the interlude so what people know is kind of about having these like databases talking to each other with data behind the scenes and what web 2.0 is about taking the the logic and the data itself and putting those into the network and allowing those to communicate and making the axle hardware secondary alright that was the intern okay okay have to rush through this stuff five minutes but we'll see alright so I feel D that this is the heart of ipfs

it's a common format for hash chain or distribute data structures based on Merkle trees and so what I showed before with these files these aren't actually files they are so when you add something to a professor it kind of looks like this you your add all this stuff well these aren't files these are actually nodes in a graph and these nodes in a graph there there is stored in that in a format that we are calling IPL D that is basically like something like JSON it's not exactly JSON because it's it's serialize to be fast but it's it's the whole point that you have a very flexible data model so you can take entire api's of today and just dump them straight into ipfs so you can take some data API that you have and just straight

up import all of that into our BFS and like merkel is the whole thing that it's not perfect and you have to like work hard to make it really a good distribute application but that's kind of like step one and so we see us as new thin way so the internet the internet of data structures and it gives you like Merkel links that are secure and immutable it gives you merkel paths meaning you can resolve through the through the graph they're sort of universal and there kind of your URLs as we said before though it's not location-based right so like it doesn't it I mentioned here URL because most people are familiar with that term but it's not actually location-based and in terms of serialization you can do anything you want you can use JSON protocol XML RLP or whatever whatever it is that you use

you can use a serialization format underneath the hood there is a canonical way of describing which serialization thing to use so that everyone has it's the same thing and so if you had something like this in case on where you were you know you outlined the followers of a user this way this is kind of like mush right look what is at Vannevar bush mean right could some user somewhere but like you don't know what context that that comes from right you have to know that this would be Twitter or something else just from the endpoint that you're accessing this from but what if these actually link to content address things right like the resources were actually based on hashes then you don't have to know anything about the way you're

accessing at all because the content itself or like tells you what it's about right and it would I feel that you can do llamó you can do whatever format you want and so on so the point here is to build something as easy to use as JSON but when you merkel eyes it and you turn these normal graphs you you should be able to have the exact same architecture of your data but where before you would just have some strings that you then would have to look up in a database here you have hash lengths and so you can follow you can resolve through the hash line so here in the second layer i have your going through the hash and then

slash followers meaning that you're you're accessing into the json object so you're like putting a JSON object and accessing through it to get like another piece of the data and you can like do whatever right so you can take like the somebody's followers those slash followers to get pull that array out and then go like slash zero to pull out the first entry and now you get something and this is resolving through merkel links the idea there is that like you would pull one object out and then you would look into it pull up pull out another link and resolve it through through the network and everything is integrity text so you can just dump all of this content into the network and anybody can serve it to you and you don't have to worry about that you have to worry about it in terms of privacy

but that's a whole different question and anyway so you can model things right you can pretty much mobile anything one you get you get to be able to annotate links with other data here's like what a directory would look like you could do like version data structures this is for example a git commit and so on and so these these will be completely native in ipfs you'll be able to write all these data structures and have them write on this this transport right like the forest thing that I described earlier and so this is really how you we start doing web 2.0 where we take all these data structures than these these KPIs and model our applications around content that we were just putting into the network and addressing through content addressing and key addressing and so in a sense when you think about model in your application and you think

about like the data model right like oh do I have users that they follow each other like what are all these relationships you can take those direct objects and put them straight into ipfs like you don't have to go to the Twitter data base at all you can just add them straight to my PFS and what you gain is that when an application pulls those in if there's nothing that has changed nothing has to be updated so you beat the speed of light because if it's if I if I'm using an application here and I request some set of objects and I pulled them down through the network and I have them locally and I have to fetch them again normally HTTP would say oh you know like go and like talk to the network and see if it has changed but would ipfs we can be like well no I have

the hash I have the content it sits here already we don't have to go to the network and of course you can do this with like all kinds of like legal records and so on like admit legal records is a great example of like why people want to use block chains kind of like in the in the real world it's it's to like do careful time stamping of legal documents right so you can do all this and so in reality we're building not exactly the web but it's really like more like a strong mesh that is secured right like you can think of it it is like a secure web or the links can't break because they're their content addressed so you don't have to maintain infrastructure and worry that like your link now is slightly different and so everything will fall apart you just

create content you get securely and you just release it into the network and so I mentioned that that I peeled evening is suspect in progress some of the interesting stuff that we're doing is we are coming with a coming up with a right way of doing traversals through the graph so being able to express descending down a path in the graph things like being able to address one level deep or like everything underneath it or being able to like look into something but then something more interesting like being able to so this last expression would be able to in in a huge graph that represents a movie being able being able to walk down the first path and retrieving the first set of blocks in so that with that you are able to just pull

out the first few things that have the data and then start showing the user like the like beginning of a movie right so being able to express these things this way it's extremely useful for replication protocols it's extremely useful to be able to like point stuff out in contracts and so on so this is kind of an interesting thing another one here is where you would like embed replicas like yeah but one expression is that another expression so if you if you like this kind of language and stuff come talk to make is getting those languages right as important cool so last section use cases and examples so this might be the most interesting to you so I BFS a lot of people are like okay well what is IP passed for what do you use it with like what is this thing and they want me to tell you like the

specific set of users that like are only gonna use it for one thing and in the thing about ipfs is that it's a Transfer Protocol meant to like help the web right and said like what is the web used for everything and then people just say like oh that's you haven't thought about this and I'm like no I'm serious like you can use this for everything let me let me list out the reasons right look you can use it to do peer-to-peer content delivery ready to think about a CDN but like a distributed CDN the superdude content distribution network you could do like a decentralized links things where where no authority determines the link you just add content on the content is now part of the network where we've talked about before you can take applications entire applications and define them with an IP

FS then give people a reference to that application that's the link they download it they run the code and they just do everything on there side so they it's not only the distribution of the application it's also the definition of the application in an example here it's like people have written video players that go over I confess people have taken entire like legal codes people have made like imager salad clones people are able to do like three models of stuff and be able to share them with ipfs because these are data intensive kinds of applications people are this is not this is the only one in this set that it's not an actual IP Office application this is a Minecraft but people are taking minecraft and making it so that there's links between worlds so you're walking through a Minecraft world and then you

find a link and you watch this on something somewhere else and all of that is served over ipfs people are throwing documents blogs and so on you can do containers on VMs definition and distribution of these you can think of these as like merkel containers where like the entire application and like the graph is defined as a proper Merkel Merkel graph we should have been the case from the beginning but anyway you can do whole package managers and distribute of them this way so we're already working on doing this to several of these package managers we're taking all of the code or all the actual compiled binaries and adding them to ipfs usually first of all you get a huge space savings because tons of stuff is duplicated but then you also get get to

distribute them faster because if machines in the same data center have it then things move faster key addressing archiving backup everything people are like archiving all that stuff all these cut a whole bunch of the data here you can archive what ipfs and there's efforts on this now when it comes to the etherium and this is like the last line I promise when it comes to theorem and by the way the logos fit really nicely together it's fine too so so you can use it for alkane data right so you can the very first thing that you can think of is like anything that just shouldn't be on the blockchain because it's too big put a put it on ipfs take a link and then serve that link with your own note but that means that anybody else can

serve it as well and it means that you can disappear and the content doesn't disappear right so you know big stuff you can use them again to define and distribute these to distribute apps right so these these centralized distribute applications you can write them put them on a fast put all the assets there people are already doing this and so on so you have the code and the assets and everything but also the users data this is an example of one of them this is ours it's our web UI for an IP FS node all of the code for this is when ipfs you pull it out you run it and it's completely distributed so like all it's a very cool experience when you do is me like whoa not touching what server

ever like it just my browser does it here swamped ran an example of like a like a B style a four times outboard done with Ikea pheasant etherium so you actually pay ether to be able to post so you could do the trunk base for just formless pointing out that and you can do the two interesting things here and sort of London stuck this way that the most interesting things where where i p FS could be useful for aetherium and i worth investigating carefully is representing the entire merkel patricia 3 state s ipfs notes so that you are able to directly index into the state tree and pull things out and and use a bunch of ipfs nodes that may not speak aetherium yet but they speak ipfs you

can then start addressing etherium state with them and use them to move around the theorem state it's just really cool and you can do that to the blockchain itself you can take the entire definition of the blockchain and just like imported into ipfs and then distribute all the content this way and so the this is a blog post from chris that talks a little bit about this very cool so check it out all right so that's Ivan cousin did it Dan instructors thank