Joel Thorstensson : Primitives for mutable content on web3

Transcript

all right cool so hi everyone i'm joe uh co-founder of bigbosslabs and today i'm gonna be talking about kind of some of the insights we've learned along the way as we've been building three blocks and now take it off upgrade and now ceramic network and really what the focus of the stock is is kind of primitives for creating mutable content on web3 and i'll go into kind of what that means so what i'm going to be talking about today is crypto data and crypto there might not be like super well defined yet but i think it's

actually a concept that's going to be really important for the web 3d movement and so there are some simple examples of what crypto data could be some kind of early example so like file an ipfs you put the data in there you get a hash and you can like when you retrieve the data based on the hash you can verify that you get the correct data the contract on ethereum is of course crypto data and you could verify the state of the contract but it kind of requires you to sync all of the history of all contracts on the theory because it's kind of very heavy to verify that data a jwg is just like a json object that contains a signature

and that's also like cryptographically verifiable so all these are kind of simple examples of um crypto data but really what with our desire i think what we also desire is uh types of data that can be objects that can be immutable have a permanent identifier and change over time and so we want these to kind of also one piece of information that i care about in my application i think if we can make these crypto data

objects real changing if we can make these crypto data objects real we can actually actually essentially make the entire internet verifiable all right so in order to kind of understand how we can get to something like this i think we need to look at the history of like peer-to-peer cryptographic data protocols so actually going to start looking at one of the early protocols bittorrent so in bittorrent you basically have this concept of a torrent file that describes some piece of data and once you have the torrent file you can connect to trackers that can where you can find the other peers in the network um

and there's a problem there was a problem though early in the bittorrent protocol that like these torrent files was like super huge so so it took like a lot of power to host a site that had all of these torrents but the maintainers of the bittorrent vertical created a new concept called the magnet link which is essentially use the hash of the torrent file and so you get this kind of cryptographic identifier which you can use in the peer-to-peer network to look up other peers that know about the torrent file get the torrent file once you have the torrent file you connect to the trackers gives you the data about which peers have information about the file you're

getting so you're essentially going from a hash to a piece of data or a file which is very similar to how ipfs works as well um but you still have a problem with with bittorrent that you don't have uh or you have a problem with bittorrent so you have this hashes that goes to files you don't have any mutability so you have the same thing in ipfs where okay i can cryptographically verify the data but i can't really change the content and that's the reason why sites like the piratebay was needed for for torrents because there was they needed like a centralized

site to keep track of the things if you could mutate torrent and change them you wouldn't need a centralized site uh like prior pain like other sites um but interestingly about bitter and this actually works really well it scales well like purely from like a technical perspective and you can actually like stream content similarly to what you could at netflix so like the technology works kind of well uh for for static files but we need mutability so there was actually an innovation that i think most of you are very familiar with that came after bittorrent and that's uh bitcoin and ethereum uh so

essentially this concept of um of a blockchain where you have blocks that links back in any kind of linked list fashion and so in in bitcoin the state that's maintained is the all of the unspent transactions so if you're building an application there's not really like super much interesting stuff you can do with that uh but ethereum introduced the the concept of state tree so we can actually keep track of like various types of states of different smart contracts and now we have this this this way of having immutable states over time but the problem still is if i want to know the state of my contract in my application i need to also verify the state of all other contracts in the

blockchain so kind of have this limit limitation so now like once we have this examples of like bittorrent and blockchain uh where do we go next so i think in order to understand that we need to look a little bit deeper at kind of the data structures of of blockchains and and generally other protocols as well and that's mercalyzedata so the canonical kind of example you'll find of this is a merkle tree right and you have blocks of data you take hashes of them recursively until you get to a root and as as that you can represent like a transaction um a bunch of transactions or represent the file system

or or stuff like that um but what if you want to mutate the data then you need to essentially like update this this merklized dag and so the example to the left here is essentially what happens in git where you have a three of the the file system and then you make a patch so like you go from xs to x to ys and then you add this this node e and then you have to basically mutate part of the tree but you can still point back to the old part of the tree so you can kind of get this nice way of muting mutable immutability without kind of losing track of history and ipld is is essentially the data layer of ipfs and so ipld allows you to represent

any type of merkle data structure so you can represent of course like ipfs file system but also git and you can even represent bitcoin blockchain and the ethereum blockchain inside of ipld uh all right so once we have this primitive of like mercurialized data we still have the problem of like okay how do i keep track of like what's the latest update what's the latest tip of this data structure and of course there are a bunch of different approaches to this problem um and i'll kind of categorize them in two different parts one and this is kind of like a generalization i won't go into like details on these things but uh i think these two categories are kind of nice to

describe the two uh two approaches so one is the public key based one so you basically have a mutable pointer that is a public key and then the the where the pointer points to you use the private key and basically sign a message that contains uh the latest hash of the root of the thing you're pointing to and then you kind of also include a nonce so you can see like what's the most recent update um and so ipns works this way you can point to like any type of data inside of ipfs similarly that's and or slash hypercore and secure scuttlebutt also works on this kind of one public key approach there's also a more kind of

sophisticated approach which is using some kind of state machine to govern the state transitions that your mercalized data structures go through and so some examples of this are textile query efficient or vtb and essentially you have this this append only data structure that's signed by some some key and over time so some some type of transitions are valid some are not and over in and from that you kind of reduce the state but in both of these approaches there's a problem and it turns out this problem is really important to to if you're building like an identity system and it's also important to like if you want to care about optics so if

you want to like if you want to keep good um do good key management you periodically want to rotate keys or if you want to keep a list of people that are in in like a message group for example you want to be able to remove someone and you don't want them to be able to like sheet and come back into the group again and so why can't these state-based based uh systems do that well so if we look at uh so let me describe kind of the problem uh so here we have this uh mercallis data structure so you see the the at the bottom we kind of have the genesis signed by the first public key and then at in the middle we rotate from public e1 to public key two

and it's also signed by the first public key and now only like public e2 can make updates so this seems kind of well and good well well no because someone can if someone steals my first public key they can basically create an alternative history that includes and points to like a previous record before i um i signed the rotation and me as and someone as like an outside observer that looks at this data structure has no way of knowing like which of these two branches of history is the canonical history um and and all of the the the systems i described above have this

problem like there's no way to know which key and which branch of history is the canonical one so how do we solve this well we essentially use a property of the blockchain called the proof of publication so once i rotate the keys i add take the hash of the update put that on chain and now i can now i essentially have a proof that this update happened at some specific point in time and if someone gets a hold of my first they will be able to create an alternative branch but they won't be able to go back in time and and put something on the blockchain earlier

so now we can actually know as an outside observer which branch of history is the canonical branch of history and there's actually like another interesting property of using doing key revocation in this way and it's actually important to identity and reputation protocols and that's that you can't actually sell identities so if you're using like a proxy contract on ethereum uh or you're using various types of like reputation on chain you'll always be able to like kind of trade the identity like give someone else right and access to but here in this example alice is going to try to sell her

identity to bob and we'll see what happens so bob is like hey i gave you 10 ether for all the reputation you built up so alice makes some updates with public e1 that's her reputation and then she rotates to public e2 which bob owns and she gets money from bob and now once she got the eth from bob she she's kind of sneaky so she publishes she before she made the rotation she actually created an up an alternative branch in uh in before that she didn't tell anyone about and once she got the money from bob she publishes that and gets her identity back so uh this makes it essentially like impossible to trade identities and and

the nice thing about this is like even if alice was like okay i'm gonna be honest and actually sell my identity bob wouldn't wouldn't trust her because he can't know that she has she hasn't like hidden this information and i think this is really important for like a bunch of various type of use cases in identity where we actually don't want people to use like betray building up reputation and selling that to selling it off to other people all right so all of this stuff that we learned we at threebox labs basically incorporate incorporated into a protocol called ceramic and so ceramic basically takes this this proof of publication system and this this uh merklized data structure and

puts it into something we call ceramic streams and a stream is essentially just the crypto data object that's mutable and it's scalable because you don't need to verify the history of all other data objects so you get like this audible commit log of commits or either signed or anchored into blockchain and based when you sync a stream you basically verify all of the events that happen and you can process the latest states and the access control is of course like cryptographically controlled and we're using dids which is short for decentralized identifiers and this is like a w3c standard there's actually getting some adoption in various kind of crypto ecosystems so the id the ad center is

nice because we can have identifiers that are described on ethereum or polkadot or like other any type of like blockchain and they can still be kind of interoperable so with the streams we get mutable identifiers stream ids that you can use and look up the latest state of an object and you can do this essentially because you only need to verify the history of one particular stream and not like the entire network of data and so since we saw before that there was like a bunch of protocols that took different approaches to mutable data

we figured that we needed like an abstraction layer to deal with like different approaches to like how you build these crypto data objects and that's what we call stream types so basically allows you to put custom state transition logic and also like custom conflict resolution logic in case there's like a conflict in the in the log so for example you might want to have a log that is allowed to branch and merge under some constraints um and just before i kind of jump into what we can build with this type of primitive i want to talk about just some future improvements for the ceramic protocol so right now um you need to run a ceramic node in order to trustlessly verify these

these crypto data objects are correct obviously that might be easy for some people but for most users it's kind of very difficult to run an always online node so we want to add a cryptoeconomic incentive protocol that allows you to basically pay a set of validators to keep track of the state for you of your streams the other very important thing is privacy and so by default uh streams and ceramic are public but it's easy to add the confidentiality basically just encrypting the updates that you add to to the data structure um but the the information that would still be public if you have confidentiality is the kind of link linking of the list and the

the signatures by the user and in order to make this private there's some approaches that's been taken already by textile for example they encrypt a layer on top of the linked data but this essentially requires you to trust some third party that is allowed to decrypt that metadata information and keep track of the state so we're cur we're not super happy with that because it's not you know fully trustless so we're looking into like various ways of like maybe you can have a group of streams and then create a zero knowledge proof that this update was part of this group of streams um but without revealing exactly which three stream is being updated but that's still something we're kind of researching and looking into all right so i think what can we enable once we have this kind of

tiny primitive for like crypto data objects what can we achieve and what can we build with this uh and i kind of like to phrase this is the the sovereign web basically a piece of the internet where we have as users full control over the data that we create and the data can like flow between different applications and between like it's not locked into like any jurisdiction or application and we can also see how we relate to other people and how we've interacted with other people so there's basically three aspects that i want to highlight identity is built in so in the internet today you can have any data that you have

you kind of have you have some data source that gives it and like facebook and like you can't really trust that the data that comes from the server is actually the post that your friend made is actually like a post that you made it's just like you trust facebook that that's the case but in in in the case of like building on like this crypto data primitives um we can verifiably know because the identity is built into the system um and crypto data is borderless and this essentially means not only like nation state borders but borders between like boundaries between organizations and companies so we can actually start having this concept of open source data that's owned and

controlled by whoever created it and and kind of replicated by anyone that cares about it and the last thing uh is we can create a linked data graph and so this is kind of like a concept that has been explored in the past um there's been like various groups i think one of the most prominent ones is the semantic web group in w3c and they basically had this vision of like okay right now we have the internet it's like http pages that links to each other uh but what if we can like not only use the http or sorry the uh html pages that link to each other whatever the html pages are kind of the view layer of the internet and what if we can actually link the

data layer instead so like we have hyperlinks between the data objects but i think one of the problems they ran into with that without project is that hyperlinks using http points to servers that can go offline they can disappear they can get censored and so you can't really build solid structures on top of that and like rely on things working well and so the only places where this kind of worked is like larger organizations like facebook and google because they can they can essentially like trust themselves to keep the data around um but the nice thing is if if you create these crypto data objects in ceramic and link use stream ids as links you can i can link to whatever content you like

i can you know link to a thing you created but then back that up myself so the data is not pointing to any location it's pointing to actual data and the mutations on that data so we can create like a really solid structure for this kind of linked data graphs and while this is possible to do in ipfs in ipfs if i link to something i link to like a static thing and there's no way like if i want to link to your thing like i want to be able to like link to the latest version of whatever you did not use like a past version and then have no way of knowing about what the new thing is um all right so i'll jump into kind of some some specific things that we're excited about people building so we actually created a framework on

top of ceramic for kind of identity data and this kind of builds on insights that we had from working with the uport team and then working with the early kind of three box prototype so idx is a data structure or framework for storing data with the user and essentially in this example here users have their own index of their own data and then a developer can come in and create definitions and here in this example the developer creates a basic profile and it has a schema that describes it's a json schema that describes what the structure of the data needs to conform to and so when user one and user two

comes to the application at various times they will have the same definition id so there's like a semantic mapping from what what the data is but they have their own record that describes um what the content is um and the nice thing about this is like if developer 2 comes along and says like hey this this basic profile seems really useful they can use use that definition id and pull that data into their application um so you get this kind of composability of user-centric data another interesting thing that you can build on on ceramic is is kind of managing data flows using nfds so the very simplest and cutest example of what you can do is like

the creator of an nfd can have a stream and so they can mutate and like update the artwork over time but i think more interestingly uh i mentioned before that the cryptographic access control for ceramic streams is based on the ids and so we actually thought about this a lot and created a an nft did method so like an identity that's based on an nft and that basically works by the current owner of the nft is the one that's contr allowed to sign and decrypt things on behalf of the that identity so you can imagine something like crypto punks you have a forum where you can sign in with your crypto punk and basically author things on behalf of your punk

and even like send encrypted messages to other crypto punks uh so i think that type of use case is really cool but i think more kind of long term and like interesting things could be like say this nft represents a physical art piece um or like a some kind of physical resource i can create a ceramic stream that like uh you put information about the current state of this piece of physical art and like okay this is uh you know been reworked or something uh so you can keep like not only the history of who owned it but the history what's happened to the thing over time and the last thing uh i'm kind of really

excited about this decentralized coordination so um we all kind of know about daos and like the the way we kind of use smart contracts to do treasury management uh but dao's kind of have ended up doing okay on chain for the most you know important treasury management stuff and keeping track of all of their nfts and tokens but do every other coordination aspect of chain and that often is stored on some centralized server um where the owner of that server could potentially like rewrite the history of like who said what and like really screw with with the dao if the if something goes wrong but there are some tools that build

really interesting things like snapshot allows you to vote off chain uh coordinate but allows you to like assign as kind of value to and and like signal about who did what in in uh work period uh source credit allows you to track kind of contributions on github and forums into like a token distribution mechanism uh but all these kind of have this problem of um of things being manipulatable when they're off chain and so you can actually take this type of system and remodel them in on top of this kind crypto data world where we have verifiability and i can verify the state

of of things um without trusting like a third party to not screw with the governance process and so boardroom is actually an example that started doing this on top of ceramic already they have this feature for comments and ideations where you can ideate on some proposal and like the things you post in there will actually be auditable and assignable to your your ethereum address and you can view it over time another nice thing about doing this is like you might have a bunch of assets that for for a governor's protocol that you keep in a hardware wallet but you might want to like interact with

like a more kind of hot wallet uh when you're just writing comments and whatnot so the what i talked about before idx allows you to link multiple ethereum addresses to one the id one identity so you can control uh not only the data of like one address but the data of like multiple addresses with one identity and then you can kind have better optic for how you manage your keys for your you know valuable governance tokens all right so i hope you all kind of a bunch about crypto data and and how you can use um use it and excited about to see what kind of stuff can be built on top of

ceramic uh if you want to learn more um you can go to ceramic.network or more easily go to threeboxlabs.com we have links to like the ceramic documentation and the idx documentation all right thanks everyone you think we might have some time for questions so right now a stream in ceramic is um only can have one controller and and

that's the id so so you actually just granting permission or that grants permission for the d80 to make updates and change the history uh we have like kind of lower level tools for encrypt encrypting data to the ids um and so you can use them to put like confidential updates into into the stream uh but we want to add like a a layer to make that much easier to deal with this kind of more privacy preserving encryption stuff um so right now the the data structure that's

implemented in ceramic is just like an append only log and so it's just like a bunch of these commits that are assigned and each commit contains jsonpatch and then we reduce that to a json object in the future we want to use something more sophisticated like [Music] has array map trees that allows you to have like very large scalable kind of key value stores yeah um so it kind of depends so right now in in ceramic like the longer the stream is the longer it takes of course to verify everything um but so if you access it through the peer-to-peer network

it might take some time similar to how it takes time to like verify the entire blockchain of course it will be much faster since you're only dealing with one piece of data rather than like all data um but if you have a node that pins the then accessing the data from that node is of course like very fast yeah so we're actually in the process of migrating profiles from three box over to ceramic and idx and so creating a new definition in idx and adding the new properties that you

want to have is the best way to to extend the profile and then you also have this kind of semantic descriptor object of like what the what your properties that you add what they are any last questions all right cool thanks all yeah so in order to do that in an easy

way you need to kind of create a specific dad method for that wallet we're actually working with no safe to do this for corsair and we have a