EB126 – Trent Mcconaghy: Scalable Public Distributed Databases With BigchainDB

Transcript

this episode of episode of bitcoin is brought you by ledger now accepting pre-orders for the all-new ledger blue developer edition a Bluetooth and NFC touchscreen hardware signing device learn more about the ledger blue and ledger wall calm and use the discount code epicenter to get ten percent off your first order hi welcome to episode of Bitcoin the show which talks about the technologies projects and people driving decentralization and the global cryptocurrency revolution my name is Sarah thank you Joe and I'm a hetero today we are going to be talking to a trip McConaughey who is the city of escribe and big chain DB in DB is a is aiming to be a new a new database technology that can enable eventually a

planetary scale database but before we get into big chain DB let's have an introduction from Trent Trent your background in trophies sure hi it's a pleasure to be on here again guys I really appreciate whenever you have me on so I will just make this one quick because I know you're going to dive into some the details but I started out spent almost 20 years in the semiconductor industry doing AI for designing computer chips machine creativity that sort of thing a lot of big data a lot of distributed computing a lot of other sort of pedal to the metal sort of coding and whatnot about almost three years ago I started working on a project which was I pee on the block chain with

a specific focus on digital art that became ascribe the company and the product and continuing to focus on that as well as more recently due to limitations we saw with watching technology we have built another product called big chain DB and released out into the market in February so we'll focusing on that I guess in this call and I look forward to talking with thanks for having and so maybe we can get started before we get into big chain TV and that topic quite extensively talk about ascribed and since last time we had you on it was about ten months ago episode 76 for anybody interested can go back and was that episode with Trent so tell us how has subscribed move forward since

then how is the product develop and you know what's what's going on with the scribe sure so overall ascribe the company which has big chain DB as a product as well as described the product and a third product called where in the net I'll talk about that in a sec so three products one company ascribe itself as it's actually being quite true to the original vision which is to make it easy for creators artists etc to securely attribute their work on to the blockchain time stamping and etc and only securely attributed it but also to make it really easy to license it for others such that for example if you're a

collector of digital art or even physical art actually if you're a collector you can truly own that art via a transfer of ownership from the artist to the collector or perhaps via gallery in a consignment model and then from one over to the next to the next so worried whereas before you had no provenance now you have perfect digital providence so that vision is is as it always has been and it's been growing quite quite nicely actually so now we're at about 5,000 artists using the system there are about 9,000 or 10,000 pieces of work 40,000 editions we're seeing heavy usage from many sectors one of the ones we're most proud of is a lot more people from the Creative Commons community using us we work closely with Creative Commons

France on that and other organizations related and actually we work to probably this about 20-25 organizations that have been using us in other interesting ways so of that several art prices probably now a total of no 10 our prices and also sort of the clowns so galleries libraries archives museums using us in various ways for things like archival things like for selling us selling artwork etc and then on top of us we have a rest api so this is mature not since fabric march april of last year and its really rock solid and because of that we've actually had several people sorting companies on top

of escribe got about seven or eight startups now that are working on top of a scribe the most recent ones are left gallery which is an initiative from a world class matter named armband indoor pool so it's left if you go to that and a more recent one yet is 23 VP VI VI and this is the start up out of the USA and they are sourcing a photography from social media and other places working the people creating that and then selling it and even of a secondary market now limited editions all this sort of thing don't have anyone subscribe which were quite excited about and I guess the other big you know that should have a big be actually quite

directly is with a scribe we always saw that people were concerned about sharing their work on mine they would say okay I get that I claim that I own this work that I have the license to this with the legal so you know we have a full-time lawyer operate lower that's worked out all the legals in all the countries kind of make a general but so they dropped an attribution but they were concerned to share it because they have this feeling that as soon as they put it online that Bailey's control over that they don't have any visibility into what's going on so they would say hey can you encrypt it for us can you water market for us and we're like know that totally goes against the point of this right that's you know it's sort of classical drm and

it's really not the way to go and we said okay instead let's shine light on to where your work is showing up so what we did was we crawl the web works out to 220 terabytes worth of text if you limited text and from that we indexed a bunch of images 15 billion images so you look at the links inside though that knee down we downloaded those and computed feature vectors from them and basically came up with it's basically like reverse image search except it shows you copies versus time so you can see exactly where you just show up on the web and that gives you visibility into what's going on so you regain what would have been a loss of control via

disability so and this will lead directly so we index these 15 billion images we wanted to record under the Bitcoin blockchain we were building on that we wanted to record on to it that we had sighted fleet but we did some quick back of the envelope calculations you know the Bitcoin network it starts to back up to about 1.5 transactions per second you know pastor with that I'll take more than 24 hours for transaction to go through so run the numbers from the numbers and it will take about a century for 15 billion images to get reported under the Bitcoin block chain so that was clearly a challenge and it wasn't just the right you know we've been iterating with galleries and museums etc where they have a million images here 20 images they're extra they have all these digitalization efforts

and any one of those who basically go stop her for really putting the stuff onto the Bitcoin blockchain so it was really starting to hit us really hard about the limitations of the scalability of the Bitcoin blockchain right and people talked about blockchain bloat at 50 gigabytes I mean my thumb drive holds more than that so that clearly something was up and that's kind of what led us to thinking about how can we approach this problem you've indexed 15 billion images but what what kind of server technologies are using to do that oh we were running on AWS actually 80s when we started doing this we hadn't really iterate over them a lot so they called us up word of the blue and said like who

are you guys what's going on because our bill was thirty thousand dollars a month actually so now we cook we quickly became their friends and as I'm like an Amazon rep just working right in your office or something for a while yeah I mean they actually flew down a team from elsewhere in Europe just to come and visit us because I mean you know what crazy that'll start up in the middle of nowhere and actually it's not the middle of nowhere Berlin's an amazing city but what crazy startup you know has the audacity to go and crawl the web right so we did it's actually less about algorithms you know you have to have good algorithms but it's more about commitment of time and resources so we committed it because we saw it as another piece in the puzzle of serving

the career surf the internet and so from that problem with being able to index and timestamp all of this all of this data this this image data you you came up with the idea for be chained DB can you talk about how that idea how the idea came about sure so so there was the the 15 billion images challenge and there's other ones too you know one customer we talked to a potential customer they have 20 million users they're going through more than 100,000 images a day right so that would in a roughly double the throughput right there so and actually that particular customer we were talking to in 2014 in sort of fall of 2014 you know we had really started full-time on

our on ascribe in the summer of 2013 sorry summer 2014 and so even then we're like hmm you know this is gonna be a challenge and I even gave a talk actually Brian runs its meet up in Berlin and I've been you know talking with him over a drink one night about this this problem of bitcoins like count you know you should really talk about this I'm like okay I'll give a talking to talk meetup sure so I did this is in full of 2014 and I said I said hey look you know like there's the Bitcoin blockchain here's the scalability issues right 1.5 transactions per second you know 50 gigabytes people call me bloated and then you look around and you look at the internet and you see hmm that looks is you know using a thirty-seven percent

of the bandwidth of the internet that's interesting and there's this thing called big data well what's that about right how big is that data and you know I knew about this world already quite well because I come from a world of big compute right running a thousand ten thousand machines to do verification of memory chips etc so um so if you're looking to this this world of big data it's you know that's the buzz word but really it comes down to distributed databases and distributed in the sense of compute resources are spread over more than one physical machine right whether it's processing resources or storage resources or whatever so there's been distributed database technology out for four decades actually and it's got

really good and this is what power is Netflix this is what power is google this is what powers all the big guys would guess what all the little guys to write if you're a startup that you know quickly rounds to 10 million even 50 million users you're probably running on abs or maybe 10 or something and you're just you know paying more money for more compute resources and it's not just your server in your back office it's actually just more machines being added two more physical machines being added to serve your customers so there's a stablish technology for this and in the database world it's you know for the storage side it's distributed databases and if you dive deeper yeah you say well you know you think about okay if I put data onto this database does it get sorted on a

new brew machine well that wouldn't make it scale so it only have two it only get stored in a fraction right so it's the idea that you're shorting up your data you know let's say you have 100 records the store and you have a hundred machines and you can serve one record in each machine but what if one machine goes down so you have to make sure that you maybe have three or five copies that's the idea of replication but you don't have you know one clock the same copy in every machine full application because that would kill the scalability so by doing this if you have three or five copies instead of n copies as you add more and more machines then the capacity goes up and if you do things just right the throughput goes up to so even looking at the benchmarks you know

in full of 2014 we saw that Netflix int'l an experiment where they 15 machines and they were getting to 200,000 transactions per second and as added more machines added more machines here we go the throughput went up up up up up such that by the time they hit up with 300 machines they were doing more than 1 million transactions per second so just just when you say transactions is because we're using the word transactions in the block chain space which actually meeting is writes per second in the block chain in the database I apologize i was being slightly loose there and you're exactly right its rights for second day in in that sense interestingly if you look into the database space they also use

the word transaction at a lower level and a transaction can include you know two rights and a read or one read or whatever but what I mean here is one database style transaction which include one read inside so so it's got kind of funny there's a sort of transaction in the blockchain sense transaction in the database sense and there's also transaction in the financial sense so you can really get mixed up if you're not careful so overall yeah i mean what had led us to this was we saw hmm there's all this technology that is enabling big data distributed databases you know this idea that as you add more resources hard drives etc then you can capacity on storage as well as increase

throughput and that's pretty profound right and this is actually what powers all these big data challenges you know this is why you know a smallish network is like these are a MasterCard you know we're talking two thousand five thousand transactions per second in the sort of visa sense in this way or Twitter you know 5,000 per second a bad ad network is 100,000 per second a good one is 500,000 per second right so you know and compare that to Bitcoin which is you know practically speaking 1.5 transactions per second so we're talking you know four or five six orbits magnitude difference and if you want to have a network on a database network you

want to support more than just one application right you don't want to be running well you might want to be running say just these are just Swift net or something but if you're doing something on a global scale you actually want to be handle handling a whole bunch of different use cases so so this is you know what we were looking at but we also saw that even for our own needs and ascribe you know we have these 15 billion images that we wanted to index so and we wanted to put them on where wouldn't cost us you know if we said hey we're gonna pay the 10 cents per transaction or whatever it costs these days to store 1.5 billion dollars can't really go to a VC for that one can you let's take a short break so I can take you to Paris I walked into Lebanon to

Bitcoin the house of Bitcoin in the heart of Silicon sonsie a home to many startups including ledger and I spoke with le québec ledger CEO about the old new unplugged NFC hardware wallet the ledger uncle is nfc-based hardware wallet that you can use with compatible Android phones the private keys are stored in the secure element and you can use them with wallets such as mycelium and grid bits each time you want to make a transaction the signature will be done by the unplugged and this way your private key is critical data will never be exposed to the android phone this is a secure way to use your bitcoins on the go in

mobility and you will also be able to pay directly with the unblurred with compatible point-of-sale terminals the larger unplugged is the simple solution for secure contactless bitcoin payments you can get the unplugged at ledger wallet calm and when you use the code epicenter a check out you'll get ten percent off your order by the way that code works on their entire range of products so we'd like to thank ledger for their support of epicenter big point so in I want a lot of companies of resorted to doing and certainly what we're doing in stratham is we're using Merkel trees to put large amounts of of hash data into one single Bitcoin transaction why did you feel that that wasn't a good solution yeah so we were

looking around right and we ask yourself the question there's two ways to get to scale one of them is to start with blockchain technology and try to scale it up using various tricks and ideas maybe from distributed databases maybe with Merkel hashing etc the other way is to start with distributed database technology that already scales and try to block unify it and of course we had to figure out what the word block chain if I meant in a practical sense i can get at that and but overall is really these two options right blockchain if I big data or big data Phi blockchain right now if you look at the the history of work that's been done in in block

chain world right some of it goes back to time something literature of the early 90s but it really took off of course and then the cypherpunks et cetera but it really took off with Bitcoin release but if you look at the world of databases just read databases and so on this goes back in the 50s and 60s there's a much longer history a much longer lineage and a much larger amount of R&D that has been done and technology that has been developed and it's not just about scale it's not just about performance in terms of throughput capacity latency it's also querying it's also permission and all of these other things right and if you want to do query over a distributed database that is actually a huge amount of engineering work that you have to do you

have to optimize to minimize latency etc and the idea of a query itself right it goes back to a ton of amazing research in the late 60s early 70s that led directly to relational databases and what we now think of sequel and we think of it as the most mundane thing in the world but it's incredibly powerful one line is sequel replaces 500 lines up custom code for your one application right so if you're thinking about how do you manipulate data do you prefer to write you know one or a few lines of sequel or you're going to write you know 50 or 500 lines of custom code so so this the basically the choice though right you can take a block tunic by to scale it up and that somehow a bolt on you know all this other or two with clearing information or you can start

with a distributed database and all that you know in 50 years the thousands of manuals for research in distributed database and then bolt on the benefits and you get from block chain ideas so we went for the ladder and we as far as i know are really the only ones have really went that way but it's worked out marvelously happy to share details cool lifted it does seem like a like a really cool idea but for starters let's just walk through all the participants in a let's say a in DB implementation it so when we spinner for listeners and us when you think of bitcoin me tend to think like okay different participants there's the people that are submitting transactions the trans actors they are

the notes that verify transactions proof-of-work miners that create the blocks these are all the participants so with big chain TV you are eventually going to release let's say a public big chain right that is like Bitcoin like a public database anyone can write to who would be participants in this be what are the different categories yeah sure so this is probably drilling a little too deep earlier in the call but I'm happy to talk about it so first of all bitching DB can be deployed in private scenarios and public scenarios you know people can take it right now and put it out as a public network for themselves right now right we are working on rolling our own deployment

for a public one working with a whole bunch of participates on that and when people go to use it anyone will be able to read anything anyone will be able to issue an asset anyone will be able to transfer NASA and there can be other permissions to if you like but for the public one it's basically going to be very very very open the one thing that isn't as open is simply validating but right now if you know if you want to be a validator on the Bitcoin network you have to spend 50 million dollars to the complete hardware have any chance so and you know that's that's basically for the public big chain DP deployment that we're working on one thing I'd like to talk about briefly before you know we

kind of drill down to this too much is just to go back to Sebastian's question when it comes to you know when you want to scale up up kind of clear everyone kind of understands sort of put as a noun does but when you weigh up one you want to take a distributed database and watch unify you know what is blocking meanness of adjectives right what does it mean to block chain apply something to make a block chain database so we actually looked around and you know we'd spent you know two years already in the space right since summer of 2013 working on described as the project and the company and we had a very good idea of the different benefits that emerge and

sort of the technical ideas that led to them so we like to find blockchain it's an adjective meaning three specific things decentralized immutable and assets decentralized as in no single entity owns or controls immutable as an assistant as in more temporary system than a traditional logging databases because database is already have logged and temper sorry and assets as in assets can actually live on this dick base of which you need a prerequisite of the first two things decentralized and immutable and live as and you can issue assets you can transfer assets and they can just kind of be there you don't need to have some

other sort of means for them to exist you can treat that as the main means and of course you can have regarding contracts and all these other sorts of things around it too but overall it's just the idea that asketh this can be I'm issued and transferred within this database network so that's really because you know with that function then we asked ok we want to decentralize this how do we do that ok we want a readability how do I do that ok we want assets how do we and that's what led us down the path you know how do we do that on on top of an existing distributed database so what are the kinds of characteristics that you're approaching

able to i give terms of performance transaction you can see from right Lucy etc yeah yeah so overall um it's the traditional performance characteristics that you might see in a database that people measure but also what is that our customers care about and whatnot right and you know escribe has been our lead customer so ascribe you know one of the things was capacity right we've got 15 billion records that we want to write but not just that you know another 20 million here another 20 million there etc so you have to have the capacity to store these records as in the metadata right um the media blogs themselves states aren't designed to store that those directly write this is where file

systems come in well whether it's s3 whether it's IP fast running on s3 or I BFS running in some garage someone's garage right so media blobs are well-suited to things like s3 and I PFS it so it's really about you know capacity is one another one is throughput it's really important because you know if you can only get you know a few transactions per second going through you'll never have something you know even cities feel about planetary scale right and there's lots of examples right I can throw throw a rock and find a banking app that will need a thousand transactions per second let alone 10,000 100,000 anything IOT level right you know even a small IOT blip deployment will have a thousand or ten thousand nodes at which you can

easily hit a thousand transactions per second right or energy metering right if you've got you know 50 million meters numbers I here are on the order of 50 notes and transactions per second recording is going on in those meters right so capacity the throughput another one that's very important depending the application is latency so it doesn't matter so much and things like are right where you know an artist makes a piece every now and you know maybe one piece of day or one a week or maybe one a month and you know that piece maybe gets resold maybe you know five times in 100 years right but it doesn't matter for other applications things like financial right so if you're doing FX trading your dead in the water if you're a hot running a hundred milliseconds right so but you can get away with 30

milliseconds if you're doing high-speed trading you're actually did on the water if you're doing one millisecond right so there you need super dedicated hardware you need to be running on in-memory databases you need other dedicated hardware you need to be right next to the trading floor etc etc and there's lots of sort of deep deep deep engineering optimization that you need for that and you know what's cool is the team inside escribe it's a bunch of engineers who have come from pedal to the metal engineering right semiconductor design aerospace design from okla engineering etc where we're used to doing things were that if they're running out of milling transactions per second etc so this does not scare us this is to us it's exciting it's a challenge so that's that's three things then a sort of key performance

characteristics that we like to think about capacity throughput and latency this others but those are you know really important and then the size that I guess sort of for database sorry you'll see that we list this elsewhere to query so the ability to query in the first place and then efficient query so you know how efficiently does it look up and querying also includes on how do you do a right right so Aquarius and just a border read it's also put it right and then finally permission right so does its poor commissioning how does that work so those are sort of characteristics that matter the last ones measurable bench workable the first concern right yep so so yeah let's let's let's review five things so one is a raw capacity like how much how much

aggregate data can get a particular database technology store I've been bit con that's 50 gb yep the the second is a throughput which means like which in Bitcoin we can think of as transactions a second so you know three transactions a second or whatever the third thing is latency how much time does it take since the client submits a transaction for it to be irrevocably confirm right so in Bitcoin that might be an hour and then it's squaring ability Mitch Bitcoin doesn't have right yep is that right yeah okay and then finally oh it's what was the fifth thing Oh wearing ability and and / machining right like a being

the ability to restrict certain participants to predefined rules so in terms of benchmarks have you benchmark the system and therefore they for the first three for the first three things or storage capacity transaction latency and transaction throughput what are the kinds of results you've got four big chain DB yeah for sure so I'll just go in the order that it's gone before so for capacity um uh and basically before maybe before i get there right so a lot of those people in the audience i'm sure are very familiar with bitcoin um one of the things that you know bitcoin has its fully replicated right which means that every single note stores all the data but if you want to have any sort of scale at all and any size at all that

means that every single know it has to be storing a huge amount of data right you know and it's sort of like it encourages centralization because who's going to have the capacity to store you know 50 exabytes of data even put a petabyte of data right so it's kind of interesting and even the bandwidth sport that so having a smaller number of replicates actually makes a big difference so capacity if you say I'm going to have a replication fact one or three or five instead of em what that means is as you increase your total number of nodes then you are increasing your capacity right in a linear fashion

right replication factor of 1 means you know to exit nodes 2x the capacity replication fact factor of 3 it's still linear relation right so linear scaling and capacity we set things up where if you're running on an Amazon Web Services XL instance that's 48 terabytes and working with a database we technology we work with each you have 32 shards so 48 terabytes 10 x 32 shards gives you more than a petabyte so we talk about that on the paper we claim petabyte capacity that's on that example there there's other ways you can roll things where you have even a higher storage per node but that's one example on the next one

throughput so this one was very important to us to do a good job one because you know as reaming towards this database that can help to power the planet how do you get to a scale that helps right and by the way to global email we ran some numbers and by our estimations email is running at 3.2 million transactions per second so there's 2.2 million emails go floating around the world on average every second so of course through spikes and stuff but that's what the average is for one day so throughput it kind of interesting for orders of magnitude right for to get a feel so for throughput that was really our aim was to actually get to 1 million writes per second and we actually actually had a lot of iterations when we

started working the big chain DB project in the late summer of last year of 2015 we said ok whatever we do we have to you know make this thing go fast so we designed it in a way where one of the constraints we realized was as you add nodes you have to let the throughput go up right in the way that traditional distributed databases do so that was a very explicit design decision that we made and as we're going along and we are basically experimenting with different approaches you know designing different algorithms trying them seeing what they got in the way and we managed to sort of structure things such that at the end of the day our algorithms that

added de spittle ization immutability and assets didn't get in the way of the raw performance of the database itself that's really really key you know the database itself what it does this is I can get into this later but what it does on its own is occurring like that's the core thing that a database does is ordering you know you've got this replicated you know the theoretical approaches you've got this replicated state machine or state machine and it's creating each one of the server nodes is creating a log and you want all the logs to kind of nsync right and so data distributed databases do this right this is actually the essence of them and then you have this all this infrastructure on top so what you have to do is you have to make sure that you have to let the

database the distributed database keep doing a thing creating that log reading that log each beam each server having maintaining its own state and then all these things that we do on top just get our way well you know achieving our goals of decentralization of manipulating assets so for that as we went along we managed to get everything out of the way and then we said okay great now we can benchmark this database itself because all the other algorithms are out of the way and we benchmark benchmark then at first the numbers were so so good at amazing when we actually discovered several bugs in this distributed database that we made it we just decided to work on and so we iterated very closely with them and with

the developers of the database and they fix the bugs they had very good turn around they're really excellent so so we actually help them to achieve their database to get the scale they needed and this is rethink DB they're really great company it's adjacent style document store we built on so we basically inherit all the benefits everything to be including performance it's designed as a real-time database Jason store which means it speaks the language of the browser you know JavaScript and and Jason and has you know excellent scalability so in the end we managed to have this plot that we're very proud of which is showing they increase going up to 1 million writes per second as we increase up to 32 notes so that's throughput latency

this really the big big bottleneck complete see is that slows you down it's kind of funny it's the speed of light right so the speed of light is slow it really is you know if you're you know trying to do even if you have a database technology that is perfect that takes zero time to do anything you want that infinitely fast you'll never able to do be able to do FX trading when setting because you know 100 milliseconds is the limit for FX trading yeah it takes you about I believe it's 150 milliseconds to go halfway around the planet so you know a round trip 30 milliseconds I forget the exact number let's do slow so it

depends on the application so latency really depends if you're in a single data center then or within a single region then you don't have to worry about you know the travel that way and it starts to matter but you can actually get things down to UM the speed needs of things like FX trading right with right optimizations when it comes to the other things where you're like for example the public became vp where it is a wine setting you're going to have delays but that's ok right um the consensus algorithm it is designed to handle that that's actually what I can sense the algorithm does it's about keeping all of the different nodes in sync and knowing what isn't in sync and knowing what is right so um and you know for everything

to be in sync or more satisfied of what it might be half a second um that's okay right because typically the way the full one settings that you need this is things for like arch diamonds and IP that's fine right if you really need speed and when then you have some higher cool structure and I see this right you know people might do FX trading in New York and they might do it in London and then they have some sort of way to reconcile that but sort of at a four level so that's latency we can they need to do very extensive benchmarking we've got engineers dedicated on that full-time benchmarking these things in learn more detailed ways and also understanding the effect of different faults etc things like you know as you

have a heavier heavier DDoS attack how does that impact throughput right so this matters and then finally towards your question of scalability I guess I impose silly answer than the other ones right so as you increase the number of nodes capacity goes up throughput goes up latency goes down simply because you need the majority of nodes to agree on this right so if they're spread out then you know on average you've just got statistics working against you but it's not a horrible slow down and typically you know it's not like we're going for a thousand server nodes you're typically talking you know on the order of 10 20 30 server nodes right so and that's a very different thing you can have thousands or tens of thousands hundreds

of thousands of clients that's fine but for the server notes themselves is actually relatively small number so that's why you know latency doesn't simply kill you that way today's magic where it is big b IG head over let's talk bitcoin calm to sign in enter the magic word and claim your part of the list or award so if we're talking about the the public a big chain DB that will be will be released so i think we can all agree that the latency will not be the same as if you're operating in a

private setting with with servers that are close by me that's just logic is there at some point where throughput and latency and scalability sort of cap out when you reach certain number of nodes or are we talking about like infinite scalability i mean i guess a latency you your your limited there but the the scalability and the throughput aspects yeah yeah of course yeah latency you're limited silly speed of light darn physics but uh with with capacity right the way that everything to be is designed right now it has a fixed number of shards you can keep swapping out any given shard and actually having more

sort of virtual storage within each that's one way the other way that we see is simply having databases sitting side by side by side by side where they have basically a shared namespace with things like I pld etc so we can get into that in a bit if you like but overall it allows the sort of horizontal scaling because you can actually have yeah a database is sitting side by side by side that that's it it's still nice to have you know as much as you can in a single database because it helps to make me worry much more efficient but you can actually go from one to two to five to ten plus databases okay so this week actually you you announced your partnership with rethink DB so correct

me if I'm wrong but so big chain TB is built on on top it adds functionality to to rethink dbd that's correct ok so rethink TB is to distribute a database system as you mentioned that it's adjacent store in a sort of traditional distributed database schema that we might think of and then big chain TB adds the all the blockchain I'm putting quotes here features such as permissions and consensus etc can you talk about so that partnership and how those two services work together before you answer I so I installed it earlier and the way that you the way that it works is that you you install rethink TBE first and

then you install be chained DB and I guess it's sort of an extension to that Internet technology yeah sure so so basically it's worth mentioning you mentioned you no permission consensus so rethink to be actually up until very recently they didn't have a provisioning system shipped they've been working on one since last fall and they just took one actually this past week as well so they have a very nice permission system we build a we have different type of permissions on top so if you think about you know a transaction in a blockchain typesetting permission is based on private keys and you know which

signatures you need signed to go through things like multi say crypto signatures this sort of thing right and so we have that sort of permission and we really sort of a Persian whatever transactions that was just a very simple transaction you know one input one output not much more and we're in the process of releasing of e2 that has all these much more sort of you know fancier features multiple input multiple output volt AC a threshold and sort of a more general way to specify some cryptographic constraints it's not turning complete doesn't have loops and none of that it's not trying to because the smart contract decentralize processing is complementary

but it makes it easy to do things like escrow at volume so um so basically permission actually kind of breaks down in a couple ways right there's these things like crypto conditions and then there's permission in the traditional way where you know this identity on this network has the ability to issue assets or this identity has the ability to to read we were holding back from actually implementing that ourselves until rethink TP came out with their appreciating system now that it's out we will be moving towards supporting that more directly and big janky be so we talked about in the way paper it's one of the things in the way paper that it wasn't out right on the very first release but it's coming down the pipe you also talked to what consensus so

this is actually a really important to stress a distributed database by its nature how do the nodes keep in sync right how do they how does how does the data keep in sync from one to the next next that's a consensus algorithm right and even modern consensus algorithms go back to research at Microsoft from a guy named Leslie Lamport you wrote this really wonderful paper in 1982 where he actually defined the Byzantine generals problem right and then proposed a solution to it and over the years technology got better better better that that line of research in 1990 he came up with an algorithm he actually said to

prove that it was impossible to do a certain class of consensus to solve a certain passive consensus problems and in the process of trying to prove it wrong he accidentally discovered a solution and it's called Paxos he tried submitting the paper in 1990 um the referees thought it was an interesting enough so he's like I forget about it but then things got more important again with the rise of the web and he finally published the paper 1988 it's notorious for being hard to understand but it's actually it's more straightforward than people realize do so and there's nicer explanations out there now by the on port himself as well as others so that and there's derivatives of taxes out there right things like raft and many others that have approved upon this over the years it's a well-established you

know consensus algorithms are well established technology once again this is the this one of the core technologies that helps to power you know big data and the modern internet so we inside reading gb it uses raft and therefore the very very poor became to be there is raft as well we have a consensus algorithm on top as well do you explain the how it works and how how it's different from say proof of work yeah sure so we have a two layer consists algorithm and maybe I'll take a step back sure happy to so proof of work basically every 10 minutes there's a lottery to decide on one of the the note the server nodes AKA miners gets to say

gets to make a call on what transactions get committed to the the chain and he so it and that lottery is based on how much electricity spends assuming equal basic power etc so that's why you know in these days it takes a lot of money to have any fighting cats at all but that's kind of proof of work right so it's it's lottery based once every 10 minutes 11 though it makes a vote and and then you know on average 10 mins later the next you know someone else gets to win etc etc etc and I follows as long as chain rule which means to be really sure that you really do have the longest chain you probably have to wait around for two or three or four five six transaction and even in

the original Bitcoin white paper he talked to the sort of the probability that you got the longest chain based on how many how many transactions have gone through or how many blocks have gone through so it's a consensus algorithm but it's actually quite slow right you're going to have to wait for 10 20 30 40 minutes for it that's proof work proof of stake traditional proof of stake is basically your probability of getting to vote to say good yes or no is proportional to how much money you have in the system how much you know intrinsic internal crypto currency etc there was clearly problems with that you know people have identified many attack vectors so people started engineering improvements around it around it where

and we've arrived to today with proof of stake it actually looks a lot more like the Federation and and maybe i'll describe federation and come back to proof of work a federation is simply a basic federation is you know you've got five nodes 10 notes 25 notes whatever and for some transaction to go through some quorum has to say yes this is good it could be three it could be five it could be majority whatever a majority or two-thirds is typically a good idea pending on what sort of assumptions are making so that's a federation how do you choose who are the members in the Federation the various ways right the original hyper ledger project for example was based on saying do you have an SSL certificate or you can say hey um you know bank a bank beat banksy bank d

you're all my buddies I'm banky we're going to make a federation together we know each other's Park prom public keys that's my list of approved knows we're good to go so proof of stake how its evolved things like Casper etc is it's a federation with very very dynamic membership based on a cryptocurrency and that's cool right it basically means you have a very open membership because you have a lot of the complexities but the complexities are basically mostly around you know the rules for membership but you can have very simple rules to or you can have external governance etc so that's sort of overall though the really three main types of ways to approach this is proof of work proof of stake and traditional Federation Stella consensus

and then sort of on top of that though you know when you have consensus you could you could say I'm taller to just crash faults I'm coloring to crash fault and arbitrary malicious faults and then on top of that you could say I'm gonna allow fully open membership to let anyone come in and do anything and maybe be an authenticator right and that's always worship thing this could be a proof-of-work system you know of course to be have any chance at all you need to spend a lot of money on mining equipment or another example is something like this teller can since its protocol where you know everyone makes a call on who else gets to like who do they trust so if I want to trade with you Sebastian

then you know you've decided that you've got so say 10 people on a list that you might possibly trust I've decided that it got 20 and there's overlap before people so it's really it comes down to those four people that we trust together so with with that what what is the consensus model than the in blood chain DB sorry sorry but yeah no no it's great it's great to give to give a brought over Google yeah census yeah models and it's grilled down you know it's a Christian but there there's a long history yeah there's a long history to this right consensus did not start with Satoshi there's this huge um belief that he did and he walked in water and a bunch of other things people that's fine if people want to you know not look into the history of computing but we care

with us you're competing because a lot of great ideas so what we have inside the chain DB it's basically two little consensus at the lowest level is within rethink DP and that is raft which is part of lineage of taxes on it designed to be more modular more atcc to use sort of like building blocks that are sort of high impedance to each other so it's sort of easy to compose them in a reasonable and it is a crash tolerant but not visiting fall to learn not not fully tolerant to arbitrary attacks one level up and this is part of the big change EB server code etc we have

technology we call it the big chain consensus algorithm and it does voting so basically remember how I mentioned before it's all about the order it's all over the order you want to get rid of the way of the database when it's writing so we let the rights come in we don't one node will vet them but then we write as soon as we can as soon as we can as soon as you can into the the is the one with that are there for good but they're not bet it we just let them right why because it's much faster to just let it right let it right get all the way but then after the fact what we do is we that the Federation nodes vote they basically each says yes this that's good or no it's not it's good if every

single transaction inside is good um the block isn't good if any transaction is bad so that emerges as consensus at a higher level for that block and if you think about it you could actually this one transaction at a time but we simply group them into blocks for speed 0 because it takes you no time to hash set of transactions and write it etc etc um so so we group them for speed no more no less and if a block gets you know if the majority of nodes voters say yes this block is good then it's considered good and um then anyone else doing reading

afterwards it has knows that it's usable that it's a truth if the nodes vote and the majority says it's bad then it's considered not truth and any transactions that are still possibly okay get copied back into this sort of incoming buffer and and try it again so basically in this mechanism we've got two levels of consensus the lower level is full tour and the higher level is the lower levels cut the higher level is controlled only plus plus so it's got a bunch of other mechanisms in place to validate and bet and verify and as time goes on we are making this more

and more resistant as well towards a Byzantine fault tolerance and in fact we have some pieces of technology right now that were iterating against to make it do that in many settings so as somebody so two levels of consensus the lower level is raft which is not Byzantine fault tolerant but crash fault tolerant yep and then the layer on so so there's basically one level of consensus that else all the nodes what data to put in and it takes all sorts of data like you know correct transactions incorrect transactions etc and then once all of the transactional data is inside the database then another level of consensus which is nodes voting to order these

transactions into blocks into wild blocks not quite so it's not voting to order the transactions and blocks so the thing is like even at a fault-tolerant database you know running raft whatever right it's already ordering databases do this they need this to do the logs right it's built into the database right Satoshi didn't invent ordering right so and you get rid of the way and let the database write these orders itself right so at the lowest level all they're ordering is being done and actually it's being grouped into blocks there too because you can the thing at the higher level that's being happened is it's voting on whether any given block is valid or not and it's valid as long as any transaction on the block like as long as all the transactions in the

block okay so the lower level is composing the ordering and the higher level might be imposing say that no extra money is created or kind of yeah so basically right what what premise double spending is the higher level of consensus and in level tensors water transactions happen and which came first exactly and this is absolutely critical to performance so basically the bottleneck is in right right so if you can actually manage get the throughput for writing really really high by you know letting the distributed database do it thing do its thing right right right right right then you're okay

and for the writing but then you have to say okay um should everyone trusts what just got written and that's what this higher level can set this is a boat right and that higher level though it's actually just voting as well you know each one of the nodes it's doing its own writing so this but there's no sort of central blocker right so um this is sort of the breakthrough it's sort of reordering what's happening letting you know ordering happening with this you know big data distributed database technology that you know runs fast already but bringing in the decentralized control via the nodes having a say on whether a block is valid or not so like one of the interesting

things in a white paper is is you is is sort of your claim that big chain DB implementations called forth is that is that correct and why is it so yeah so this comes down to once again if you think about a distributed database a traditional one right each one of those server nodes is writing a log right just writing a log one order than the next in the next in the next this is the essence of a consensus algorithm a traditional one way like tacos her raft or what are these so this not like Paxos says okay suddenly you know imagine you're writing a piece of code and it's a for loop right oh Ken that for loop code would suddenly be mutated such that it's like splitting into a nested loop you know

like it's just not kind of in the code at all anywhere you don't have the constructs right so it within distributed databases there is no construct for a fork because just those you know abstractions haven't been built in they would have to be built in at the very core right and and that central problem of sorting out ordering like solving ordering exactly what paxos and raf do and it's just an order that's no more no less right you you can't they're just slotted in there that's it and it's sort of how the order is resolved is based on what's getting stored within each server node and then what is the protocol of rules around how to interpret that from each other server node so basically order is

input like order is comes as a you know first-class citizen with traditional consensus algorithms like paxos and forking is not even in their vocabulary so let's talk about some of the applications for Beijing db1 what I really like what the the website is that you have some really nice illustrations there on how be chained DB can be implemented and sort of more I guess traditional centralized just infrastructure where you would have like an Amazon Web Services instance and a database etc and all the way to having it completely decentralized with the theory am running as the vm and I PFS as

the file system and big chain TV is the data store could you talk about so the different use cases and what would be some practical applications for each of those types of scenarios sure sure yeah so to sort of summarize right there's a framework on which you can use or I'll of deployment where you can use big chain DB where you keep your stack the same right and you just and right now you maybe you're running in your sack you've got you know five different databases you've got you know maybe my sequel for your sequel database and maybe MongoDB for your no sequel database and maybe you know neo4j for a graph database or something and maybe

some others to trip on right it's pretty common these days so guess what you throw in another database right this one though is special big chain DP because you and don't you as yourself don't own or control that yourself it's it's you and this Federation of other organizations in your ecosystem right so as far as your system is concerned and your developers and your CIO etc you've got this other database that you're you know you've got hooked into your system but it's special because it's data that you and the other organizations in your system are working on against together right so that's really cool because it means that you can you know maintain the rest of your stack you don't have to go you know all in um you can just bring extra benefits to some of your existing applications or

you can also another way to think about this is you're thinking about a new application and but you only really need to have say time stamping right you don't need to have full on decentralized processing or full-on decentralized media storage so great you have your team of developers that understand modern web stacks or modern enterprise tax just use them right keep doing that stack now you just have a new database that you can plug in and use to get some of these main benefits of decentralized and we know this very well because you know this is how we built described for IP um you know we came up with this idea in 2013 you know long before Imperium you know the a3 and white fur became one

six six month later and and they've been talked to this you know in various places and stuff but what's cool is you know a scribe itself is running on on a modern web stack all the components you would expect running on AWS except Roku etc and at the place that we're where the rubber hits the road that really matters for you know who owns what that is on a decentralized database right now it's in Bitcoin we're in the process of porting it over um we will fully deploy it um when the public big chain DB it goes live and so that's up critics and i'll talk about general and i'll talk about some very specific use cases so

the general there's you know sort of the portly decentralized stack that i just described which is really easy to adopt especially for the enterprise the other stack is sort of this full blowin fully decentralized stack which we're very excited about of course can be deployed in a private setting many people are for things like business logic or for um lots of other applications but what's really exciting to me is the the public deployment where you have this you know world computer as a lot of the etherium folks talk about which i think is really awesome right and so you can have a decentralized database really you know a planetary scale database that no one wants or controls a decentralized

file system which and it's I PFS is emerging as the winner there and processing on top and if you're a public databases varium public network is certainly the leader here with some other technologies we believe that are really interesting and helpful there to things like 10 dermot where you can actually have decided eyes processing with other languages right if you're IAM vm is really designed for solidity a theorem language which is really great for a lot of applications but for other developers who don't have a lot of code in Python or only really feel comfortable in JavaScript and that's cool too because you have things like tenormin so and these things together

right decentralized processing a centralized storage and finally decentralized communication just sort of built into the protocols and stuff that is really the elements of compute computing ie you know together working together a world computer right and then there's the applications on top right decentralized app stats and you know there's already starting to be an explosion of them thanks to especially at nerium and elsewhere too and that's very exciting and of course in 3m right now um isn't quite at the scale that people want but there's you know several really smart guys working on it and I've had deep long conversations with them and I'm excited what the directions are

headed right so will it will get there and that's great because it means that for example imperium will be able to keep up with big chain TV you know the faster it goes the better it works for us but also you know we've seen it's it's really hard to develop a decentralized app if you don't have you know in let's say you're used to being a web programmer where you have say an MPC model right so m is for model which is you know instantiated traditionally as a database you don't have that maybe you only have ram what do you do right or a file system so it's much easier when you have a through-and-through database to work against or you know the other way of thinking about it there is the traditional abstract right so linux apache my sequel PHP worthy em to my

sequel for the modern decentralized stat so this is sort of roughly like the higher level right sort of partly decentralized where the database really matters at the core for decentralization the rest can be centralized and fully decentralized where you actually have all the pieces decentralized and their shades of gray in between for example to write you can have big chain DB with ipf s and that's it and by the way to ascribe itself will be moving towards a fully decentralized deployment bit by bit by bit so we're working with several organizations on that quite excited for applications themselves maybe I'll just give a quick sampling one of our

companies were working very closely is ever ledger so ever-larger is about diamonds on the blockchain ran by a woman named Leon camp who is a force of nature she's really awesome and she sees that there is major fraud and corruption in the diamond world and it's basically been a very opaque industry so you know bones get dig it dug from mines these mines themselves are often in very corrupt countries some countries are have actually become so corrupted they're not even allowed to dig diamonds anymore and then these diamonds you know these rough stones they come they get taken to certification houses and will sorry cut first taking a certification

houses there's a good vibe in the world that matter and and then they get measured and certified and an ID is even laser etched in there actually and by the way on the path miss there's a something called the kimberley process which basically sort of to help that if you're authentic or not and then once it goes past these four houses each diamond you know has this sort of piece of paper that supposed to be attached with it and then it gets set through various distribution channels that ultimately ends up in the hands of retailers whether it's online retailers or the local tiffany um and basically each step along the way there's tons of opportunity for fraud because that piece of paper could get separated from that diamond that ID might be lying and there is many many

examples that you can see for example one of the major certain houses got hacked and their data records you know it got very different on the heels not getting hacked and if you look at the the retailers themselves that are selling the diamonds so we actually work with our ledger am and what's interesting with blocking technology right it encourages companies that are sort of traditional computers and encourages of the share data in new ways to get sort of an ecosystem wide benefit in the case of ever ledger the different certification houses share their data with ever ledger and then we worked with our ledger to cross-reference at data against the retailer data and we discovered basically by

cross-referencing that and apply machine learning techniques we discovered seven percent fraud on one of the major retailers there was 20,000 diamonds a day going through this retailer it worked out to a 750 750 million dollars with a fraud in one year so that's quite exciting and that's just one retailer this is pretty important right it's an 80 billion dollar industry the fraud was just seven percent from that initial data mining it could be as high as thirty or forty percent so we're talking up to 32 billion dollars that's that's diamonds but it generalizes so in general supply chain transparency is a big deal for the world of blockchain technology yeah i agree i mean that this is this is something that were particularly interested in at strada

mrs. is bringing more transparency and ability to supply chain I want to come back to tell you you re mentioned earlier which I thought was really interesting is it's having that database component to to a lot of things a lot of the things that people are doing with the block chain is being with the time to them things and notarized and having that component there and that stack is it's really essential we ask yourself that question all the time is ok well our clients are gonna be you notarizing data through the Bitcoin blockchain but where is that data being stored like or are they storing that on their servers or in some cases that doesn't really make sense because you want to have it and you want to have a centralized I want to come back before we sort of wrap up here

I want to come back to the idea that well so with Bitcoin yo we have the Bitcoin protocol and so the core protocol and it's released as a public blockchain but a lot of the new protocols that have been emerging like your am and I PFS and now beach in TB have this functionality within it so that they can be public but they can also be private and there's some shades of gray in there so you can be on the ethereum blockchain but you can also you know deploy athyrium as your own private network if you like for semi private network and so can you talk about so you also talked about a lot of the the

public implementation of botching DB and I suppose they would be some so if semi public or you know some fields or industries that would implement their own implementation of of a lot of btb and then some totally private implementations I'm particularly interested in the public one because I think that can bring a lot of value talk about how you see that playing out like would be the validating notes like can anybody join and can anybody use the network is it a public good does it yeah how does that work yeah so we're spending a lot of geologists and we'll be having more announcements in coming

months on this overall we will be rolling out a foundation for this and it will be obviously a non-profit we are working with organizations as know that have demonstrated commitment to the health of the future internet and this is organizations that have demonstrated to care in the past or are demonstrating a care in the future by dedicating real resources to this and overall the key thing is that permission will be set such that anyone can issue an asset anyone can transfer assets anyone can read all of that and that's really kind of the core things right and

the capacity will be like so big etc that it really can be a store of world data there's some really awesome apps that can emerge on this too right the one that gets us excited actually brings us full circle is IP creative works right so the the you know what is the Mandate of a museum it's to preserve the cultural history of what artists have done over the last you know up decades and centuries um and there's cultural mandate museums and Archives and many other places libraries etc right what does that look like in the 21st century

in the 22nd century in the next 10,000 years right it's going to be digital it's going to you want to have it live but you want to have it as well where it's you can trust that it's going to be around right and that it's immutable etc so it's really great fit so we're going down a path for this which is basically we're using ascribe the company itself the and the product described the team that's still working on ascribe to drive to IP in this public became vp and working with other cultural its institutions to for example we continue to iterate with butyl commons france and a lot of other organizations that really care about this so ascribe the team

actually itself almost acts like a non-profit right it's it's quite well tuned towards the needs right we have a lot of people that are you know art pros you know art curators etc sort of in that space that we work with so and it's not just art right it's not just digital art it's also things like music and n books and and film and video and video games and all of these things right this is all sort of creative cultural artifacts where the attribution desert story public and the promotion for Providence when people it deserves to be public to this is a really big

flurry up there's also other apps to around this we see a lot of marker lots of companies doing really great things in identity and reputation and much of this makes it on public big chain db2 or some other you know technology this as that skin so there's others to write some sometimes things though do need to be more private you know a lot of banking things simply to banking regulations they need to be in private networks that's fine right so there's really sort of shades of grey but it's really the the public big chain DB that is you know drubbings us as an organization philosophically we're you know we're excited that we can help to play a role in creating this this future

internet this future society that we don't want yeah that's uh it's interesting prospect I mean they why would you go you know get a MySQL database or you know it's upsetting possibilities for developing applications without needing any sort of infrastructure just like the the i guess the permission lyst side of it is what gets me really going about this it's just being able to utilize this public good that is by much like bitcoin much like a theory em and I PFS another the protocols but yeah and so this this foundation would I guess pick the validators and yeah yeah so the

foundation will be picking the validators how we see it though so right now the technology does not exist to have open membership at scale in a way where you can store all the resort store all the data you want right you have to make a trade off right and so some of the other works out there have made a trade off to say we're going to allow you know thousands of notes but it is severe cost scale right severe we've said oh but they're trying to scale up right and you know some of the efforts I commend in big ways right I think what like you know things like a sprint speciale okay awesome right great and rock lon there's another way to approach it though so rather than starting with you know

thousands of nodes as validators remember you can have hundreds of thousands all that as clients so rather than starting a thousands of nodes as validators and trying to add scale start with a smaller number of nodes as validators while still supporting hundreds of thousands or millions of clients so start with a smaller number of alligators have scaled from day one and then evolve the technology to support a larger number of validators and that's exactly the direction we're going where things can be open membership you know two years three years four years from now right with with things like the stellar consensus protocol or other protocols like that right so there's some emerging stuff right there are two but it's sort of at the end what we will all end up in a

place where it is broadly decentralized even in terms of validation as well as the scale of the planet is curves right no one is there yet we have a you know a scalability that the planet deserves a scale and a more limited set of notes what's cool actually is we have a prototype for what we're doing what we've been doing at our archetype if you will it's called the DNS right the DNS has been powering the internet for the decentralized gns has been following the internet since the late 90s before that you know it was actually going back to the 80's early 80s he was it actually a text file you know one guy was maintaining it you know for many years

and then he switched over to his buddy right and it's kind of amazing right and they just kind of had to do it but you know as the internet got more important and it made sense to decentralize control and this is what happened and so we've been working on the architect achill architect of the dns his name is david Holtzman and we've actually been working closely with him for more than a year because this is actually a um you know a linchpin technology now there's things you know when when they created the DNS there was a business side to Jamarat we work with him too they had no after I can and you know over the from you know 1999 2002 now you know i can has devolved it's not as good as it was 15 years ago right there's this

challenges and you know there was some design challenges technical in the dns it wasn't perfect but it's pretty good for the time now we have another 15 years worth of learning from Krypto etc and we have all the lessons from i can we're drawing in those lessons you know for this next-gen database for the planet and something that's more broad in scope than the dns at the very least for IP because that's what ascribe is doing but we also have a lot of other people who are very interested in using it for a lot of other applications right so that excites us like I said it's sort of it drives us it's what gets us up in the morning it's you know what makes us excited for the future well that's a great note to end on Trent oh and thank you for coming on the show once again

and hopefully we have you got on again in the future as as big chain TB it continues to be well um thank you for having me i really appreciate and thank you to our listeners for listening we are part of the epicenter story we're a part of let's talk bitcoin network you can go to let's talk bitcoin calm to find all kinds of shows about but block block chain bitcoin and decentralize technologies you can listen two episodes of episode in a bitcoin every monday they come out on your podcast feed as well as stitcher youtube or anywhere where you get your podcast whether the itunes or android or wherever and to our loyal listeners you can always leave us a review wherever you can it could be in

itunes or any other platform if you do just send us an email at show episode of a cool calm and we will send you a t-shirt and some stickers and we thank you for the reviews that you bust so far as I'm not really helpful in getting the show more people so thanks so much and we look forward to being back next week