#192 Aviv Zohar & Yonatan Sompolinsky: Of Spectre & Ghosts – Radical Ideas to Scale Blockchain Tech

Transcript

[Music] this episode of epicenter is brought to you by the I prize and the energy Innovation Hub the I prize is an international startup competition to build the machine economy go to Epic center.tv I prise that's i p r i z e to learn how to join the competition hello and welcome to epicenter the show which talks about the Technologies projects and startups driving decentralization and the global blockchain Revolution my name is Brian

Fabian Crane and I'm M Roy today we we are going to talk on a very interesting Niche but potentially very powerful Topic in in cryptocurrencies we're going to talk about cryptocurrencies that don't have a blockchain as the fundamental data structure it has like something else so we are going to talk about a protocol called Spectre which was developed by Dr Aviv Zohar and yonatan stolinski both of them are at the Hebrew University of Jerusalem aiv is an assistant professor there and yatan is a PhD student there so let's get started Aviv and yatan uh thank you

for coming on the show hi thank you for having us yes it's an honor so so tell us about a bit about your background how did you get started in the field of uh cryptocurrencies starting with aiv okay so um I think for me it all started when I was doing my my own post do I finished my PhD um and I was at the Silicon Valley lab in of Microsoft research um as post dos often do uh I was looking for an interesting new topic it was 2011 so bit coin already existed but I think almost nobody heard about it at the time um and a friend sent me like um a newspaper clipping uh Bitcoin got into

the news because it got to $30 a a coin and uh I thought it was really interesting because I was uh really interested in protocols and economics and how they uh intertwine so I went to read about it and I ended up writing a paper um with other people in Microsoft research at the time which was apparently one of the first academic papers on bitcoin not the first one but uh very early on we I I remember our mentality at the time you know Bitcoin was the first bubble maybe was crashing and and we thought we had to get our paper out as as quickly as we could before this thing vanishes um so uh basically that was uh

that was our driving motivation and uh I think after a while I I joined the Hebrew University I started working with yonathan and we delved much more deeply into cryptocurrencies and what about you yonathan well um my bachelor's degree was in mathematics in the hiu and then um I went to computer science and uh the first um staff member I went to was a viiv Zohar and he kind of told me about Bitcoin and I thought this was a too practical for me it wasn't an interesting topic and I said you know what I I'll look for someone else and then a few a few months afterwards I I uh discovered that um I rediscovered aviv's um uh role and I went to him and

we started working on it and I think it accelerated pretty fast of you right uh yeah definitely I think it was an early time for Bitcoin there were a lot of very fundamental questions to ask at the time and and youan I'm very glad that that after all with what we're going to talk about today uh you guys did manage to find something very abstract and retical and you managed to come away from from doing work that's all too practical yeah it's a it's a very it's it's I feel lucky for um for this um uh marriage between Theory and and and practice it doesn't happen in any other in fields that I uh I come up across uh here I mean it's we really work on the

theoretical stuff thetical algorithms of even myself and then we find them like very very practical so it's I feel lucky for that so most of this show is going to uh focus on one of a new kind of cryptocurrency design that uh that aiv and Yan have kind of pioneered but before we started uh before we start going down that path when I checked aviv's um homepage it lists his research interest as multi-agent systems and algorithmic Game Theory now now a like can you tell us like what what these fields are and what what do they concern themselves

with sure um so multi-agent systems if I was to describe it very uh briefly is a subfield of artificial intelligence really um and in artificial intelligence you want to understand how computational agents behave when they're doing things in a Smart Way multi-agent systems focuses on these systems that have a lot of agents that are behaving in some intelligent way um and in that sense you can think of of uh systems that are connected on the internet right if you you can think of peer-to-peer file sharing um as an agent running on your machine is managing things for you he's downloading files he chooses where to

get them from and so on and he acts on your behalf he wants to do intelligent things so multi agent systems look is a field that looks at the uh and result the system that emerges in that sense I think Bitcoin could also be categorized as a multi-agent system the term is very amorphic in some sense um where every minor might be doing clever things um in the in the in in how it behaves how it mines which transactions it chooses and so on um and one of this the tools that are used very often in this field is Game Theory if you want to understand how somebody acts when he's intelligent you can think of him as a rational actor as a player um so you can analyze the behavior of

the system and maybe even design it um and that's where algorithmic Game Theory fits in basically if you want to talk about intelligent agents you need to think about algorithms and Game Theory and how they are connected um and so my research before Bitcoin has been about systems um in general and thinking about protocols as game theoretic uh interactions so if you think about the protocol maybe that sends of the internet uh you can think of it as a little bit of a game it the protocol allocates resources maybe we decide who gets the bandwidth and when um and you might compete with others for bandwidth so what what happened if your computer tried to get more bandwidth out

of the internet at the expense of others um so does the protocol uh which sets the the rules of the game so to speak uh uh induce good behavior when you think about players as being strategic um so that is a natural foray I think into Bitcoin where the protocol really pays people right and so game theoretic tools are very relevant to understanding how how nodes would behave how agents would work so um I hope that uh covers it yeah no absolutely I think there's a there's a lot of very interesting uh interesting overlap and I'm sure this is going to get even more relevant right

when you think about like some sort of computer agents doing all kinds of things you know in the future when maybe some of the decision makers in the blockchain universe are going to be Ai and Bots and and uh so will be a I'm sure there will be these interactions are going to become even more relevant uh yeah so I I even think simple systems have interesting interactions like if you even think of um you know miners that are connecting to minor mining pools and they have a small agent does something very silly just it chooses which mining pool to connect to based on the profits that they would expect that already induces a very complicated system of behavior it's nice to think about that what that would

do what what are uh the results so so this is everywhere I think in Bitcoin because incentives really matter in the protocol so is do you think aiv that there's a link between um like smart contracts as a technology and uh the academic fields of of multi-agent systems and algorith M Game Theory cuz like smart contracts essentially give way to like control and partition Financial Resources right and uh is it the case that like smart contracts will become sort of the implementation layer for many ideas from from your field so definitely we've seen many things come into cryptocurrencies that

were um that have grown Within algorithmic game theory with in economics uh we see things like uh reputation systems prediction markets all of these things were heavily researched in computer science before uh cryptocurrencies show showed up they're implemented with smart contracts um but you know more generally smart contracts do everything that computers do right so it's again very natural to think about algorithms that move around money and to start to think of how they would optimize how they earn more um so game theoretic tools are at the foundation of this and even if you look at things underlying uh the high level you know

the the very low layer of Bitcoin has an economic or game theoretic argument for why Bitcoin is secure we're paying people to mine and so they're doing more of it right um so in some sense even the foundation of Bitcoin relies on economic and game theoretic concerns without it there is no system so you guys are well known for for a few things but one of them is something called ghost which probably many uh people have heard about in some context uh most mostly in the ethereum context so ghost stands for greedy heaviest observed sub tree which is quite a mouthful but uh basically it's a way

to improve some of the game theory around proof of work can you run us through what ghost is and and how it works okay so so ghost is um a slight change to the to to the Bitcoin rules I guess that we we came up with when we were doing some analysis of what happens to bitcoin when you try to scale it up so one of the problems that when you try to scale Bitcoin up either you increase a block size or you add more blocks per second right you increase the block time from 10 minutes to something lower you try to get more throughput then um you end up getting more and more orphan blocks um so these are blocks that are

created in parallel at the same time by two miners that have been trying to work but because blocks propagate relatively slowly if they're larger for example um then they end up creating conflicting blocks more often than usual uh if uh and so ghost was an attempt to try and still use the weight of a block um and support the the chain that's selected so right the Bitcoin usually takes the longest chain rule throws away everything that's off the chain and just completely ignores it uh instead what ghost would do it would say right your a block that is off the chain still supports in some way the weight of its uh of the chain

itself um and the this mouthful that you described greedy heaviest subtree is really just the description of the pro of the algorithm how it works um um you can you can look at a at the Block structure instead of a chain it's actually a tree every every block points to a predecessor uh we may have many leaves um and when we travel along the chain we pick the Chain by basically greedily selecting the the the child that has the heaviest sub tree um I guess you need to go into the paper a little bit more to see the details but what we were hoping for this was a very early attempt of ours to to try to adjust the protocol to

make it a bit more scalable that if you can speed up the protocol uh a little bit and still maintain the same level of security that you had before um is that right maybe Jonathan wants to to add something to what I just said well it's I yeah I agree with what you said more or less it's it's generally I would say it's not so much about the game theory it's more about how to utilize orphans to to enhance the security of the main chain that's the main benefit of ghost you have you can and you can scale it you have more scalability options because because you use orphans to support the main chain yeah so so maybe just to explain this a little bit for because some

people I think will not not quite understand these relationships so if if we have H in in Bitcoin right we have a block every every 10 minutes so if a new block is created it it will take some time to reach uh to propagate netor work so you know maybe some miners in China only get it 30 seconds later so in in those 30 seconds they would be mining on actually a block that's already outdated so most likely if they find a block it it would basically be thrown away and uh and sort of the the the work or the mining hashing power would be wasted uh and uh and of course the the more you go down with a block time uh the more this propagation time matters and the more uh orphan blocks occur and and does in a

way the more waste of of work occurs and so if if you but if you essentially kind of still count those orphan blocks uh and and they still contribute to to security then maybe that's less of an issue and and maybe you can do some of the things that that now people are conly saying no you can't do it uh in the case of Bitcoin right one of the big arguments against increasing the block sizes that people say uh you know it will it will favor bigger miners it will create a problems for minus aren't so well connected it will slow the propagation of blocks Etc so it's a it's an interesting Direction uh Yes actually I think you this you mentioned a an important advantage of ghost when there's a

network split then you know some miners um uh didn't hear about the C the updated chain and they mine uh on a fork then what you want to happen is that despite the conflict between these the chains of this these distant uh factions and network you want uh all this mining power to still support the the block uh the last block that they they all agree on in the main chain right so you don't want all the work to get discarded uh you're able to tolerate or um temporarily Toler tolerate the fact that there's a decrease in the in the in the consensus um rate in the network but you still want all blocks to work against um

to to enhance the security of of previous blocks that's the one main advantage of ghost so ethereum ethereum seems to do something similar right where if um there's a main chain and let's say I create a block that wasn't included in the main chain even then I can be be paid for creating what is called an uncle block which is a block that is not on the main chain but is referenced by some other Block in in the main chain so H how is like ghost different from what ethereum did like did uh did ethereum Implement all of it so um ethereum came out and we were pleasantly surprised that they mentioned

the ghost paper in their white paper uh we kind of thought it was flattering we didn't know ethereum was coming and we actually didn't know that they had intended to to implement um but what we ended up finding out is that ethereum doesn't really Implement ghost uh per se what you what you're mentioning uh this payment to un to uncles is something that could have uh been interpreted as a different papers of ours which was published at the same time as as ghost it's called the inclusive blockchains paper which uh in which we suggest that you pay uncles as well um so ethereum what what ethereum doesn't do is it

doesn't use the weight of the block the proof of work that was invested right into the into creating the block to somehow add weight to a chain if if there is a fork for some reason that you need to decide between um in that sense ethereum didn't add more security uh by using ghost um so what what we should say is that uh this uncle's only variant of of ghost that ethereum claimed to have is a very nice one it's a ghost itself if it's used purely has a some some problems with it um that maybe we won't go into but uncle's only ghost has a has a very good uh um I think security benefit and actually root stock that

kind of I guess competes with ethereum uh did Implement ghost as as we described it or at least an Uncle's version uh ghost uh maybe with some additional modifications um so it's definitely doable but for some reason I don't know why ether chose not to and just to just to clarify that because so so if you say Okay ghost adds you know the weight adds to the security of the chain do you mean that let's say now we have uh you know we have two blocks found around the same time at the same height uh do you then mean that let's say there are some additional blocks uh found on on you know one side uh that those then somehow contribute to

that and they somehow uh make that the give like weight in a similar way that length matters with blockchains or can you explain so let me try to explain it um abstractly yes so let's suppose we have a a split in the chain at some point we have two different competing blocks at a certain height um so what ghost would do is it doesn't look at the length of the chain above each one of them that's not what counts the choice on which block will be accepted depends on the size of the entire subtree of blocks below one of them so if the network is building blocks in some way that it does not build a long chain they're doing a lot

of they're building a lot of Orphans but they're still on top of one block then that block would get a lot of weight despite the fact that the chain is not very long there uh the attacker might be living on the other side of the fork and he might be creating blocks using his you know a data center that's dedicated to it and he might even be building them in a long chain but as long as the compute power on on the side of the network is is greater they don't have to be building blocks on top of each other to secure that block on the fork right so somehow this Fork gets really heavy weight as long as the entire network Builds on top of it um the the risk that ghost kind of has is that maybe we we're going to have ties between the two

systems it breaks ties a little bit slower than uh than Bitcoin does so if you use an uncle only version of ghost so maybe it'll take a little bit more time to resolve the fork but once it is resolved um it's going to get overwhelming weight even if the block time is is really fast so even if the network is creating a lot of Orphans which we would have usually been thrown out in Bitcoin we still get a lot of weight on that block right does that make sense yeah it makes sense and I guess in the Bitcoin example this wouldn't make much of a difference but then it would make a difference if you if you have a different kind of protocol that has has much faster blocks and more orphans is is that correct yeah iess I guess what we're imagining is a world

sometime in the future where Bitcoin needs to process thousands of transactions per second and we really want larger blocks and faster block times we don't want to wait 10 minutes at the supermarket to to get a confirmation so you know the naive attempt to just increase the block size or to speed up blocks which I personally prefer a bit more would have gotten us into TR with Bitcoin um it ghost is still not a perfect solution uh you know the uncle's only version of ghost gives you a little bit more security at the expense of a slightly slower time resolving Forks so you couldn't really scale up to let's say 1 second blocks that would be too much um but uh ghost

for us was just the first step in a progression of algorithms that kind of move in that direction Bitcoin with the blockchain which is like the straight chain of blocks and now you had you had you propos ghost in which uh there's like instead of a chain of blocks there's like trees at at each level right like so one block can refer to let's say two or three blocks in the in at the previous level and and so on and now your logical next paper is the Spectre paper which is the main topic of uh for today um which I think in a sense like is an even General version of ghost and maybe an

even more scalable version of uh of uh of Bitcoin so uh please walk us through like what Spectre is and what it seeks to do yeah so I wouldn't uh advertise Spectre as a generalization of ghost um although it does uh ghost is an increment incremental step towards Spectre uh the main idea of Spectre is just to um to integrate any every block that's created uh into the ledger so we don't want to discard and uh get rid of uh blocks because this will be a waste of the security a waste of the throughput you want you really want uh

in a healthy system that every block is integrated into the public Ledger um so this is uh technically what Spectre does every every it's kind of grows a massive Dag dag is simply a graph certain family of graphs but it's a graph containing all blocks this is how Spectre works but the main agenda of Spectre is is to create a protocol um where the protocol imposes no um bottleneck on the on the SW of the system what you what we want is that um the protocol will be secure under any sprut and the limitation of scut would be the network infrastructure um specifically uh the nodes bandwidth

available bandwidth for operation uh in today's systems um the bottleneck uh and and Bitcoin as we spoke earlier is is the the protocol security not the network infrastructure and the challenges that Spectre set out to solve is to create a protocol whereby you can increase the throughput to any level that the network can support and and under any such level the the protocol will remain secure yeah I mean maybe I I just want to sort of throw in how how I how I see this uh and this is not a very technical explanation right but in in in Bitcoin right we have or in something like ethereum you have these kind of

transactions getting like accumulated quite slowly kind of center around and then everybody kind of stamps it and you know kind of every 10 minutes you can say like okay you know we move one step forward we move one step forward we move one steps forward you have this very kind of slow and and tedious process and then and of course you know every block Builds on the other block right so you can't you can't just do something but it has to exactly rely on the previous block and if it doesn't it gets thrown away and then if if you look at this uh Spectre or some of the other projects going in this direction you have all kinds of blocks being created at the same time and very very fast and and with like No Limits and you know

transactions just go in and and then somehow they all you know so it's not like there's one height and one block and then the next one but they're like yeah like this graph web form and somehow they keep connecting with each other and and confirming and then you have this seemingly this massive uh speed up of um of transaction and of course the difficult thing is it's really completely different so it's it's very hard to you know for me as somebody not very familiar with it it's very hard to think about does this make sense does it work what are some of the risks what of some of security flaws it's very hard to do that coming from Bitcoin right so maybe I should uh address that concern uh carefully right

um so in Bitcoin we have a very nice security analysis that explains to us why the protocol Works basically Satoshi didn't do it exactly in that form but it later got formalized into a theorem that says you know we have certain assumptions about the network uh that uh blocks can be sent around quickly that nodes are connected um and then a transaction that's embedded somewhere in the blockchain um becomes irreversible becomes secure as more blocks are added right this is the basic claim in Bitcoin uh in Spectre what we do we we we we have a formal model of of the protocol of how the network works and the assumption that we make is that if the

protocol um Works in a connected Network pretty much like Bitcoin um um and blocks are built by by a majority of honest participants that have proof of right the proof of work uh is done by uh by the honest nodes and they have more computational power than the attackers then again the same thing happens uh as in Bitcoin uh the probability that the transaction that was accepted uh will ever be rejected for some reason that will change our mind um decreases very fast exponentially fast in the system so um uh maybe it's a good idea to start to talk about how Spectre is kind of built what it really allows is to uh for every minor instead of just

to pick a pre a single predecessor just right in Bitcoin uh every minor creates a block the block has a cryptographic hash of the pr preceding block that's what makes up the blockchain these these links uh pointing back in time somehow to previous blocks in Spector we just allow every minor to write several blocks in its past and you can think of these as um blocks that were created in parallel that this minor is aware of so we don't force you to just pick one you just you're supposed to tell us about every block that you've seen um so this is just a slightly different data structure instead of building a chain it builds this long thing that still grows uh and merges together splits that occur in the

structure um and the the idea is that if we speed up the protocol we we're going to have a lot of blocks built in parallel so we're going to have to do a lot of these merges together um and then what's really important the real magic is to somehow end up with a consistent set of transactions uh you can think of it as a utxo set in Bitcoin that we we've agreed that these are the transactions that have occurred in the protocol so just like in Bitcoin every node that creates a block sends it to everybody else so we have pretty much the same view of this data structure um if we get a block we have a list of the preceding blocks so we know what we're supposed to read and and request from uh other nodes um and

uh the only difference just like in Bitcoin you may not have heard about the last couple of blocks maybe right before they propagate to all of the nodes so in the dag scenario you you know about most of the graph that everybody else has seen you might not agree about um certain uh last portion of it that was added and then what the protocol tries to do is to take this data structure and apply a function to it that will basically give us what is the set of transactions that did happen and the properties that we'd like to have is that um maybe two main things that we want is first that when you put in a transaction it will get accepted at some point hopefully very quickly and

the second thing is that once we accept it as uh as as a valid transaction that's in uh the data structure then we never change our mind right if we if you got money then there's never a decision that says you don't have it um and if and and these are basically the two things that construct a consensus protocol uh we want to make a decision really fast and we want the decision to be irreversible at least with high probability um so this is again is the goal of Spectre the end result of course is that people can write blocks together as quickly as they manage to we can speed up block creation by by making the proof of work less less hard right there's if there's a parameter in

Bitcoin that says 10 minutes you need to create a block pretty much every 10 minutes we can go for one second blocks which sounds outrageous you're going to have a lot of Orphans but they all get merged together anyway and we're still going to Output a consistent set of transactions okay that's the magic let's take a short break to talk about the I price a competition being run by the energy innovation Hub the I price is all about the machine economy the rapidly evolving relationship between humans and machines with huge technological revolutions coming like blockchain and artificial intelligence some crazy changes new developments are ahead of us like autonomous driving

self-organizing Supply chains DNA replicating robots and so much more if you're doing work around these areas the I prize is your chance to do like Elon Musk and take it to the next level it's a competition that's being run until July 28th startups can apply in three different categories and have a chance to win up to €250,000 in seed funding even if you just have an idea you can apply as an individual and get a stipend office basement in Berlin and mentorship to grow your idea so whether you're just mulling over a world changing idea in your basement have built your first prototype or founded your company you can participate and make it to the great finale in Berlin on September 28th so go to Epic center.tv / I prize that's i p r

i z e to learn more about the competition and how you can apply we'd like to thank energy and the I priz for their support of epicenter so maybe maybe I can like sort of create a sort of imagination for for Spectre and please tell me if if it makes sense so so let's imagine that like the four of us um we are all minors right and there's not just the four of us but there's like Sebastian there's there's other people around the world that are also miners right and all of us are basically connected using a peer-to-peer gossip network of of some kind similar to very similar to Bitcoin now in in Bitcoin what would happen is like every 10 minutes like all of us are

trying to solve these puzzles and let's say Brian ends up solving the puzzle like he's the winning minor and like in Bitcoin he would create a block and then 10 minutes later there would be some other winning minor and they would create another block and and so it it builds on block by block and it makes a chain but in inspector it's like all of us are trying to solve the puzzles and it is very much possible and very much expected uh the block time is like 1 second so right this second it is very much expected that let's say three of us create blocks so Brian creates a block yatan creates a block and M creates a block and we broadcast all of these three blocks through the gossip Network to others now that was this time

in instant let's say 1 second later the next time instance um Aviv creates a block and just Aviv creates a block but then aiv can refer to the blocks of Brian yatan and M so he can so aviv's block can have like three parents all all of these three blocks can be the parent and then the the instant thereafter t+ 2 seconds seven miners around the world who are not us but some other seven miners they create blocks and it could be the case that some of the like two of two of these miners included all of our blocks as parents and the rest of the five they maybe just included Yan's block as a as a as a

parent and so like miners keep building these blocks and they refer to each other as parents and like you get this sort of data structure that is not a chain but this thing which is called a directed acyclic graph and all of the nodes and all of the minors have let's say substantially the same data structure and from this data structure each node computes somehow the set of valid unspent transaction outputs and as long as all of these nodes uh have the same view on what constitutes the set of valid unspent transaction

outputs uh the currency works yeah that's a that's a great explanation I think yeah definitely so uh the the real trick here is deciding really what would happen right the the risk here is if if let's say uh you and Jonathan right Mar and Jonathan create blocks in parallel there's always a risk that somebody put in a conflicting transaction in these two blocks right uh May was supposed to pay Brian uh money but he also paid h m money the same money at the same time right and we can't have money going to two different people one of these transactions ended up in me's block one was in Jonathan's

block um and then later maybe if a block was created referring to the two of them we cannot accept both of them we need to Output a set that says only one of them is is there and the other is rejected um and we need to do it in a way that everyone agrees right that's the basic consensus problem um so so maybe the the most interesting thing to notice first is that the dag structure that we we built instead of a chain still has the the original chain inside it right if if you would ask just every minor to just point at a single block obviously he would point to one of the blocks that he heard of um when you have a list of them that one is included as

well right so if you really wanted to use the dag to just reconstruct the longest chain protocol you could have done it it you just you know the dag also has a chain inside it and the longest chain in the dag is also the longest chain that would be created if we were just running the Bitcoin protocol and not keeping a lot of links so in some sense this data structure just tells us more you we ask the minor tell us about more blocks that you've seen not just the longest chain tell us about everything and then we we later come in with the algorithm we could have picked the longest chain but the algorithm does fancier stuff and integrates uh the blocks together somehow and gets us a transaction set that builds upon all of the blocks instead of just the longest chain right

so that was the challenge um and uh uh the protocol itself Spectre is uh a little bit complex even uh to just decide what are the blocks but but I think we can walk through a few steps to get there um so maybe the first step to understand is that it's it's nice to have some relation on the blocks uh in the sense that if I have a block that defeats another block or precedes another block I can use that to determine whether the transactions of uh when when I see two conflicting transactions in these two blocks if block a defeats Block B in some sense which we we need to decide

what you know how we how do we decide if a block defeats another but once we've made that decision and we have two conflicting transactions in block a and Block B we can say the one in block a wins and is accepted and the one in Block B loses and is rejected right so the entire problem of um deciding which transactions made it into the Ledger eventually comes down to deciding which block defeats the other Block in some sense in a kind of a pairwise relation um and then what Spector does is also something that um somehow is a intuitively embedded in Bitcoin itself in the Bitcoin protocol it basically

takes a vote right it um when we have two blocks we need to decide if block a defeats Block B or Block B defeats block a so we basically take a vote we say who is in favor of block a who is in favor of Block B and we take the majority's decision right so what is the vote who votes here basically every other Block in the dag we consider them a voter right so uh when you created a block um you basically voted about every single block that came before you and that will even come after you you say do I think this block defeats another block yes or no right so Spectre is somehow you can think of it as a voting

algorithm in some sense that decides if block a defeats Block B or vice versa um so uh why do I say that this happens in Bitcoin you can think about the longest chain rule in Bitcoin right if there is a fork in bitcoin's chain the longest chain is chosen and a minor will build a block on top of the longest chain one way to think about it is this is a vote right so a miner looks at two of the chains he says I prefer this one to that one so he invests mining efforts and dedicates it to the longer chain that basically makes that chain a little bit longer that's like adding a vote to that specific option right so our intuition to think about Spectre in

terms of voting actually came from kind of understanding how Bitcoin works and you can also think of the Ghost Protocol that we mentioned before as a voting protocol basically saying which way should I go which block should I support this one or that one um so that is maybe another step towards understanding a little bit about how Spectre works when you read the Bitcoin white paper um Satoshi I think the genius of Satoshi was like sort of at one of the Geniuses of Satoshi was to just identify that like double spending is sort of the fundamental problem that needs to be solved right and in the Bitcoin protocol um we can think of it like the Bitcoin

protocol uh enforces Purity against double spends right at the at the data structure level so in in the Bitcoin protocol there's this one data structure the blockchain and the Bitcoin blockchain doesn't have two conflicting transactions like it's pure in the sense that there is there's never going to be two conflicting transactions in the Bitcoin blockchain as long as the protocol is working well now with Spectre you're sort of relaxing that assumption that the Purity doesn't need to be in the data structure the data structure itself can have conflicting transactions and like the data structure is being built as quickly as as the network can and it can have conflicting

transactions but once the the data structure is is built each node has a way of filtering out so whenever there is two conflicting transactions each note has a way of filtering out one and keeping the other and like as long as so so there's this method of filtering out right so you get a data structure and there's a method of filtering out the transactions that we want to reject like that they are double spins and we want to filter them out and as long as my way of filtering out these trans actions is the same as Brian's is the same as aives is the same as Jonathans is the same as Matthews around the world as long as our way of filtering the out

these transactions is the same will be in consensus and the currency system will keep working right so it's like it's you're removing the um the role of identifying double spins and removing them from consideration to sort of this this higher level uh uh which is like a computation that each node is doing rather than in the data structure itself yes we we can actually do a thought experiment what would happen if you let miners put conflicting transactions into the Bitcoin blockchain right so I would claim that it would still not hurt the protocol so think of

having a long chain and you have two conflicting transactions in there in different blocks so one of them came before or the other um what you could do is if you read the blockchain from start to finish you could say okay this transaction came first this one should have been thrown out so I'll throw it out I will filter it exactly the same way that you you just mentioned so Bitcoin basically says Okay since we're going to filter it anyway we might as well not write it in the blockchain to begin with just save the space and and avoid the trouble um so in some sense what if you we understand the core functionality of Bitcoin as just giving order over the transactions and if we have order we can filter them exactly

the same way and then Bitcoin says okay let's just not write them in which is a you know as a matter of fact very simple but in uh when you have block dags you are expecting to have a lot of blocks that were built at the same time by Honest participants they didn't see each other's blocks so it it wasn't a malicious thing maybe sometimes they include the same transaction twice maybe they include different ones that are conflicting um so you say okay they didn't do it on purpose let's just filter out the transactions in some way um so that that that realization basically says okay what we need is um you know ideally we'd like to have an order over transactions that would be the you know a very powerful thing but Spectre doesn't exactly give

you an order it gives something slightly different transactions um in different blocks might defeat each other block a might defeat Block B but it isn't an order because block a might defeat Block B which might defeat Block C which might in turn defeat block a so we might have circles in there that's a peculiarity of Spectre um but it still doesn't prevent us from outputting a a consistent transaction Set uh it still works right it's a little bit nuanced to exactly understand how but uh but the protocol still works in that sense so I mean it sounds pretty amazing right

if you can if you can do this thing and if if one is able to produce blocks at such an incredible rate and and not uh lose um security and uh also have this this great scale ability so I'm I'm curious what are the the places where this design is is a disadvantage why would one build uh you know something like Bitcoin you know if this had been known to Satoshi in 2008 should you have done for that or is would there still be some benefits to something like Bitcoin so I I think the main problem with Spectre if we want to put it right

up front is that it's very complicated right it's very hard to understand it's also very hard to code because of that so this is a problem with the protocol uh and we're working on nicer versions uh that should be I I think more understandable more clear to people um that's maybe a very big problem the second thing is about is maybe nuanced a little bit nuanced if you have a protocol that provides an order over the transactions like Bitcoin for example um you can apply smart contracts for example you can run ethereum on a s similar blockchain and ethereum really for ethereum it really matters if you

think about it the transactions in ethereum are really inputs to algorithm Ms and when you run an algorithm and you put one input before the other it changes the the result if you would switch them in the order right so ethereum is very um maybe sensitive to the order of transactions in its blockchain um Bitcoin is usually not sensitive to the order of transactions for example if I was paying Jonathan and Brian was paying May at the same time we didn't ites doesn't matter if you wrote your transaction first or I wrote my transaction first um we' still both get paid right order only matters in Bitcoin when we are when I am paying two people

with the same money when I'm double spending so usually I don't need to decide about the order exactly so what Spectre does is leverages that exactly as a property right if I if I can use money and most of the time order doesn't matter then maybe I don't need to decide the order sometimes it's okay to have these circular uh scenarios um and the problem in manifest itself in Spectre when I do a double spend okay so if I do a double spend Spectre isn't going to be sometimes I should I should maybe qualify this if the network is under attack at the same time by some powerful minor then maybe the double spends wouldn't get ordered

we wouldn't know if block a defeats Block B or if Block B defeats block a um so this sounds like a really bad thing but honestly it's not not so bad why because the person who's receiving money for me that I'm trying to dup by double spending sees both of these transactions okay they weren't with they weren't withheld it's not that he sees only one transaction he thinks he has money he really sees two of them one that I pay Jonathan one that I pay mayor for example and he says okay these these transactions have just not been accepted yet we didn't decide which one came first so we're delaying decision basically he looks at his phone or his app and he says okay there's a pending payment but this payment is still not

accepted it's still pending so if we wait a long time and the network is really under attack during this whole time this payment is still not processed so I cannot accept this as a payment that I've received um if you're buying at my store I'm not going to give you the product that you're buying um so so this mechanism if you think about it really just punishes the people who double spend on on the other hand if you're a regular participant you just use Bitcoin you just send it to somebody and you don't double spend um then there's no problem uh your transaction gets confirmed really really fast Spectre can really go to uh block rates that are maybe 1 second for a megabyte block you know as long as you

can handle the bandwidth um and then your transaction gets accepted within seconds uh we have I'm making an assumption here that we can propagate a megabyte block to all of the miners in maybe a second but this is what really happens uh today even in the Bitcoin Network there's a the very fast relay Network and fiber and so on that allow miners to connect to each other and have this infrastructure to relay blocks and even without it block propagation time was uh pretty fast um uh in the internet uh after all a megabyte is not a very large portion to send so what happens is that the protocol usually if you're a good player

gives you money quickly if you're double spending it might delay the transaction but it doesn't hurt the person who's receiving the payment and uh tying that back into smart contracts right Spectre is not as good if you wanted to run ethereum on top of it because if two people were interacting in a contract with ethereum then order matters here it's like inputs to a to to an algorithm and it's important if one of them speaks first or the other right and that of course depends on the algorithm like right if you're running some voting contract over ethereum and I vote first and then you vote and we we switch the order then the outcome of a vote doesn't change if we switch the order so maybe

this kind of contract would work but if a contract pays somebody money based on how fast he got there if he was first or second then of course order matters um so that's maybe the weakness of of Spectre it took a step back to allow less strict rules instead of giving order over transactions but it gains a lot in terms of of scale and speeds of confirming transactions let me just clarify one thing um following of of um description we do we the assumption that one megabyte of a block propagates in 1 second is not the security of the protocol is not contingent on this fact

so um user that observes the network and sees that uh in in the current Network conditions it to it takes a a block of 1 Megabyte 10 seconds to propagate will merely just wait additional time so it doesn't need to update all other nodes that the propagation delays faster that this this parameter is not hardcoded in the Spectre uh in the Spectre protocol that's the main separation may have perhaps referred to that the consensus is run locally in each nodes the order in of the of the dag or the almost ordering of the dag is run locally in at every node and the only thing that miners do is just mine blocks and tell us about all previous blocks that they

saw so this is important because again if if one megabyte you don't put embed this assumption inside the protocol this is a crucial point so that's that's super interesting like it's like with Bitcoin as as the underlying so there's an underlying peer-to-peer gossip Network inside Bitcoin right and the gossip Network can improve and does improve over time that blocks get sent between miners very quickly but that really doesn't translate into a performance Advantage for the Bitcoin Network as a whole so so today let's assume let's assume somebody figures out a way to propagate blocks from China to Iceland twice as quickly and that upgrade happens bitcoin's block time

will still be 10 minutes it'll still do a 1 Megabyte block number of transactions per second will be the same right but effectively what's sort of Happening Here is like as the speed of the network increases you can shrink the block block time even further without um without there needing to be some external coordination is is is that you what what you mean I'm not sure so let me let me make uh this uh clear there is um the protocol does um readjust its difficulty and does um um aim at a constant block creation rate and we can at a constant upper bound on block size so we can say

for instance it will be hardcoded that we're aiming at 1 Megabyte per second or or one block per second of 1 Megabyte Etc this will be um hard uh hardcoded in the difficult uh uh block block creation difficulty adjustment protocol but the the users when when they confirm transaction so let's say I'm a merchant at some uh point of sale and I observe the network so when I observe the network the time it will take me to confirm transactions will depend on the network conditions that I see so if I see that the network is healthy now and it propagates 1 Megabyte within 1 second then I will confirm a spectre

transaction within seconds however if I will see that there's some hiccups in the network and it takes 5 seconds or 5 minutes for blocks to propagate and for data to propagate inside the network then I will readjust my acceptance policy and I will wait further so there there is indeed there's no need for coordination when uh in the usual level when they confirm transactions and they will indeed enjoy fast confirmation time when that network is health is healthy and slow confirmation time when it's not okay but uh in contrast to that there is a a parameter in the protocol that says how many blocks and what size of blocks we create per second okay understood understood so um so give us an idea

about like what kind of scalability benefits does can this so this is a pure payment play right this this cannot be a smart contract cryptocurrency this has to be like a like a pure payments cryptocurrency that's uh that's super fast to confirm like I send a transaction it confirms very quickly and all around the world we can do many more transactions because like blocks can be created in parallel and they can be created much more quickly quickly so the advantages seem to stem from creating blocks in parallel and creating them very quickly but the disadvantage appears to be that smart contracts are not on the table at least today so can you quantify the um the

speed and scalability advantages like give us an order of magnitude estimate of what they could be well the order of magnitude depends on your assumptions on the network right because Spectre is not the barrier to to scalability then my answer depends really on the on the network what do you assume on the network so let me get into numbers let's assume nodes are able to support bandwidth wise they're able to support 1 Megabyte per second then we can create uh 10 blocks of 100 kilobytes per second or one block of 1 Megabyte per second or you know two one one block every 10 seconds with uh 10 megabytes we can we can use all these variations

uh and Spectre will remain secure so again the the the bottom neck is the network infrastructure not the not Spectre so it's more kind of a a network question now there are problems that that come up when you want to scale to thousand transactions per second um for instance node storage right so at a certain level you don't want nodes to be uh forced to store too too much transactions but these are network level um constraints and yes we want to solve them directly but Spectre is not it's not part of the game Spectre is secure anyways if you assume that these that there was like a putative Spectre Network that performed that had Network conditions similar to bitcoin when

Bitcoin is not under attack like like the normal Bitcoin Network today um like similar number of node similar number of mining nodes and in a sense like you are using the network bandwidth more efficiently in order to make the network go faster using this design right so could you give us some numbers there like what would be the scalability advantage there well I guess in today's Bitcoin Network conditions you can you can assume that nodes have 1 Megabyte per second um available to them um I I think storage will become a problem in today's

conditions because you do the Assumption you do the calculation I think it's around 86 86 GB per day if if you if you do 1 megab per second if I'm not mistaken and but that would that would that will go in the order of of 2,000s or more transactions per second um admittedly some nodes won't uh will be a push out of the pushed out of the game because they can support 1 Megabyte per second of bandwidth or or storage so there is a trade-off here because once you you you stretch the system to its limits then some nodes become out of the game are pushed out of the game so I would say 1 Megabyte uh in today's Bitcoin Network condition is a bit um

too high I think I would go initially for um I I'd say 100 kilobyte um yeah 100 kiloby per second I guess maybe if has different opinion yeah what what I'd like to say is you know I think you just need to accept that if you want a very large Ledger it's going to take a lot of storage right so obviously there's there's no free lunch here uh but having said that even bitcoin's current infrastructure supports not holding the entire blockchain in in your in your node right you could have special nodes that hold uh everything but it's okay if you just hold the last couple of days worth of data um so if somebody connects to you they can sync um so maybe onathan said 1

Megabyte per minute per second you might need a bit more because you you might be sending copies of the blockchain to several of your peers in the gossip network but but honestly with what is currently a home internet connection uh I think you can get a few megabytes per second it's very easy um maybe the the most important point is that what what Spectre removes is it removes the problem of block propagation time right so we can go up to the band with level you need to decide whether you want to do it right so Ju Just just maybe to put things in context right maybe the um the viewers aren't aware orist right Bitcoin does currently create a single megabyte block

approximately every 10 minutes that translates to three transactions per second if you go to a megabyte per second right a megabyte in bitcoin's terms can hold roughly 2,000 transactions if you look at blocks today that means 2,000 transactions per second instead of three or four that's three orders of magnitude right so there of course there are going to be a lot of engineering challenges in between there uh you know scaling up Bitcoin currently holds the entire utxos set in memory maybe you know when you scale up this large it will be hard but but these are not problems with the consensus protocol it pushes right so Spectre basically

pushes all these problems again back into the engineering that we know how to handle right a node can can buy a lot of storage right if you think about prices of storage how many how much does it cost to buy a terabyte or two every year not not a lot of money so even if you think about a megabyte per second you scale it up to a year so I I I happen to disagree with yonathan I don't think it's a lot of storage uh given that prices of storage are complet declining all the time and we're not talking about Bitcoin today right Bitcoin is not going to have 3,000 or 2,000 transactions per second today it's going to take years to get there um and it's going to take an engineering effort

uh to do so so what we really love to have is onchain scalability which is really massive um and uh while Spectre really doesn't do well with with smart uh contracts and so on there's no reason why on the same blockchain right or on the same block dag you couldn't have a smart contract system running in parallel inside the blocks and using something like the longest chain to to decide on its order it wouldn't be as fast smart contracts would still be slow maybe on this order of 10 minutes or so but they could live in the same structure right because we could just pick the longest chain so we could build

hybrid systems it's possible it's not very difficult to imagine how they are um so I think in that sense what we'd like to do is is is Spectre kind of gives us this ability to do payments really fast um I think we should grab that opportunity wow yeah this certainly sounds like an amazing uh amazing technology and some amazing things happening there now there are currently some projects in the cryptocurrency space that are doing work in this direction uh probably the best known of them is is a project called Yota um which is using this Dax as well

could you guys comment on on what they are doing and and how you think it compares the Spectre so I think it's a little bit hard to uh give a a good opinion of Iota um we've looked at the white paper that they wrote um they they have some interesting ideas for the protocol um but one thing that is missing for us to fully evaluate the paper is we're we we're academics so we're used to a very high level of formalism basically theorems where you state your assumptions um you prove that you get certain properties from the protocol right so if you if you were to go to

look at uh Spectre uh we we took our 90 page paper basically to to very carefully State what the properties of the protocol what the assumptions are on the network and then it's it's easier for us to understand if you know either the assumptions are wrong or the proof is wrong but if you accept the assumptions and the proof then this the end result has to be true ioda gives a protocol it doesn't uh do so very formally so it's very hard to analyze uh the white paper is also I guess kind of old so I'm not really sure what's implemented in code versus what's in the white paper um so we had concerns about I guess the white paper but it could would

be just that we're misreading what the protocol does because it's not stated formally enough for us um I guess other uh Works in other papers in the area sometimes uh I think have the same problem uh for us cool thanks that that's very helpful now now before Jonathan you mentioned that you guys are also working on on a foundation to to work uh on on this technology can you share a bit about that sure so um I'm founding now a new initiative the Paragon Foundation it's a nonprofit research and development

foundation dedicated to uh dedicated to implement Spectre and follow-ups protocols um so we we intend to develop the protocol up to testnet level um and um do simulations and and show its uh its improve it U performance and then we hope that teams will come um to the will join us and uh collaborate and take these take these uh protocols and Implement them commercialize them so all of our development will be up open source and we really are excited to uh to found this and it's really now uh initializing in these days and you should also mention that you guys are

not doing a crowd funding campaign uh because this is certainly what most people would do with with this kind of uh proposal or yes so um the the Ico Market is today a bit crazy and um so it's that's one reason not to go directly down this path um but another more important consideration is that Our intention is to um to be a research development project that's more dedicated to blockchain technology in general and not to specific uh protocol and Commercial product so if you you know if you develop one product then you will um you will defend its merits at all costs and

you will try to push this uh specific product and say you know this is this is the best solution out there and um and I hope you you're convinced now that we are more upfront uh about the merits and the uh drawbacks of Spectre so Spectre is just the first Milestone of the Paragon foundation's uh road map but we we intend to to um UTI uh Implement and deploy up to test level any um any research uh that's um um out there uh in the open source in the public domain that's the main long uh long run goal okay well thanks so much uh Jonathan and for coming on it was super interesting talk about that and I think

this is uh well we can't wait to do more episodes on this and hopefully also see some you know Spectre getting getting to an implementation getting test and all of those I think I think that would be very extremely interesting so for our listeners we we had like lots more to cover right like uh we had like lots of topics you want to cover like how would Spectre behave against Network partitions how would hard folks work and like things like that but I think we're just out of time so maybe we'll have you back again aiv and yatan and discuss some more when you have an an alpha test net running yeah that'll be great uh thanks guys for having us we know it's a very

complicated protocol uh people who want to learn a little bit more about it uh can go to a medium post that uh we wrote uh it tries to lay out the protocol and the different things that it does maybe it'll do a better job explaining there are some um nice illustrations that I hope will help people understand um and if anybody's interested in implementing besides the Paragon Foundation we've heard from a few groups that are interested in the protocol we're always happy to collaborate and help uh we're going to just uh we love to see our work out there uh we're not trying to commercialize anything ourselves nothing is patented everything is free for

everyone to use and we're very sympathetic to people who want to do uh you know who want to use our work I think it's the greatest thing an academic uh group can have right an impact on the world yeah absolutely so of course we're going to be linking uh to the medium post the we've mentioned and you know some of the academic articles we talked about in their website and all of that in the show notes so if you want to go deeper you can check that out and uh yeah thanks so much for listeners once again tuning in so we are part of the let Pon Network you find this show and other shows on light bitcoin.com uh and if you want to support the show you can leave an iTunes review for us that helps new people uh

find the show and thanks so much and we look forward to being back next week n [Music] [Music]