#219 Mance Harmon: Hashgraph - A Radically Novel Consensus Algorithm

Transcript

[Music] technology's partisan startup strategy centralization and the global blockchain revolution my name is Brian Furman train and I'm ahead right today we have a very interesting episode lined up for our audience we're gonna interview man's hormone of the project hash graph hash graph is pioneering a very unique consensus algorithm that offers something super interesting to the

blockchain consensus toolset man's welcome to the show thank you thank you for having me glad to be here so we're going to discuss hash graph during this during the episode but perhaps before we get into consensus algorithms hash graph and such complex topics tell us a bit about your background and how you came to be involved in the blockchain space sure well I have a deep tech background I started off my career doing research for the Air Force senior scientist for machine intelligence I worked on a team of five doing basic research and machine learning specifically reinforcement learning and the comm the the the combination of

reinforcement learning with neural networks convolutional networks back in the early and mid 90s before it was known as deep learning incidentally that's where I met my business partner and co-founder and creator of hash graph Lehman Baird dr. Lehman Baird we were on that same team working together for the senior scientist I also taught computer science at the Air Force Academy I was a course director for cybersecurity there at the Academy Lehman was also at the academy and was a full professor at the academy I didn't went off and managed a massive software program for the missile defense agency for the US government

basically a massive simulator that allow the government to learn how to protect its citizens and its allies from incoming ballistic missile attacks Lehmann and I decided we wanted to become entrepreneurs so we we we started our first company I left the military and it was an identity and access management company distributed single sign-on solution back in the days of the Palm Pilots around the year 2000 and installed that in 2004 went to work for the acquirer a fortune 500 and became the senior exec in the company for

product security stayed in that role for a short amount of time and Lehman and I decided to start our second company again in the space of identity and ran that for six and a half years sold that to private equity ended up going to work for a friend Andre Durant who's the CEO of ping identity I became the head of labs I stood up the labs organization for ping and also headed up architecture for paying in 2012 Lehman decided that he was seriously interested in this space of distributed consensus sort of independently of anything that was going on with blockchain and Bitcoin you know his vision required far more performant

and secure approach than what's required for just a cryptocurrency and so he went to work and he sort of Nod on this problem for years in in in 2015 he got to the point where he he'd solved the problem and it's what we now call today hash graph it was in that role at ping that this happened while I was at ping and so I decided to leave ping and started our third company with Lehman mine and Lehman's third company and it just turns out that ping happened to be a first investor in swirled the company that's commercializing hash graph the

technology and that's how we got to where we are today cool well you guys didn't have any interesting back ground that's a bit unusual from many in the blocking space and distributed leisure space I think more experienced and probably your average person in this industry well Lehman has been inventing things for as long as I've known them known him excuse me it just happens that what he invented this time was exceptionally good and at exactly the right time in the market right it's so you know is you know it's not unusual for for Lehman to come up with things there are a lot of

inventions I'm sure that we talked about some other time but hash graph is what we're pursuing today yeah cool yeah I mean of course we will speak more about hash graph later and how exactly it works but I do think it's interesting when people approach this problem of disability or blockchain or you know the thing that these technology is trying to solve without coming you know from having sort of gone through the sequence of Bitcoin and then next right because I think that kind of enforces a certain way of thinking um but still it was interesting to hear that Lehmann started working on this in 2012 so was he aware of Bitcoin back then did he kind of

follow it and did that in any way kind of shape is thinking they said okay I'm I'm not gonna do it that way or I see some problems there and that's gonna inform the design of hash crafts you you know kind of how that yeah oh no I I know exactly how it happened I mean Lehman and I this was in Austin Texas by the way Cedar Park suburb of Austin and there's a particular Starbucks about the corridor in Lima and I lived a mile apart there's a Starbucks on the corner and we spent many many many evenings at that Starbucks talking about the different approaches that he had had been taken and the simple answer is no I mean he did know about Bitcoin

but it didn't really inform his thinking at all because the problem set the use cases that he had in mind he understood very clearly and early on the Bitcoin would never be able to address them and so it wasn't as though he said well here's how blockchain works and here's what I don't like about it etc etc it was more like I have this you know clean slate to start with a sort of tabula rasa approach to coming up with an algorithm that can address this set of constraints that I have that are implicit in the use cases we care about and so it means clearly he

understood what was going on in the market for a Bitcoin but it really had no influence on the design of hash graph at all cool well not even though hash FCC so you know independent in its origination and kind of been developed in parallel you guys when you explain hash crafts often kind of analyze okay these are the existing type of kind of consensus systems in blocks and how they work would you mind kind of running us through how you viewed those different categories and yeah kind of your perspective on on the strengths and shortcomings of them sure yeah well so when Lehman had the inspiration of gossip about gossip you know that

enables this approach and we'll talk later about what that means it you know it had a unique set of properties in the market and what we did at that point was began to do a serious deep dive into the full range of categories of consensus algorithms to just explore how hash graph was different than all of those and it turns out that there are only a handful of categories now Lehman had done an exhaustive literature search and so he understood what these approaches where in the categories were but when you put it on paper and you start to you know to point out the differences the streaks and weaknesses you can you can more clearly see really you know the

power and performance of hash graph as it relates to the field and so the field is pretty small in terms of categories in the permissioned world you know hyper ledger and Korda from from r3 and etcetera of course they don't use blockchain they've replaced proof-of-work blockchain with what we'll call as a category leader based algorithms and these are represented by Paxos and raft and PBA pbft practical Byzantine fault tolerance and there are dozens of variants on on this category and it works very simply the nodes in the network elect a leader they send all of their transactions to the leader the

leader has responsibility for taking those transactions put them putting the transactions in an order and then sending those transactions in that order out to every member of the network out to every node the nodes of course just take them in that order and commit them to the underlying database in the same order and that's how they keep the databases in sync so this is ok it doesn't scale that well you can you know get to dozens of nodes you can not scale to thousands or tens of thousands of nodes you know in fact in the literature the most I've ever seen is about a hundred notes that scale in this manner the problem is that there is a leader and because there's a leader that's a

bottleneck these systems top out at about a thousand transactions per second roughly speaking if you look at IBM and you know hyper ledger and cocoa Microsoft cocoa and corn and others they'll they'll they'll announce or they've marketed the fact that they can achieve a thousand transactions per second they're all using this category and the problem is that there's a leader so there's a bottleneck in terms of scalability both in the number of nodes and the number of transactions per second but more importantly maybe more importantly is the fact that it's vulnerable to distributed denial of service attacks and nobody ever really

talks about this but each member of the network by design has to know the IP address of the leader or at least how to communicate with the leader and if that's the case then it becomes possible for any node that is compromised whether that be an insider threat you know a disgruntled employee or the notice being compromised by an attacker from from you know a foreign country etc whatever the case might be if the attacker can direct a botnet to do a DDoS attack on the leader then they can attack one computer at a time and stop the communication flow for the entire

network now what happens is that the leader the network recognizes the leader has gone offline and so they decide to elect a new leader that happens very quickly by the way you know within a couple of seconds normally but the attacker knows the IP address of the new leader and so they just changed the target of the attack and you just follow the leader with this DDoS attack so this is a fundamental problem in the architecture so the other problem is that it's not fair in a certain technical sense so there are use cases where maintaining the actual order of events as they flow into the network is extremely important for example maybe a distributed stock

market if it's possible for a single party to prevent bids or asks to flow into the market or to influence the order of those transactions to unfair benefit then then that's a problem right and and with the leader based system the leader has responsibility for exactly that for putting the transactions in an order and if the leader so chose they could drop transactions they could influence the order to unfair advantage and so in other words leader could be bribed so this category of use cases that require fairness is fairly broad

it's not just trading platforms it could be games if you had a an MMO and you know people in there in the game environment two people reach over to pick up the pot of gold at the same time the one that actually pushed the button first should get the gold it shouldn't be possible to influence that or auctions if you're creating an eBay a distributed eBay the person that pushed the button last should actually get when the auction and so in this so this fairness property is really important for a large group of use cases and leader based systems just inherently are not fair so that's the problem with

leader base category of course the public networks don't use leaders or most of them don't we're beginning to see some some of the public networks beginning to use some types of leader base technology but proof-of-work blockchain is inherently different all the transactions flow to all of the miners in the network the miners collect all those transactions put them into a block and then they compete with each other to solve a hard crypto puzzle this is the proof-of-work part of this and the miner that solves the puzzle first earns the right of publishing their block of transactions to the rest of the

miners that then gets put on top of their local copy of the chain the blockchain the miners unilaterally decide which transactions go into the block so it's isn't fair in the sense that they can prevent transactions flowing into the network and if they put it in the block they do this they decide the order of the transactions within the block and so it's not fair in ordering either and so proof-of-work blockchain in blockchain in general where there is a miner that decides which transactions go in the block and the order is not fair now proof-of-work blockchain is more secure than later based systems because

you can't predict which miners gonna win the crypto puzzle first right who's gonna solve the inverse hash puzzle first and therefore you can't do a DDoS attack against the network in the same way you can a leader based system so it's more secure it's not fair of course performance is terrible compared to leader based systems today a few transactions per second and extremely expensive when using proof of work and and so those are two of the major categories now in response to the problems of proof-of-work blockchain the market has come up with a new generation of technologies that try to replace proof

of work with other mechanisms we call these economy based solutions for the simple reason that the the approach is to make a fundamental assumption that the stagers in this case change the term miners to stay kurz for economy based systems thus takers are rational and they're gonna do what's in their own self-interest in other words they're gonna try to maximize the amount of money that they make so that's an assumption and then we come up with a system of incentives that when applied across all of the stakes will result in sort of an emergent behavior that the

stagers come to agreement on the blocks that should go on top of the chain in other words they come to consensus and their various ways of doing this but what's important is to understand that one the fundamental assumption that the stagers are rational is not a good assumption of course we we assume that those running the nodes the owner of this hardware that's running the node this in the in this economy based system wants to maximize the amount of money they make but if that node is compromised by a virus or a disgruntled employee or anything then that

assumption is is out the window it's not a good assumption the virus doesn't care whether or not the person that owns the computer makes money the virus only cares about bringing down the network that's the point of the virus perhaps and so it's not a good assumption to assume rationality as the basis of consensus for the entire solution number one and then secondly the whole approach is extremely complex in the sense that it's impossible to formally analyze it in the sense that we would formally analyze an algorithm and right math proofs certain properties about that algorithm the the community

is talking about proving formally that code is correct and that the code will do certain things for example it's a lot has been made that we can write proofs that certain smart contracts will result in the thus taker either winning more money or losing money once they've entered into the contract that's not the right question to be asking right we care about that but that's not what we fundamentally care about what we fundamentally care about is whether or not a single attacker can do something maybe unknown to us today but can figure out a way to attack the network and cause the network

to basically come to a standstill cause the network to not be able to come to consensus that's the math proof we care about we need to know for certain guaranteed that it's impossible for a single attacker to prevent the network from coming to consensus and those proofs don't exist for the economy based model and they don't because the system is complex it's chaotic it's an economy it's a model of an economy and so for the same reasons if it's not possible to write a a formal proof that for example a stock market will never crash again because it's too complex and it's a

chaotic system you can't do the same for this category of of approach these economy based solutions so those are three right leader based system proof-of-work blockchain economy based solutions the question then is well what can you do formally I mean we want to guarantee the best security possible for the platform if this platform is going to be processing trillions of dollars of value that's fundamentally important and we want it to be extremely fast we want to be able to scale up way beyond orders and orders of magnitude beyond where we are today so what do you do it turns out there was there is a another category we'll call

pure voting based algorithms they go back decades 30-plus years and they work in the following way let's assume perhaps we have 10 nodes in our network and if I want those nodes to agree on the order of transactions I can ask those nodes to vote on a question that's related to the order and what that means is that each node I ask that each node the same question each node will cast their vote and when they do so they send it to every other node in the network and then when a node receives a vote it sends an acknowledgment not just back to the sender but again to every other node

in the network and then there may be multiple rounds of this voting process that's required for those nodes those tend to come to agreement on the order of transactions and if you have an a you know a a stream of transactions it becomes even more complicated the point here is this the amount of bandwidth required the amount of traffic that's flowing just to accommodate the voting process just explodes and it explodes in the number of nodes and then the number of transactions and so for that reason this category is not practical at scale and and no peer voting based system

exists at scale today however this approach has phenomenal security properties these pure voting based systems have been shown to achieve the gold standard in terms of security provably in the mathematical sense that I've been describing and that standard is called asynchronous Byzantine fault tolerance everything else prior to this point has been something less than that and only these pure voting based systems have been shown to achieve asynchronous B of T but they've not been able to achieve it at scale so those are the four leader based systems proof-of-work blockchain economy based

solutions pure voting based systems that are theoretical and don't scale the question then is how do you achieve the properties of these pure voting based systems yeah well maybe let me just like jump in here a little bit please because I feel there are a bunch of stuff you know maybe I don't fully agree with so you know for the last years as I'm sure most listeners know a bit more than that I've been working on a cosmos and tenement and of course so what I mean is is essentially like an implementation of people so it is has kind of these voting based properties right and proof of stake so I think games theoretically it's extremely safe in terms of

performance I know they have run we have done some benchmarking with like I don't know 100 nodes there's a few hundred nodes and something like 10,000 transactions per second and now and and again if the leader base so now there is a validator that proposed each block why today is kind of a leader and it is of course an interesting question like how can they be attacked I think DDoS is ism is something that can be mitigated I think you thought that the point you made about fairness is a more fundamental one and that seems to be I can see that being a massive issue and of course we are gonna gonna

speak about and I think what is where you make good point is that even though I think the system seems scale pretty well up to a point right and the point maybe a few hundred validators something like that but you know it will not be you know you won't be able to do ten thousand violators right like that it seems clearly infeasible or even a thousand you you know you performance will get much worse and of course if you only have hundreds that may be enough if you're sharding right in a single shard it may be sufficient to only have hundreds of validators to to ensure security of the system well I mean my my fault on this is actually a hundred

violators perfectly fine if those are like you know independent running their own separate system there's some sort of eek is reasonable distribution of power I mean we look at a Bitcoin today and you know it's for many times it was maybe three miners or three mining fools have actually a majority yeah and and that's you know concerning even though so far it's work so if you have a hundred in this reasonable distribution then I think that's fine of course the question is always okay who are always valid is what are their incentives how much power do they have how are they associated if if what years you know you kind of made

the point before Oh someone have a longing a virus and bring the system down well I think then there's question is of course if they want all the same hardware the same and the same setup then yeah that could be a risk so diversity in their set up as well well certainly there's that but it's you know it's still the question of is there a way to bring down the network period and unless you can so what we always have done throughout our career our careers Lena and I is start with first principles right start with the fundamentals and as and and start with the math and create the best most solid foundation possible and once you have

the best building block then the building you build with those those blocks turns out to be fundamentally different in some meaningful ways then if you start with something that's not quite as strong you know has a flaw in one way or another or can be attacked in one way or another because these vulnerabilities compound right and so for example just to make the point what Lehman has designed is a way to scale this you know a sharded system a protocol for shardene that maintains the

asynchronous BFT nature across the entire system right now this is not yet public it will become public in the coming weeks and months but you know in most cases when you shard that it's it the security of the system as a whole is degraded in this case because we've started with asynchronous BFT as the building block it turns out you're able then to create a sharted solution that system-wide is asynchronous bft and then I would say just one other thing there are a number of different platforms out there that are experimenting with a combination of these different architectures and and they're doing so because they need to

achieve scale and they want to achieve high throughput and etc etc etc so they're doing it for good reasons and I can appreciate why they're making these combinations the problem is that when do you do create the combinations you may improve certain aspects but you inherit the vulnerabilities of all and so the vulnerabilities associated with leader based and the vulnerabilities associated with proof of work blockchain or economy based solutions if you combine them you have then a you know a larger set of vulnerabilities to defend against than if you stay pure to one of the categories so before we move on to hash graph I I'd like to know sort of the

history ends of genealogy of voting based protocols so like Leslie Lamport and Shaw stack they defined it by Byzantine generals problem in 1982 in a paper right and they define the problem as being the city and there's a set of generals around the city and these generals must coordinate an attack at the same time yet some of the generals are probably corrupted and will try to make sure that part of the generals attack at one time and the other part attacks at a different time so you have traitors inside this group of generals

but despite nice traitors these generals must come to an agreement on when to attack and Byzantine fault-tolerance like the Byzantine generals problem stayed and an open problem for for quite a while until algorithms that were based on leader election and voting followed by leader election were what kind of proposed so the idea would be one general would become a leader that general would propose a time to attack the other generals would vote and then once enough generals vote affirmatively you could like sort of collect these

votes and attack at that time and the problem there also was that none of these protocols in a computer science algorithm like in the performance ends were performant and finally finally in 1999 in in MIT there was a there was a PhD thesis and paper published on practical Byzantine fault tolerance I think it was Castro Miguel Castro that published that paper and practical Byzantine fault tolerance showed for the first time that in a leader based system and there's a leader and other people voting on what the leader proposes that

that can be made practical and turn it into a computer system that can be used to vote not just on one piece of data like when to attack but on thousands of transactions so in in in the in the family of leader based algorithms you see this pattern like there's an idea it's not practical for a decade or decade and a half and then some person comes and makes it practical in in in the system and at some level with hash stuff what you're saying is there's another family that's the voting based algorithm that's different from this genealogy this part of the consensus tree and that has existed as an academic

idea and what you're doing is you're making it practical can you point us to like like who who came up with this family and like do you know what what are they large what are the important papers and in wording based algorithms yeah well so thank you for the genealogy I think you've done a better job there than I could have done off the cuff Lehmann thank you if that was good that it was great so I don't I can't tell you off the off the cuff what the papers are by reference but I certainly could get the references the the starting point is the same right the pure voting based algorithms that go

back to the 70s and 80s is the starting point and then there's sort of two trees if you will that that would branch one is the the paxos and raft and PBF tree that you've mentioned and by the way they the whole ddos set of attacks and problems that are associated with this category there is a phenomenal paper again I can get you the reference and you can provide it you know when you when you publish this is deal specifically with pbft and and how it's vulnerable to these various types of timing based in DDoS attacks but so so that's a branch and then there hasn't

been another branch I mean there is no there's no in between paper from the 70s to hash graph you know hash graph as an approach is brand-new and what makes that possible for the very first time is this inspiration of if you're gonna gossip information and you know there's sin transactions across to everybody that you can do something special a little bit special there that makes it possible to use this internal data structure that you build up that is the hash graph and then combine that with the use of a peer voting based algorithm without ever having to cast the votes that's entirely new and it's not as though they're steps from the 70s to

2015 there's one step from the 70s to 2015 and that was Lehman in the hash graph so I think you just kind of summarized hash graph but I'm pretty sure nobody would be able to understand him from that so so you said right to gossip so we have two gossiping ups transaction that then kind of a data structure emerges that that is called the hash graph and that that can somehow be combined with this voting type of approach approach to create a consensus agha and so maybe it can be can unpack that exactly what's going on here first of all gossip I don't think many everyone will be kind of it will be

clear what what that means can you explain what does it is well gossip protocols have existed for decades as well it's a common approach for sending information to a large population very efficiently over a data network and the idea is you know it gets its name from what you what you observe in the work place or in your social circles you know one person tells another person a rumor and then that person tells somebody else the same rumor in the room are just you know flows through the population exponentially fast Alice tells Bob the rumor and then you know at time T plus one Bob is telling Edie and Alice is telling Charlie the rumor etc it close

to the population exponentially fast that is a gossip protocol and you can implement that in data networks and the question then is what is it that you are gossiping about and you can gossip about a lot of different things you can gossip identity information you can gossip transactions to sort of opaque arbitrary transactions that are only understood by an application that sits above what we do is we gossip about gossip and that's gonna be a little bit hard to wrap your head around initially every node in the network has a local copy of a database and the goal is for all those databases to stay in sync some people call them a

ledger right have the ledgers stay in sinc if if I write a transaction if I create a transaction and I want to update my ledger that transaction has to go to every other node in the network so that they can update their copy as well so all transactions have to go to all Ledger's that's the minimum bandwidth required for this gossip protocol what we do is create a graph that memorializes when people talk to each other when nodes talk to each other and so if Alice talks to Bob Bob creates a circle that goes into this graph if you were to draw the graph you you know it's

got lines and circles Bob creates a circle that memorializes this syncing event between himself and Alice and and as part of that circle Bob creates some transactions that that would be understood by the application and they're the payload that then go in this circle all right also in the circle is encapsulated to timestamp and a couple of other hashes and and I'll get to what those are in just a moment but that's it you know there's a data structure that represents this circle and then the data structure is a payload of events a timestamp and into hashes and that gets signed that data structure gets signed

and maybe just a sort of biggest ones up here gossip protocols are being used by existing auctions like Bitcoin or Hume or others and basically used to again like propagate transactions fill up the mempool propagate blocks right but do not used in the consensus right there kind of like separate networks or in Bitcoin I think it's even something some like a relay network between miners through to like propagate blocks faster maybe does that use also some kind of gossip protocol probably they're used everywhere I mean gossip protocols as a category are used all throughout all kinds of applications across computer science so yes that's a very well

understood approach and we're certainly not unique in the fact that we are using the gossip protocol what were you Nikken is what the what we're gossiping you know we're we're gossiping the transactions as part of that event but we're also gossiping to hashes that represent the last event that was passed to me by somebody else that event with its package of transactions gets hashed that hash goes into my event this new event and the last event I created it gets hash in its hash goes into my event and so this event with all of the the different data items that I've said plus

the two hashes that link back to two prior events that then gets gossiped to the network and when members of the network received these events through the gossip protocol they can by using these hashes create the hash graph the hashes link it make it possible to link together these events in a certain order that creates the hash graph and provably each node in the network ends up having the same hash graph they're identical at least at through a point in time they're identical and so new events are always flowing in on top you know you're always gossiping and talking to other members of the network and network just

runs as fast as it can then at a moment in time the it becomes the case that all the events prior to that moment in time are known by all the members of the network and you agree on the hash graph so that's important that's the gossip about gossip that we're talking about then and the inspiration was that if we do that there is enough information there that we can take a pure voting based algorithm and and use the information in the local copy of the hash graph and instead of sending votes over the network you know asking each member of the network to cast a vote on

a certain question we just locally say for this question what would each member the network vote if they were to cast the vote in there there's enough information in this hash graph to answer that question for every member and so it's gossip about gossip with virtual voting you never have to send a vote over the network and and you're done that's it and it maintains the properties of the pure voting based systems in the sense that it's asynchronous bft it achieves for the first time ever the gold standard in security as it relates to distributed systems and it does it at

scale and also in addition because you have to send the transactions to everybody that's the baseline and all we're adding is two hashes to that event that message that's being sent we don't think you can do any better in terms of bandwidth efficiency so we've achieved the the best one can achieve in terms of bandwidth efficiency and we've achieved a synchronous BFT for the first time that's why we make you know I realize it's a really bold statement but that's why we make the claim that Lehman has actually solved the problem of distributed consensus at scale with the hash graph like it is actually pretty pretty easy to imagine the the

contribution of hash graph right like we can Hawken back to that imagination that there is a city and the generals gathered around the city so it's a 50 generals the three of us are are are part of these 50 so we each have our individual armies Brian Mane's and me and we are part of this group and the group of 50 generals we must attack together and we have to sort of agree on a time and let's say not only do we have to agree on our time but we have to agree on a sequence of times so maybe if you have to agree on like when will we fire the cannons when will we deploy the artillery and like there's a sequence of

times that that we want to agree on and if we agree on these sequence of times correctly then our attack will happen flawlessly if if if you don't agree on it or some of us actually on one sequence and the other is another sequence then our attack can fail and then there are some generals inside the camp that traitorous and they they want half of us to attack through one plan and the other have to attack through what the plan so like what is Bitcoin like bitcoins the idea that hey all of us are going to solve puzzles and one of us is going to win at solving the puzzle and then the person

that wins at solving the puzzle will announce one time to do something like Oh fire the cans and then he's going to solve the puzzle and like broadcast the solution and the time of firing the canvas to all of the other generals then all of the other generals now all of us will start working on the next problem and then one of us will solve that problem and that the solution to that problem is additive on the solution or to the first problem and that the second solution broadcast the second time to do something else like deploy the artillery then a third round will happen all the generals will solve a different problem again one of them will solve it and so

on so that's Bitcoin effectively a leader based a leader based system is one of the generals let's say me will become the leader and I'll say let's fire the cans at like 12:00 p.m. and I'll sign this statement and I'll send this message to all of the generals through my messenger so generals are connected to messengers like delivery boys that can go and deliver the message the other generals will like receive my message ins and like vote affirmatively right ok yeah let's agree to attack at 12:00 and then these votes are then sent through delivery boys to other generals

and if at some point I as a general receive a message with enough food saying let's attack at 12 I accept that as the as the car time to like you know fire the cannons or whatever and then somebody else man's becomes the general that proposes the time to let let's deploy the artillery all of us vote on it and then finally if there enough votes we agree on the time to deploy the artillery then Brian let's say creates a message okay let's send in the cavalry at this time and then we vote then we decide on when to send the cavalry so that is how like a leader based system will work and

me like hash graph appears to be in both both Bitcoin and a leader base system this thing that is getting exchanged between the generals is data about about the strategy itself like at what time should we do what but enhanced location level information that's the application level information right like what should we do when we're as in hash graph what you're saying effectively is what if in addition to application level information when to when to do what we also record the sequence of what the

messenger boys have done in order to send these messages across to the generals so I am a general let's say I say okay attack at 12 and I send it to manse manse receives that messenger boy num in man says later it says okay I received a messenger boy from from my hair that told me this right then you add information about what you think like I think not only should we deploy the cannon set well but we should deploy the artillery at at 1:00 p.m. and then you also put in some past information that when was the last time you sent a messenger boy out to some other general and you've compiled all of this into a

packet and then you instruct your messenger boy to go to another general and when the general receives your messenger boy he was going to create another data packet but include the behavior of your messenger boy in his data packet and so once sort of these these messenger boys end up ends up delivering and and you start to collect information about who proposed what and how the messenger boys worked after a certain point in time everyone will agree on who proposed what and what the messenger boys did and I based on that they can

come to the conclusion on when to you know deploy the cannons when to deploy the artillery something like that that's it and that's it's great that you use that fundamental analogy for for the different categories and and that's right I mean that sort of demonstrates when we say we're gossiping we're not gossiping transactions of course the transactions go along for the ride as a payload in the event but the event is describing with the two hashes the you know who we talked to last effectively right it's it's the it's the hash of the last event that was given to me in the house or the last event that that I created and so it's the gossip about

gossip the metadata that's on top of the transactions that's important in creating the hash graph and that's important also you know it's interesting to me that hash graph as a term is becoming increasingly used and you know there are a lot of there are a lot of platforms or an increasing number of platforms in the market today that are based on DAGs directed acyclic graphs and of course hash graph is directed is a directed acyclic dag graph scuse me as well to dag the fundamental difference what makes a hash graph a hash graph is what that dag represents and in our case

it represents the gossip flow the flow of the communication across the network as opposed to the transactions that's how we're fundamentally different than everything else and and and and we define hash graph to be a dag that represents the communication across the network I mean listeners will probably have some familiarity with dak and dak based systems we did do a podcast about this before about a something called Spectre which is a paper by two Israeli academics on this topic now that is also seems very interesting so what why is

this gossip about gossip so fundamentally different from like having a dark base structure that has transactions or blocks well the informations not there to to calculate the votes you have to know how Alice would reason about the information Russia received from Bob as opposed to you know if I just sent Alice a transaction and I don't tell her anything else she doesn't know where that transaction came from and she can't reason about where I received the transaction from in the first place right all of that information is lost if all you include are the transactions you really need to include the information that represents

the flow of the transactions across the network in order to be able to to to use the voting based algorithms right right now that makes sense something that's very well put right so it says in essence you could kind of think of it like this right what you guys are doing is like almost like continuously uh that all the nodes kind of update okay this is this is what I am seeing and that gets kind of all adjusted and so of course they can say okay now I'm I'm using these simple rules so say this is the order of transactions and I know what what everyone else would do because I have kind of synced up on what information they've seen in order yeah so everybody when I talk when alice

talks to Bob for example she has a graph a hash graph Bob has a hash graph they're identical up to a point and then Alice knows some things that Bob doesn't and vice versa and they share with each other the Delta what Alice knows she that Bob doesn't she gives to Bob and vice versa at the end of that sinking of vent their hash graphs are synced in other words Bob now knows everything that Alice knows and vice versa and they can reason about where transactions came from who talked to who and win and because everybody's gossiping with everybody all the time the local copy of the hash graph is always being brought into increasing sync across the network

and at a moment in time they are in sync and they will forever be in sync from that moment in time so I think that actually brings up an interesting point so there's a term finality right which is used in for example in things like tenement right where every block is final right so you know okay block takes like three seconds then its final in proof-of-work systems that's much harder so one doesn't never really have finality zone one uses T is like quasi map finality people are probably familiar that in Bitcoin for a long time you know or maybe still two to a large extent one would kind of say okay six transaction or six confirmation so once it's transaction six blocks deep you

kind of consider it final but you know it's not exactly the same thing and then in aetherium as well with Casper I think there would be some kind of finality coming but it will take some time to get there what does it look like in Ashcraft at what point you know how many seconds does it take until you can say tile and how does that depend on you know the number of nodes that are participating in this so what I can tell you are sort of what we've seen empirically and then explain it so a information that's gossiped into the network it's it goes out to the entire network exponentially

fast the amount of time it takes to get to every node is logarithmic in the number of notes and and so that's an important point the amount of time it takes information to get from from the originator to everybody else is logarithmic in the number of nodes in terms of finality what we see is that on our courage it takes about eight gossip periods I'll call that a gossip period it takes about eight gossip periods before the transactions are final on average it can be more it can be less but on average it's about eight gossip periods before the transactions are final

now finality in our case is fundamentally different than saying proof-of-work blockchain and proof-of-work blockchain you have a partial order on the transactions you care that you know what you want to know for sure is that that two coins I'm sorry that that a person can't double spend a single coin and so if Bob tries to spin that coin in two different places double spin the coin then the partial ordering of those transactions to ensure that you can't do that is all that's important in a total ordering system every transaction gets put into a total

order across all transactions for the entire universe that's a much harder problem to solve and so we solve when we say finality what we mean is that we have a total order on all transactions across the entire universe and that becomes really important when you start thinking about treating this like a database you know if you literally have databases underneath the notes then you care about the order of trends of every transaction and and you have to have a total order to ensure acid compliance across all the databases and and we have that the other thing that's different is that when we achieve finality we're a hundred percent sure of the fact that

it's final and we know for certain that every other member of the network knows for certain that it's final and it will never change guaranteed so it's fundamentally different than the Bitcoin for example where you have blocks and with each new block on top of the chain you are probabilistically more certain that the that the you know that the coins not going to disappear if you're the merchant for example and so we're you know we achieve a higher level of certainty or finality than what's currently achieved in in Bitcoin and that's also you know part of the

definition of BFT just being BFT you can't be BFT unless you have 100% certainty that at a moment in time everyone agrees on the order and the order will never change and so Bitcoin blockchain is not BFT by definition and so you were speaking about eight gossip periods and and how long it's like let's assume we had now a public hash graph network sure sure so it fits so empirically what we've done in the lab and we've done lots of tests both in a single data center cross country you know using AWS for example from Oregon to Virginia across data

centers across country and then globally and what we're able to see in those contexts with a varying number of nodes we haven't achieved hundreds of nodes yet we haven't tested hundreds of nodes yet we'll achieve it we just don't have the quota from Amazon to do hundreds of those yet is that you know it depending on the number of nodes and the Geo distribution we go anywhere from sub-second finality total order finality with a hundred percent certainty two seconds of a finality and it may end up being that it's you know tens of seconds finality on the order of transactions when we go global at scale it remains to

be seen we're still we're still doing that work so of course that kind of already gets us a little bit into a topic that I'm sure many people will be curious about which is you know is there gonna be a public hash graph network and especially you know how would that look like in a public chain context now we're speaking about notes and this cost of gossip so presumably let's say if I could spin up lots of nodes run lots of nodes and kind of gossip fake information softer I mean I presume I would be able to disrupt this process you know up to you know how is that security well the real question you're asking is how do we go from a permission

that work where there's one vote per computer and you have to give permission to a computer to join the network to an open network where you can't have one vote per computer because to your point you know somebody can stand up a bunch of sock puppets and you know if they stand up enough then they can have you know two-thirds of the total vote which would dictate the order of transactions you in all cases well you know whether it's hash graph or leader based systems or any any of these distribute consensus algorithms what's required to be public or open is a scarce resource you have to make it expensive for a would-be

attacker to stand up all those sock puppets or you have to make it impossible for them to you have to make it impossible for them to to have enough of the voting weight if you're waiting votes to be able to influence the order of the transactions and so a public network built on hash graph would be the same you know that's why the crypto currencies are actually required in the public networks a lot of people view the crypto currencies as sort of the point of the public network that's not how we view it I mean if there were a public network that's built on hash graph it would be to make it possible to build globally distributed applications and it

turns out you have to have a scarce resource and the cryptocurrency serves as that scarce resource and so it certainly would be a feature but it's a necessary feature for the bigger vision of globally distributed applications and and yeah there there would be a cryptocurrency associated with that and there would be a way of using the number of coins that perhaps one owns when they run a node is a mechanism for weighting the influence of their vote on the consensus voting process and so instead of it being one computer one vote it's one coin one vote and then there's a

whole set of issues that you have to deal with if you take that approach which which I'm not prepared to go into today certainly we will you know we would explore that in great detail you know when that becomes appropriate for a public network so that's very interesting of course that makes perfect sense I think there aspect of having kind of coins or steak or something like that that raise votes and protects from civil attacks you know that I think that these were where many of these networks are going and and that seems very reasonable one thing I'm curious here would in such a scenario would you need a block reward

or some other kind of incentive ization for people to participate in this or it's not not well yeah no absolutely I don't think there's a free lunch and a lot of the systems that we see today really don't account for all the costs that are associated with being a node and I viewed those as flaws in the economic underpinnings of of those platforms so for example you you know if you're running a full node then you are spending money on bandwidth you spend money on storage and you spend money on CPU cycles and I think all three of those a full node would be you would want to compensate a full node for the

the amount of resource that they provide in each one of those categories and so if you're going to have a public network at scale and it's open consensus it's you know anybody wants to run a node can run a node then you have to have a system of economics that actually counts for all of the costs in order to be viable at full scale at you know at maturity and yeah certainly that would be part of a public platform so of course immense haschke looks like a very interesting consensus algorithm and deployed in a public setting could could make an interesting asset or or coin on however we might

choose to see it my question would be what what other applications mine it's a two-part question first part is what other applications are you deploying hash graph to in the short term apart from potential public system and the second question is since you you and your team are approaching the problem of consensus from from a very different perspective to people from the blockchain industry itself how do you see the blocks in industry evolve if your consensus algorithm proves successful interesting okay well to start we yeah we started by addressing

enterprise use cases right I mean that's where this world's is is focused as a company and the the first use case ping identity I'd mentioned that I was a head of labs and architecture for paying the first actual application built on top of hash graph was an identity focused application that actually saw some sort of an obscure problem in the world of identity it's something that associated with the protocol code called open ID connect Oh IDC and peeing has then taken that and proposed that to the OID F the standards body associated with this as as a way to address the problem in the

standard and they continue to pursue that even now that I'm going we then approach the credit union industry and the credit unions as an industry wanted to create a platform that makes it possible for the 6000 credit unions in North America that represent 105 million members to build arbitrary distributed applications I mean not just Fiserv related applications financial services related applications but in addition to those messaging applications information exchanged as it relates to risk for example of course identity related applications and and they've now

created a new organization called Cu ledger that stands for Credit Union ledger the industry CEOs voted as an industry to create a new service organization Cu ledger and Cu ledger now is capitalized it has a CEO CEO came from MasterCard a senior exec from MasterCard and they this year are rolling out this new platform built entirely on on hash graph and there will be an app store they have an existing you know robust third-party developer system that already builds applications for the industry well now they'll be able to build distributed applications and market those through the App Store as a channel to the six thousand credit

unions and so that will be the first large-scale deployment of the technology and that's happening even as we speak and then and then there's you know a large pipeline of other customers that are cross industry there's healthcare there is supply chain management there are other financial services organizations that are using the technology none of these have been yet made public and they will be there near they're less mature and then Cu ledger is but you know they're coming and you'll see those this year and and so that's you know that's what swirls has

been focused on as an industry on the public side what I think will happen you know with or without hash graph and I think hash graph maybe will accelerate this in some ways is that the stack of technologies that are there to deliver the product the the elements of that of that stack will become increasingly differentiated I mean we start with Bitcoin it's like one monolithic code base and in some ways it's a mess what we would like to see and what I'm sure we will see as the industry hers is that each layer of the stack for example with hash graph what we have is

a consensus I think of it as a consensus server maybe server is not the right word it's an SDK built in Java entirely in Java that handles communications across all the nodes it implements that gossiped protocol that we've talked about and puts arbitrary transactions and consensus order so that is it that's all it does that's the foundation that's you know the the consensus layer the bottom of the stack if there were going to be a public network then there would be a whole range of services that go on top of that layer that can sense this layer there be a services layer and you know one of those might be a

cryptocurrency another one might be storage another one would be smart contracts etc etc etc anonymity services what you can imagine what kind of services you might want I could see and expect that there will be competition for providing Best of Breed in the market for maybe each one of these services and for the consensus layer itself and that's how markets mature and and so we you know we will certainly hopefully accelerate that process we have our own ideas about how all of this technology should work in a mature well architected stack you know that serves all the use cases of the industry and I

think that the industry will you know come to agreement on on what the definition of those pieces are and you'll see companies spring up that do one of those things and they do it really well that's what I would expect cool well maybe one last question here what's the timeline of hash calf where you guys have currently and what's the road map for the next you know maybe 12 to 24 months yeah well so the the work that we are doing on that consensus layer that I just mentioned is coming close to version 100 bill 'ti that work has been going on for years this is a complicated thing so hash Graf

in some ways is far more complicated to implement than blockchain is to implement and you know making a gossip protocol that is highly performant is a work of art and and so lots gone into that so we will continue to harden that can sense this layer and then even with or without a public network it's still important to have that services layer that I just described all of the services I mentioned have value even in a permissioned context right and so you know we will build out that services layer and everybody wonders if there will be a public network it's fair to

speculate that there will be you probably would be right and it's so you know we're not making any announcements but we're doing all the things that anybody would would do in our situation ok ok I think that that is the most elegant and non announcement announcement that I've heard that's right that's a non announcement yeah I mean you guys certainly have already published a lot of material about about how sheriff writing in detail technical papers and comparisons and I mean that's great there's always a lot of substance there to dive into and there's also some nice nice interviews nice talks that are

available well I think we're at the end of your episode thanks so much man's for joining us today it's been a pleasure to learn about hash TRAF I'm super excited to see what what's gonna come out of that you know whether your claim so true and it's gonna be as revolutionary I don't know but it's certainly original and it's certainly something truly novel with interesting properties so I think that that I'm you know I'm thrilled about I'm sure mayor and many others are thrilled about to see what comes out of that yeah thank you now thank you so much I mean it is different than anything else I've seen in the market we're very proud of that fact we don't want to look like anybody else

and so our approach has been fundamentally different as well and it's all a you know very staid well-thought-out mature approach to both the technology and in the marketplace so we're looking forward to it as well thank you for your interest yeah and thanks so much for listening for once again tuning of course we're gonna put links to to the hash of white paper and many of the other materials website in the show notes if people want to learn more about it and dive deeper they can go there and yeah thanks so much for listener for tuning in so we put out episodes of episode every Tuesday usually although sometimes a little bit later and you can subscribe to show and

iTunes SoundCloud your favorite podcast application or you can watch the videos on youtube.com slash every sooner Bitcoin thing and you can support the show by leaving it us a review on iTunes or you can of course send us your tape in Bitcoin pick on cash or Easter so thanks so much and we look forward to being back [Music] [Music]