Patrick O'Grady: Avalanche – Building High Performance VMs With HyperSDK #506

Transcript

hey everyone today we are talking to Patrick O'Grady who is the VP of engineering at our Labs official we will be chatting about the hyper SDK which is kind of one of the ways by which the Avalanche network will allow Builders of evidence subnets to have very good performance essentially on on the subnets that they build so we'll get that we'll get deep into that topic and then if time permits we will also kind of understand the messaging protocols the inter-subnet messaging protocols that would be available in the Avalanche ecosystem so welcome Patrick

thanks for having me excited to be here I was telling uh these guys before the show that you know epicenter is one of the first podcasts I started listening to you and I first got into crypto so many years ago now but it's been great to you know follow along as uh all the cool guests you guys have had on it really uh honored to uh have a chance to hang out to you guys so Patrick how did you end up being VP of engineering at our Labs official building hyper SDK yeah good question so um I I actually grew up far away from all this stuff I I grew up in Wisconsin uh you know I didn't grow up on a farm per se but I I certainly didn't grow up in the in the big city

um and uh you know I had a had a great childhood there like a lot of stuff and stuff and then I watched this show uh when I was growing up called Chuck if you ever heard of it it was like on NBC in the United States and it was about this guy that like got kicked out of Stanford got like a computer downloaded into his head and became like a CIA asset or at CA NSA asset and but the part was he got kicked out of Stanford which is like this alluring place to go to and so I got this itch like maybe I should try to go to Stanford and so uh you know one thing led to another unfortunately got in got into Stanford and came to Silicon Valley and haven't uh haven't left since but while I was here um I got first Jason to crypto was

actually at Stanford there was a Class A long time ago taught by biology actually uh who had the 21 company where it was like connecting Bitcoin web services to to like a web like you basically the whole class was like building different things it hooked up to the 21 minor so I actually keep uh like 21 minor has on my desk as a reminder kind of like the trajectory of where we've all gone but it was all about like accepting micropayments and everything um on the web and so that really is what started things but you know spent a while dabbling around in college on different crypto things but ended up working at coinbase uh after college um and so while I was there I worked on

a project called Rosetta which is like a universal read and write uh interface or open source specification for any blockchain and the idea there was you know coinbase wanted to list as many assets as possible but a lot of times there was a ton of overhead of actually adding those assets right so could could we actually turn the tables and have assets integrate something that would allow coinbase to just add them automatically and that was the idea with Rosetta but while I was working on that you know my interest was always going back into L1 I really wanted to get into like protocol development um but I felt like a lot of what happened at coinbase really prepared me for that so I uh you know learned a lot about what it meant to work with digital

assets at scale and you know what were the types of features that people really cared about whether that be exchanges or or users that coinbase had um and so funny enough Kevin who is uh going to the co-founders of avalanche reached out to me on Twitter actually and was wondering like hey have you heard of avalanche like what uh what do you think about it and um a few of the my mentors at coinbase actually had known uh Emmett from previous work you know throughout the years and uh you know I was really interested in working with great people and so decided to chat with them and uh one thing led to another and joined Avalanche actually joined Avalanche avalabs as a uh engineer um senior Junior and then a month then

was promoted to VP engineering so so fun fun few weeks we'll say it but uh yeah so that's that's how I got to where I am now awesome they're very very impactful Twitter message there so yeah really cool I think uh obviously I guess the main thing in avalanche is its consensus protocol sort of the main Innovation I would say and uh that's that's what most people Rave about in the community there that this is like solving like a unique bottleneck that I guess other um blockchain stoned I already still have can you just I guess we had him you know on already and talk about this but

I guess just for the context it would be nice to hear again from you you know why is Avalanche consensus so uh important yeah so um I want to preface this and say like you know I I definitely joined a little after this major Innovation so I I don't want to take credit for any of this uh but yeah so the idea right is you know when Avalanche came out a lot of interests had shifted towards proof of state so you know proof of work we we think it's interesting has some interesting properties but the amount of uh like the centralization of mining parties um and like how much green Greenhouse submissions that were associated with the industry was not great um and so like but proof of stake at the same time at at that around that time

you know you did you couldn't have more than a couple hundred participants without the whole network becoming really slow and unstable because of all the overhead of network messaging between different parties and everything uh along that kind of uh thought pattern um and so uh The Innovation with Avalanche was what if everyone didn't have to communicate but you could still use proof steak and so you know hearing that for the first time many people are like I'm not sure and so after you know a good A bit of research and simulation testing everything uh what ended up coming out was that you could have a sampling based gossip

protocol that was probabilistic but pretty you know could be parameterized to be pretty safe uh that was dramatically more efficient uh in network complexity than traditional pbft like applications where there were two interesting properties one was that there was no bottleneck at any point in the protocol so like you didn't one node didn't have to communicate with everybody ever the second one was that not a single party rarely had to talk to the entire network so like uh what that meant was okay you know when I'm starting a consensus process on a single node what I'll do is

instead of saying okay I'm gonna go collect signatures for everybody or like if there's this round robin where you like everyone sends something to someone you instead say I'm gonna pick 20 people at random by stakeway ask them hey what do you think about this block right now or like what's your preference they send it back and if some you know Alpha those of those folks respond the same way I consider that to be successful and then if I do that you know beta times in a row we're good and so people looking at that would say well you know conceptually that's actually pretty straightforward and and like you know you can follow what I just said uh but what that allows right is uh a totally different like Network complexity into I think what is uh

forgive me if I'm mistake about log log uh log n of node so like in the case where you have on average let's say on mainnet every sample is 20 peers um and then you end up doing 20 samples you only end up actually ever communicating with it most 400 nodes to finalize something and the you know the base case or the best case whereas um you know there's over 1300 nodes on the network so even in the current Network size you can see how much more efficient it is on the networking layer and so if you actually want to have a proof of stake network with you know thousands and thousands of participants um you know it provides that optionality without leading to a delay in finality so blocks on the Avalanche network from

time to Creation to time to finale typically occurs in about one sec so the fact that you could have this like huge base of participation without giving up what people really want which is this really really fast finality that is you know at the speed of the internet it offers those properties you know uh back when the white paper came out I remember like reading it and thinking oh this is like the this is like the end state of consensus algorithms I even had the thought I cannot prove it but it feels like you cannot have a consensus I'll go more efficient than ever lands right like this was just just Instinct at that time

how has the implementation actually been in practice as the consensus album lived up to its expectations or have you had to sacrifice many elements of of the of the initial initial claims in a lot of buried the working system it's a really good question and I think what I've never really heard discussed on any podcast about really any of these protocols is great you know you propose this paper well what does the code actually look like compared to what the paper has um so they're actually a number of changes that were made uh during the protocol implementation uh one of which actually a credit to uh Christian kashid and his team at Uni burn uh wrote a

paper I think last year about some issue with the paper which was never actually implemented because that was also discovered during the protocol process but I'm not going to cover that you should take a look search it if you want to check that out yeah so the paper itself was actually about a number of different subprotocols within the Avalanche critical so we call that like the snow family of consensus and then on top of that the Avalanche consensus process was which really defined how Avalanche worked over a day well four months ago the X chain which was the one the chain running the day was actually fully Sunset so there are no there's no more any uh like actual dag uh process he's running on the network and everything actually uses a separate

sub protocol that was not in the white paper called snowman now without going too specifically here uh two two in depth because I I don't wanna we could have hour-long conversation about this uh but the idea here is that the Avalanche consensus process uh like as defined in the white paper really was about connecting um or basically having at any particular height multiple vertices that could compose a single you know single four chain let's say so the X Gene you know there were just like any dag had multiple vertices at a particular height you know children of those vertices or children of that would then refer to those vertices but you could have like conflicting transactions and as long as like a transactions didn't double spend

each other like things would eventually get finalized but like you know because of network synchrony and like uh communication properties right like you may have duplicate transactions at different vertices right like because if you had a Vertex over here you would have vertex over here uh you know and the vertexes were seen independently there's just a lot of really complex scenarios that that had to handle uh Snowman on the other hand was about having a linear chain like most people understand where there's only one particular height one container now that was required for like the C chain and the P chain which just operated as normal chains on Avalanche um and what we found was that uh people really used the linear chains a lot more than they used the dag related things

because you could do a lot more on it are the Avalanche day protocol is pretty Limited in that you couldn't have like shared state that was mutated by different transactions so in the case of let's say the C chain right like you have a smart contract you know everyone's balances are in a decks and the next track transaction that comes in wants to modify uh you know that that operation the decks well you need some shared state of which to apply that to but if the deck never actually has like a canonical State at any point how do you even do that you can't right so you end up really limiting the functionality with the dag and you're sacrificing that for throughput which is the hope that says oh you know

if you're just doing a ton of transfers great a day could be more efficient because you actually can like sequence things concurrently you don't actually have to like distribute everything to a single block and then vote on that you can just shoot out what you got and then put that in consensus right away but you give up some you know some uh some complexity now uh or complex operations you could do on chain uh but uh you know there this is really summarizing that there are ways to try and add that functionality to the deck but uh we felt that instead proceeding down the path of snowman and then adding uh some uh advancement system and called snowman plus plus uh was much better because that actually unlocked the use of warp

messaging which is the cross-up that communication protocol now I could kind of touch on how all these are connected but long story short uh the protocol has evolved quite a bit from the original white paper and most of those specifications of the evolutions are now uh in readme's in the Avalanche go uh code base if you want to read about right yeah so I guess we're already sort of dove into it right this episode is mostly about hyper SDK and and subnets now you already mentioned the PJ The Exchange the C chain so it was sort of uh subnets that came at the launch of of avalanche sort of out of the box built by everlabs uh sort of for different

purposes right the P chain sort of ministering the validators and and other subnets and the C chain being like sort of the first evm chain for smart contracts um but yeah let's maybe just take it back a bit and explain a bit what has the subnet and then maybe also what is unique about subnets in avalanche versus like other application specific blockchains yeah so that uh I think that discussion on consensus kind of uh took a stat a little bit too far we'll back it up a bit so yeah Avalanche has this notion of a primary Network so the primary Network you know if you validate or stake on Avalanche that's what you're participating in the primary network uh actually one minor correction what you

said is not actually three submits but is one subnet with three chains now one of the most complex parts of avalanche is at the beginning which is a huge bummer as you try to like help people or talk about Avalanche because it's the first thing you end up talking about and then everyone's like you know there's these chains like what is they're staking like how is this all connected so the whole notion of a subnet and the primary network is a subnet is that a subnet defines like the set of validators that are participating and then within a subnet you have chains now those chains can have independent execution but the chains within a subnet can communicate easily between each other because they're validated by the state group so um but the idea here is when

Avalanche was launched or like when avalab started working on the idea of the Avalanche um just having basic functionality in the primary Network so in the case of maybe just staking or transfers uh didn't seem appealing or a way to really test out uh you know Avalanche consensus in a in a distributed system and so the call or the decision was made that like it would make sense to have Smart contract functionality too because that's really what people wanted and you know while uh an entire virtual machine could be created from scratch what's your own programming language akin to like a move or something like that um even at that time there was still so much interest in the evm that it it just didn't make sense to really go too far

afield from that so the C chain was added which uh you know provided the ability to deploy smart contracts so the primary Network though is just like any other chain in the sense where you have like a single state or virtual machine that everyone validates and participates in and is limited in in scale just like uh you know any like any Mainframe computer would be right like all it processes All State Transitions and like although there's this like this cool consensus algorithm underneath um you know you have to balance resource usage uh with like what your expectations of uh actual like participation are otherwise it becomes like the super centralized thing really fast right so you know if ethereum said hey instead of you know targeting 15

million gas per 10 seconds we're going to Target 3 000 or like uh three three billion gas per 10 seconds right like who would be left a bunch of super computers on like a few colos throughout the world right so in the same way like your target throughput is very correlated with the number of values you have and how distributed those will be and so some networks have chosen to have like a very high minimum throughput bar something like a Solana Avalanche has taken a different approach which is trying to really maximally distribute this like primary Network and then on the side have this notion of subnets which lets people create these specialized areas for blockchain

execution that maybe have very different trade-offs whether that be different types of virtual machines which I'll go into or just having very different throughput targets so like hey maybe for the subnet you actually need to have a much more powerful machine than you would ever do for the primary network but because you know we want to maximally distribute like the participation in the primary Network we keep the throughput much lower there and Avalanche lets you get like actually do that distribution right like if you can only ever have 100 validators you may not be as interested in like having this crazy distribution of participation right because why right if you have if you're only going to have a hundred but

if you could have like thousands you know you may benefit from having like a much broader sense of or broader notion of security yeah so is it is it correct to think that you know like when you look at kind of the development of advantage in some senses Avalanche started off with a time they said one ditch visually other other elements I mean compared to ethereum every other one is at a time disadvantage like five six years probably before I have that but even if you look at the other ones uh Cosmos polka dot Solana all of them started two or three years before Avalanche

Avalanche has been kind of like one of the one of the late comers in in that sense to uh in the space and and I think that the decision to just in the beginning have an EDM chain the the C chain was was actually really great because if Avalanche would have taken the the target of actually building a new consensus algorithm and then building like new kinds of virtual machines at the same time something which something like a Solana did right both they were innovating both on the consensus I will go and then on how um how Financial logic would work on the Chain Avalanche would have taken both of

those targets as simultaneously probably may have been too much by just taking the evm and doing its teaching actually Avalanche has has a small like Hazard has an ecosystem already like there's people like Trader Joe's that are kind of operating across both the ethereum space and Avalanche space and because the code looks the same they can deploy across both and I think many many uh initial Avalanche successes have that flavor where these are essentially projects that were trying to do something on ethereum and then they realized okay we could do it on the Avalanche and either get a new community or or we could get cheaper transactions or faster transactions so

that feels to me like that the first phase and now kind of the point at which you get really involved and meshed into the average ecosystem is like the second phase which is okay now that the consensus is stable how do we actually build performance virtual machines different from the evm in in that in that ecosystem yeah I would say on that point tons of pros and cons of that decision right so like I think in the pros Camp right we certainly were able to minimize the number of systems we had to build from scratch and you know anyone that tells you like oh you know you should just do everything all the time like has never attempted to build like a complex Distributing system before right you can

minimize a number of things you have to build kind of to prove out the major ideas like you should totally take that chance as long as it's differentiated at the same time still so I think that was something to really helped us get off the ground so far behind which I don't think many people really even realize right like the Avalanche mainnet launched years later than a number of these other ones right so like when people are comparing us directly like oh you know there's still so much to do it's like yeah I wish I didn't have to sleep but of course unfortunately my body shuts down at some point but on the con side I think uh at the same time people look at us and say oh it's just another evm4 right like oh you know you saw a fan to these uh bunches whatever

over a period of time like it's just another one of those and so you know there's a lot of work we're doing that people are just like well how much work is it actually to like maintain a fork of the evm like it seems like it shouldn't be this hard right and you're they're totally missing the point of life where most of our work goes into so like for example on the evm front we actually implemented that on top of the VM interface that Avalanche go provides for any virtual machine to use and we figured you know we didn't copy any of the networking the consensus you know we didn't connect any of that we used all of our own stuff and the idea was if we could support ethereum or like the evm functionality on top of this VM

interface we could probably support a good amount of different virtual machine interface and so we spent years really like dog fooding that and optimizing that layer and so I think we're currently on version 26 of the integration between a virtual machine and Avalanche go and to the hyper SDK which we'll talk about in a bit uh is that's how it actually communicates with Avalanche going the same way and it's built on top of all the work we've done over the years to integrate all these different virtual machines into Avalanche and so Avalanche itself like the code now that you should think about is really just like this consensus platform you plug into as a developer and so when you're building a virtual machine you're really just telling

Avalanche go hey orchestrate this virtual machine process handle the consensus handle the networking but that has been you know born through years of hardening the evm integration so yeah but on the note of like having projects start there it's been great to see you know some of the the teams that have you know found success there I think we're seeing some of that play out with some of the move chains as well we're like oh you know they tried it on Aptos it didn't work so now they're going to tread it on Sulu and see if they can build a community there um and so it'll be interesting to see I think you know the smart contract layers that are shared how that continues to be like especially with these steps moving from you know across to l2s now instead of you know other alt ones uh but uh

yeah it was a very interesting big decision I think and so far I think it still didn't net positive for for sure right so I think before we actually wanted to get into IV SDK like to just tease it again we also wanted to First cover a bit the like one of the core features that sort of this architecture brings that Avalanche has that every validator also validates the the mainnet which is is word messaging can you sort of talk us through a little bit what is word messaging yeah I think this also touches you know on a really interesting notion of like why would you even build a subnet to begin with right you have like you

have the castles SDK to build your own blockchain you have l2s you can just do that you might as well just deploy a contract like a Solana or Aptos or sui like how does subnets even fit into this and why is Avalanche really design the way it is uh to begin with so the whole idea for a lot of this architecture was based on the assumption that people wanted to create their own blockchains and that was the case years ago and is the case now the language in which or like the semantics in which those blockchains exist has certainly kind of changed a bit as people have had different ideas but the notion that communities want to create their own blockchains and they want to participate them and you know really have their own thing

but that the biggest problem eventually would be how you connected all of them so you know if you create a chain and I create a chain like we can have our own communities do things but what we've seen time and time again is like things really get super charged and exciting when everything's connected right when your contract can call another contractor when my chain can interact with your chain it seems to like multiply the benefit of blockchain rather than just being additive or even you know subtract like a Net Zero game or something like that and so Avalanche was designed from the beginning to have this at the core which was if the primary Network specifically the P chain can keep track of who's participating in

what subnets it'd be really good for two things one making it easy for exchanges integrators to stake everywhere because you only integrate with the p chain to actually stake but the second was this idea that it could also be used as like a registry store to manage messaging throughout the network so I'll give you one simple like kind of comparison here which is in Cosmos which was certainly a Pioneer and a lot of this cross chain messaging with IBC and all the great stuff they do there is no like Center chain that is a registry so every Zone has to communicate with every Zone like a connection and channel state that they're constantly updating for every message or like whenever there's a

validator change and so as a result it's not necessarily encouraged to have like every Zone talk to every other Zone because there's so much overhead of passing all this like State Management stuff around Avalanche takes a totally different approach which is you know validate this one chain and then there is no like additive cost per message you send other than just verify the signatures because you just look it up on the primary Network and so word messaging which we call our cross subnet messaging protocol is basically this way to securely send bytes whether that be an asset a transaction data whatever you want from one subnet to another where you the the validators basically on the The Source

subnet create a BLS multi-signature and then the validators on the destination subnet just look at that BLS multi-signature and say hey like what is the source subnet stake distribution or like who is participating on it and then say you know the subnet the destination determines if a message is valid based on the percentage that it sets is this valid is this message signed by X percent of stake and if so accepts it and so it's a very simple base protocol but it uses the benefit of having this like registry chain kind of in the middle to like actually authenticate those messages and so you as a VM you know Builder or someone using this you

just have to really just say get the validators to sign a message and then move the message but you the person that actually moves a message has no additional trust in the network whatsoever and you could just as cheaply move from subnet a to subnet b as it is from moving from subnet a to sub at C which is a huge benefit to avoid having like correlated risk as you take an asset and move it like further away from its origin right like if you have some an asset defined on a then you move to B and then B moves it to C now there's a problem on a c may have no ID unless you know manual interventions is uh done and I think Paradigm actually has a great article on this in the cosmos ecosystem like this risk of kind of asset

contamination based on how many hops you are from the source and so with this you know mechanism you really just don't need to do that so uh but yeah so that's the gist which is Avalanche was built for this multi-blockchain world and assumes that that was just inevitable and in that world what do you need the most you need a shared integration point so like all the stake is secured by the entire network and you need a really good messaging system the actual VM building is then what makes it all come alive so you set like the the BLS signature and and there's a certain percentage of stake that needs to have signed the message is that like

the state distribution from the specific subnet so some some staking on that subnet but then how much does it need to be verified by the the Avalanche validators like I guess in a subnet not necessarily all Avalanche validators are part of it is there like some threshold that's also needed there or so that's correct yeah so the actual primary network has nothing to do with messaging other than like basically maintaining that registry list so you could imagine a system where hey if I want to send a message from subnet a to subnet B I have to like first send it to the uh primary Network it gets persisted somehow inside and then sudden that B can read it this is a point point between the subnets themselves

and the way that this kind of works is it's all opt-in so a subnet that like gets a message there's no requirement whatsoever it has to process it at all depending on where it came from so it's on the destination to say first of all which subnets I actually want to accept messages from and then two what is the state percentage that must sign it for me to consider it to be valid so like for example in a virtual machine that decides to integrate it where you have like a kind of like a homogeneous asset mix where like every asset looks at like the same you may have a very high security window and say like oh you know we need 95 of stake and we only want to accept messages from subnets that have so much value uh like

associated with their token and then on other subnets or like you know for example on some other VMS we've seen where the tokens are actually prefixed by their source you may just say across the board like if 80 of some subnet signs this token like we'll accept it and then people can choose interact with it if they want so you're given that option when you actually implement the virtual machine right so we are kind of imagining it is you know I tend to think of blockchain says I mean it's kind of wrong in in details but I tend to think of like you know scribes or accountants maintaining an Excel sheet right uh jointly maintain any extension that's

the Royce description of a blockchain I haven't just submitted Google form and then you're good yeah yeah it's it's kind of like the easiest way to totally like think of it and so it's it's like the ambulance Main night the C chain peach in an exchange so you can think of that as like the biggest Association of scribes or accountants in the ambulance ecosystem so anyone who wants to be an accountant anywhere in the Avalanche ecosystem you have to go and join the mean Network so you you build up like this huge pool of two thousand three thousand maybe in the future tens of thousands of scribes

that are that are kind of like maintaining that that Excel sheet of the main the main subnet and those are the three chains there and then whenever a subnet needs to be spun up um what that kind of main Excel sheet is kind of storing is what is the subset of accountants from that main Network that will that will kind of maintain that that subnet so if you create subnet 37 there's a record in the main Excel sheet okay for subnet 37 these are all the accountants and that's kind of like economical place where there's stool so that mean this is storing who are the accountants for each of these subnets

and so when I'm subnet 36 and I get a message from subnet21 all I need to know is is that message valid and I can just consult the main I can consult the main Network say what's the list of scribes and what what percentage should have signed this message for me to accept it mean it would give you that response and then you can trust that response and accept that message based on that yeah basically that's your input into like the message verification process to say what are the POS keys of the people that are supposed to sign and what is the weight associated with each one of those BLS public keys and then you can just say whether or not like locally that's

valid so like but it's really up to you how you actually want to like maintain that delivery so like you mentioned there's no main net component that it like the messages don't have to go through the mainnet secondly one point really small to clarify right is that when you're talking about the scribes they're never assigned to a subnet everything's opt-in so Avalanche right now is very much designed for like a sovereign World rather than a shared security world so people that are coming at it from the world of hey I know what l2s are I use ethereum I put data there not how Avalanche works at all subnets maintain all their own data the primary Network just manages stake of subnets and the validator registry for subnets

but you don't actually put data there and it does not offer any sort of like proof capability to prove that something that happened on the subnet was invalid or something like that now that may change in the future based on like what appetite you know people have shown for like it'd be great if I didn't have to provide my own security um but so far and you know how Avalanche was originally designed and at the time at which it was designed there was no interest at all insurance it's actually pretty funny how much it's like shifted entirely like shared security seemed to like really become this thing maybe a year and a half two years ago like I don't actually want to provide mad security I just want to use someone else's whereas before that like it was all about this notion of like we're our own Community we have

our own decentralization we have no you know no master that we like kind of work with or like you know we're our own Sovereign Community we don't want to work with anybody else we want we we Define our success uh so it's interesting how much that Spectrum has uh since that time but uh yeah so for people that are using subnets you know if you were to shut off all the values on a subnet it's gone so it's it's like the subnet provides its own security and it's opt-in so you you join it um or you state to join it but maybe also I guess to also like Avalanche doesn't have slashing right and there is so there's really let's say there's a malicious subnet somehow there's actually no like

recourse for the main ad validators that validated it sort of in a way what you would do so like in the case where you had someone that wasn't participating right you lose your reward so like the the penalty is really like hey you were going to get this staking reward now you're not so it's not like there's no penalty whatsoever it's it's similar to like you just you'd get inflated out of the system more or less but on the subnet front there's no load for the subnet on the main Network so like if a submit was like I don't even know what it would like there's no control at the sudden right you can do whatever that's the best part and most complicated part about it which is where the hyper SDK like ends up coming in is

to try and bound that a bit for people that want to get started but you know it really does its own thing and so if you are on a subnet that all of a sudden like goes absolutely wild you can just stop validating it no penalty to you other than anything related to that set that and so like unlike something where like a polka dot for example where there's resource constraints per parachain to make sure that when you're assigned to a pair of chain you can't just like get totally wrecked by whatever the parachute is doing uh there is no notion of that on Avalanche because it's all opt-in so if you want to participate in the subnet you better have the resources required to participate but if it's malicious you know you should just not and there's no

penalty for that outside of that community so maybe that Community will say hey you didn't get your reward and you'd be like fine with me like I don't I don't I I don't want to anymore based on what your subnet is doing so that's not necessarily a problem per se now the one very small side I will say is that a lot of the feedback we've gotten has been that this like kind of strict uh connection between the primary Network and the subnet has made it much more complicated because you have to like validate the primary Network and all this stuff like that and the subnet and so like you know how can you possibly be efficient if you have to do so much at once just to run a single subnet compared to like an L2 where I can just turn on a server connected to ethereum

good to go um and so there has been a lot of conversation in the community about changing this a bit and so having subnets or participants actually just like verify the P chain or what is required for like just to populate the registry and not actually validate the primary network but just read it and then store data on it and then that would make the like subnet running process like much lighter and with that wouldn't really sacrifice any functionality so there have been a lot of conversations about that nothing really has started I would say but um you know I think with any network right as the community interacts with it there's like this constant Evolution and it's a really good metric of like how

how much people actually care about what's going on is like how far did you actually end up deviating from the original ideas based on interaction with everybody you know I guess unless you're like a satoshi's vision fan in which case I don't have a study but I think in the case of like Avalanche what we're thinking about it how the ecosystem has continued to change and what does the community decide to do in in reflection of that based on what people are actually trying to get going yeah I mean this this such a big variety of what I mean when you look at the entire crypto space and that includes like near Solana's Cosmos maybe something you like a Celestia

there's such a huge variety of what the central chain is trying to offer as as Services right sometimes it's Central chain isn't offering anything at all which is the cosmos hub maybe the most minimal set of Services the central chain is trying to offer historically then maybe you have like an avalanche which is of English Services of okay registering validators and tracking who validates what then like maybe something higher would be change that are of trying to offer data availability so if you have a subnet you go and run your subnet we'll make sure that your data is accessible it's it's some in case

something goes wrong in your system we have the data to process who did what wrong and start a code of some kind that's kind of like ethereum probably like the celestial Visions are kind of there and then like the uh the polka dot and near Regions would be yes you can come and run your subnet and we will ensure that a malicious transaction never never comes in like a central place will kind of Ensure ensure that your accounting is always right and your data is always available and so there's this whole range and yeah it's kind of like Market forces will

ultimately decide what what model ends up the winner yeah I mean I think the one thing I'll say in response to that is I tweeted about this a while ago too which is I think the ecosystem generally should be supportive of this ski like so widespread approach and like that resilience and like interest to go down so many different paths is what makes crypto so strong like and resilient if we all send like Focus so much on a particular thing and say like everything else doesn't make any sense like if you're doing this like I don't know where have you been like another rock ends up destabilizing everything because like if that particular approach has some sort of setback you know whether that be like regulatory legal like

security wise who knows what it ends up we gain everything because we now all made this assumption that everyone has to do this way so when people talk to me about like do you think there'll be a single chain or like is Avalanche the only way you could ever do something I always emphatically say no and I hope you know I really hope it never is right like I think that the notion of having this like really interesting heterogeneity in the space although it may seem sub-optimal is like the not to use uh like the cliche word but like the anti-fragic fragility of everything is I think what excites so many people um but uh yeah I really think that this is a great transition too to the hyper SDK step which is tried to at least on

the Avalanche side make that easier to take advantage of right yeah let's go there for like 40 minutes in and getting to the the meat of the conversation so right we talked about like Avalanche like sort of trade-offs it's making the the concept of subnets and now essentially what your main focus is in Avalanches as far as I understand at least is um this this concept called like or the project called hyper SDK and you just yeah basically tell us what is hybris to do yes I mean for a long time uh if you wanted to interact with Avalanche in a custom way we would say or I would say like oh yeah

you can do anything it's so flexible like there's so much you could possibly do but if you want to get started quick you know we have the subnet evm which lets you like spin up an easy app really quickly and that's really what got people's attention with some that's in the first place I like tweeted this thing out about like start an evm with a single Json file and then that like people loved that post but as a result like for a long time all people ever did on Sudden that's deployed ETF because it was just like this super Json easy contig and then I realized one day I don't remember specifically what it was but I was like you know I keep saying it's so easy and straightforward like there's so much you could possibly do like just go do it I realized that like

you know to actually do that from scratch is not a super straightforward task and it requires uh like to do it really performantly quite a bit of engineering work and so I decided at one point hey let's just in one particular case offer like a very opinionated view of how you could build a foundation for a blockchain that lets people go from maybe step 0 to step 10 and then let them or like maybe Step Zero to step eight and then let them go from Step eight to step ten because there's a lot of functionality you want that you would want to build on top of the Avalanche go process that is not necessarily provided out of the box one of those for example would be

something like states which we can go into later is not something provided by the core of avalanche but any high throughput or modern virtual machine would probably want that so I you know thought about a lot I was like you know there's a lot of things here actually that if we just implemented one time in one opinionated way could make it much easier for people to build their own thing and to me Avalanche right if all it ever is is like yeah it's a great place to run like modded EVMS I guess to me that like is such a short fall of like what the original hope and vision were uh makes me sad I guess but generally you know if if you want to see change in the world you gotta be that change I guess sometimes and to me like the hyper SDK is something I'm very

passionate about and have a lot of fun working on so the hyper SDK in short is it's a toolkit or framework uh really for people that are trying to build their own blockchain that takes advantage of the best Avalanche has to offer that we may not have been able to take advantage of in other virtual machines we have because we're trying to preserve maybe some sort of functionality over performance and I think in today's blockchain world so many people are interested in scalability that targeting you know scale at all costs was really the North Star and continues to be the North Star for the hyper SDK is to say if someone comes out and asks what's the fastest and most scalable thing I can

build on Avalanche we wanted to have an answer at least avalabs did for what that would be and that's the hyper SDK so maybe I actually start want to start with really conceptual question on like what do you mean when you say virtual machine yeah that's a good place to start yeah often like our conception of virtual machine is okay it's a turing complete virtual machine when the some kind of programming language is solidity with which you can smile write smart contracts and you can deploy them but I think that you mean virtual machines to be something broad right what is yeah so uh in the in the context of avalanche a virtual machine is anything that

is the subnet specific logic that is running so that includes State Management that includes RPC it's more of the virtual machine idea of like you run an ec2 like it's a general computing surface that you can utilize with some like functions that are provided to you so like I agree with you like the typical virtual machine people use is like ethereum virtual machine right like it's just the logic that executes the transaction like hey this is the transaction format this is like how that transaction is actually executed great now we've done the state transition what's next now I think because there's so much complexity here virtual machine has kind of broadened and narrowed based on who

you're talking to about what exactly it is but in this case you should think about it more so as specifically you could say any sort of binary that implements like the Avalanche go acquired consensus calls and that can be anything so you know you don't even need to have blocks you don't even need to have you know any of this crazy stuff it's really just a binary that can run and communicate with avalancho now the components there just to be specific right things you may want in something like this what is a block so you would Define a block what is like a transaction look like you would Define a transaction you know what do you store like what do value is actually like store on their disks uh you know how do

you access this thing like what sort of API calls you want to have it's all up to you as a virtual machine developer and so you have to remember right that Avalanche the goal here is how can we create as low a level framework as possible that still allows for this messaging interface and shared staking layer that if you were going to create your blockchain from scratch you could still do so in almost any case on Avalanche and so the virtual machine we're talking about here is much more like uh you could think of it almost like a server like it it's much more holistic than just a state transition logic that you may find in like the evm or the VM

you can almost think of it like a mini node kind of particular to your blockchain and now hybrid SDK sort of is a like I guess toolkit and and allows you to to build these takes like some some things out of the box give you something could there also be essentially like another SDK like oh totally getting things someone builds that has like yeah different opinions about how I O should be done in something exactly so the hyper SDK basically says you know what I describe is basically super broad right it could be going could be rust it could be yeah your blocks could be potatoes like the the whole idea with the hyper SDK is to say okay you have a million options we're

gonna we're gonna choose a bunch for you so this is what blocks look like this is what transactions look like this is what addresses look like and then the whole idea is will give you areas where you can inject custom logic or your own design that does not impact the throughput that the network should be able to obtain so you want your own custom thing like you want your own transaction types maybe you want your own you know you want to use certain types of cryptography like there's a cow distraction like you know you want it you want all that but you don't want to like actually Implement you know block distribution you don't want to implement like statement you don't want to implement State syncing

you don't want to like so it takes a very particular set of trade-offs and said we assume most people want this abstraction break Which is higher than what Avalanche go provides out of the box but lower than just deploying a smart contract somewhere and says most people want to probably start around here and that's that's the idea of the hybrid SDK with with the underlying assumption that you want to go fast or you want to have a lot of transactions so one of the things that you one imagine wants to compare hyper sdk2 is the cost NOS SDK they even have similar names

but in the cosmos SDK like you can use it to kind of bend your own application specific blockchain you implement your transaction types and what the cosmos SDK will provide is a way to communicate with the consensus and networking that is attendant but the cosmos SDK kind of also offers opinionated ways so here's how you do staking here is how you do governance here's how you do auctions here's how you manage a token right like so but hyper SDK it seems like has a narrower set of Focus where

where you are not seeking to offer here's how you do governance here's how you do auctions you only say probably like here's how you do State Management here's how you do block distribution here's how you offer probably some kind of RPC or API to people seeking data but you don't want to get into the question of how to do governance how to do how to do staking how to do auctions or how to do token management or any of these Financial logic in a sense yeah so I think like the way I would answer that would be it could like there's nothing

that prevents it per se as much to say as we would prefer that to kind of wrap this aspect of it like you could Implement you know a separate kind of layer on top of this to say oh you know here's a module store of things you could pull in and the idea here is that the hyper SDK is really just concerned with this like super high throughput fabric that that you weave whatever you want into you also have to imagine that some of the things that the cosmos SDK has to provide are already provided by Accolade so like for example the staking module that doesn't exist on chain so you don't actually have to worry about all that and that's where a lot of the like bugs and complexity are for some of the other sdks which is just handled by

athletic um the one thing though that is different just for very briefly the technical audience is tendermint handles state always so if you want to use the custom SDK you're using tenderman's state the hyper SDK provides its own state and it's pluggable so you don't actually have to use uh like Avalanche goes State orchestration at all you can provide your out so you could hook it up to AWS if you want to or you could just have your own high throughput store underneath it also provides its own networking which Cosmos SDK does in the lab so I think maybe there's some other ABCI plus plus stuff adds more flexibility there but for a long time at least you you just tend to handle that

for you so it provides them I guess in some ways less functionality on some parts and that in other ways much more control into the core than I think there is anywhere on any blockchain building framework and so um but to answer your question like could we shape the hyper SDK to like have all these different modules and different things that Cosmos SDK provides which people have shown interest in certainly you could totally like build up that scope if you wanted to but I think that the way that we approach it more so is we're platform and Tool Builders we want to not like build whole thing up to the top and we'll have examples for sure but the value seems to come from can we maintain with a minimal feature

set that's really high throughput layer and then have all these super specific logic that you care about maintained elsewhere otherwise it just won't be reliable we just think there'd be too much scope to actually maintain based on what people care about and so as a result you know we've achieved really really high throughput but you know have spent less time on like extremely complicated transaction types because that's not really in the immediate scope of what our first phase is really about yeah so you're mentioning a lot like obviously here performance and and that seems to be like a core of your interest right and I guess also like what hyper SDK set out to achieve to like really utilize

like the potential of avalanche I guess in that sense and yeah we would like to I guess hear so what does like where is this optimization really there you know what are like the top things that hyper SDK helps you optimize which uh maybe other systems don't yes I mean I think with any like thing you're starting from scratch right you uh you throw something on the wall based on your initial work and then anyone in performance will tell you all your original assumptions probably ended up being wrong and you just have to Benchmark everything and repeat so Benchmark your ideas repeat and if you just out of nowhere say yeah this is definitely going to be the most optimal

but you haven't tested every individual component you're probably wrong somewhere there's no way unless you're like I mean maybe I'm only a you know 2x engineer and there are 100x Engineers that know exactly what part I think it's gonna be totally messed up as they build it but my experience has been with this repeated benchmarking and so but the the three areas we've really identified as the primary bottlenecks at least within the context of avalanche are State Management data distribution and deferred validation or verification and all of those are really what give it its speed so and like really a speed at which it assumes that people can still join the network via just a network and I'll dig into what all what I mean by that so the first thing State Management

blockchains modifies State you know that is the state transition right so if you can't store and read State extremely fast and make sure that the overhead of a ton of like State transitions in a row is not well maintained you know that's going to be your bottleneck at some point right like you're going to run into problems because you can't keep up with the transactions like the nodes unless you have like crazy disks and everything like that uh and then secondly if it's not well maintained such that people can't efficiently join the network you're going to have problems too because if what say you get 10 million blocks in and then like you realize oh the only way to sync is to sing

from scratch but it takes longer to sync a block from scratch than we're producing them you're going to have problems like the Network's gonna die unless you like somehow snapshot it gets really complicated right and it becomes very centralized then because then you have to offer disk backups for people to go get so the first thing that we really focused on with State Management so we built our own database from scratch uh two of them actually one of them is released one of them isn't uh the one that's released is a Merkel path based Radix tree and the idea there is it uses path-based storage to really efficiently store the current state and then some number of diffs back now I have to give credit a lot of the ideas

of this particular implementation and like the theoretical paintings of it came from some of the work that Peter Schlage and the Geth team has done so I want to say that like they definitely thought deeply about this and pioneered a lot of the ideas in this like path-based architecture so I just want to give them credit where credits do but we implemented a different version of it from scratch that integrates uh State sinking with it and then that can run on any key value underlying database the second thing that we implemented which is not out yet is one that takes all of the tree management and everything related to actually doing all the stuff you would do with the Merkel cherry and actually interweaves it with the key value stored underneath so

instead of having to let's say have some data structure and then you know you run Pebble or you run you know level DB underneath it's one and the same and as a result you can much more efficiently manage that data and so State Management again this is not a super original idea is one of the primary bottlenecks of these high throughput chains and so you may have to store everything in memory you may have to use nbme drives whatever the second data distribution so if you have a lot of transactions you got to have a lot of data and that data has to be distributed to other people to actually validate it and accept it in consensus if you can't distribute the data as fast as you're processing it bottleneck so that

Avalanche go gives full control over networking to the virtual machine so like to the hyper SDK so that means you can either use standard P2P stuff Implement any P2P interaction you want on top of it or use something outside of P2P in general let's say that you have like a permissioned use case for like you have your own network and you have like a really high throughput bus you want to use you could totally use that the hyper SDK provides like a P2P primitive for Distributing data between the participants in the hyper SDK like in a hyper SDK based Network that allows for like chunking the blocks up and then Distributing them in a way uh that is as fast hopefully as we're processing that data and then the last one is this

notion of deferred verification because this is very dense I'm expecting also to come back and maybe summarize some of this but the last one is deferred verification and the idea with that is if during the consensus process you actually have to verify the block end to end you will really slow down the ability for the network to actually uh finalize new data because before any node can actually vote on that new data it has to actually verify that it's correct otherwise it may vote on things that like are invalid in which case it would get slashed or you know penalized or the network would become unstable so what can you what's the absolute minimum thing you can verify

such that you can still participate in consensus and so at the hyper SDK breaks apart everything other than what is minimally required to derive a deterministic result and only does this pre-process step before voting before actually doing the full execution now in summary right you as a VM developer probably don't care about any of these or even know they exist in some cases you just want like this is what I need to build and I want to hit this throughput or like this TPS Target so the whole idea of the hyper SDK is let's take all these like really useful ideas Implement them audit them make them production quality such that when

you build in your own virtual machine you just say I want these types of transactions and I want them to do these things and that's it and that's the whole premise of what we're trying to deliver with really high throughput like this kind of fabric underneath everything and so it changes a bit how VM virtual machines are designed but you could apply you know all sorts of crazy transaction Types on top of this as long as they're deterministic and that's really the only requirement kind of like every blockchain is is like every blockchain ultimately needs to do something in our state management and data distribution the third one which is you don't need to verify the entire

execution for you to sign something maybe like this is you have like optimistic verification of some kind so yeah so I would clarify that particular word optimistic because that has become very charged as well so optimistic as far as I understand it typically means you'll Val like you'll basically execute something or you know maybe some form of executing it and then at a later point in time could be proven Wrong by some proof the idea with this deferred verification is that you validate enough such that that can never happen it can never be proven and correct so it's still going to be correct you may just not know what the result is

yet but you agreed on the inputs that will lead to the result and such if everything is deterministic you'll get the same result everywhere so it's a very it's kind of a breaking apart when you need to like push these throughput right you end up hitting certain walls and you have to I will say that like as you try to develop High throughput systems it really requires you to take everything you know about blockchain at least everything I knew and like really pull it apart and say like why is this time here was this done here for a specific reason or just because it was convenient so like to give you some comparison it eats like um especially like the communication between the consensus layer and the

execution layer how it works is the exec like the consensus layer will get a block like some consensus layer block and then it'll ask the execution layer but hey can you verify this block and make any state changes in a like pending State and then if it can the execution layer will then respond that like you know I I've executed the block um and then you know we've done this come together um but in this case what I'm saying is what if there was some way or like some sort of constraints you could impose such that the consensus layer would just tell like the execution client hey this is what you will verify make sure these people

have sufficient funds to process that that's enough that's all I need and you don't actually have to make the trendy modifications you don't actually have to like generate the next route all the steps that really take a lot of time during that process you just don't have to deal with um and then you like pre-verify all the signature so that's also not part of that process so uh basically you can take what is typically a monolithic verification chunk and then separate that into what its constituent components are and the hyper SDK really tries to be very specific about exactly what part of the verification it's doing when and the invariant we offer virtual machine

developers then we will ensure that this is done correctly even if we separate all these parts like you basically give up control over how execution happens and as a result you get more throughput so right now hyper SDK is you know Alpha software and it's headed towards a production release what kind of benchmarking have you done and what do you expect to be able to deliver in production for a for a subnet meaning like number of validators multiplied by let's say simple money trans money transfer transactions what could you give in a setting like that

yes I mean I think this is the funny thing about like a TPS right which is definitely not there's like a hundred ways you could manipulate this number to serve your interest and uh I've actually been really interested in like a standardized framework that we could use or any network could use to plug into that tries to approximate like the complexity per transaction so that like the tps's can be viewed similarly I think Aptos also released a similar benchmarking tool so that you could use that amongst different at least move virtual machines but the to give you a short answer the goal when the hybrid SDK was created was to shoot between 50 and 70 000 transactions per second of simple verification and as I got deeper it's like one of those things where like

if you have a number you want that number to go higher so like you just keep looking at that number you're like all right I got this this time this this time with the Deferred verification stuff we think that we'll be able to go over a hundred a thousand TPS and so that's now the new Target for hitting when we're actually finished implementing the Deferred verification because we've now removed you know all the bottlenecks we can find and really the bottleneck now is how quickly two things how many cores and how quickly can you move the data between nodes based on where they're located so naturally the TPS would be higher let's say obviously if everyone's in the same data center right because like the network latency between two nodes instead of being you know I'm

going from you know uh Japan to the United States is now within you know a couple feet of each other but that's what we're targeting is in that range and so um so from a performance perspective that's really important to us but like I said talking TPS is such a wild game that I'm not I very rarely say like hey we hit 65.3 transact because it just turns into a war very quickly on Twitter so I really try to avoid that just to say that what we're trying to Target here is hyper throughput not necessarily like the throughput layer and so this to wrap this kind of up is it's a real set of trade-offs you want to make right

like if you need to have data availability somewhere storing the amount of data you need to process 60 to 70 000 transactions per second even compressed that's a lot of data like more than any other like more than most blockchains process in their entirety so like if you want to achieve certain throughput you know you have to make trade-offs right you can't I can't store 20 megabytes per second on this area and that's just a function of the transaction size like you can't do anything to get around that and so even with like some of the crazy like uh some of the new like data availability things that are being added to ethereum it's still expensive and still would saturate old so if you want to achieve like crazy levels okay well Matt you're using you

get into validium territory or like L3 territory but you have to put this data somewhere and so do you want to use you know some other mechanism have like put that in some other thing that you pay for separately or do you want your network the subnet really to own that in its entirety and the hyper SDK is really about providing you that option which is maybe your community just wants to run the whole thing and they just want to achieve that throughput and they're willing to own and secure it to make that happen so like my personal experiences it's like running an organization that runs loads across like 50 networks including the Avalanche main net my sense is you know that the

like a submit or a soldering chain or your own chain in any way starts to only make sense when your application is kind of really successful in terms of transactions per second or it needs certain High custom features like you know the perfect example is kind of like a d y d x like a network processing a billion dollars in Perpetual volume per per day already has success started off as a smart contract and now it makes sense to go Application specific right like with their own blockchain that's when like building your own chain

makes sense uh it doesn't make sense if kind of you're just starting out and you hardly have any users and if and if you think of like that model that okay people are going to be building their own chains when their applications has kind of like success and the need for throughput is kind of cleared then usually those kinds of parties can afford to bring 50 or 100 or 200 of their own validators ensure data availability themselves and not need not actually even want to rely on other people for it I totally understand your point and I think that that's why like having this spectrum that people are interested in supporting

makes sense that being said you know I think there are parallels early on like other technology Journeys uh you know by providing the functionality where you don't assume you need to have this like load throughput you can just Target a totally different world to begin with can be quite an unlocking for us right so like if I agree Corral in your security writing all these nodes costs everybody you know millions of dollars in USD to set up that's quite a large commitment for people originally but if you can make it such that that architecture and that framework is much more compatible with like a pay-as-you-go kind of setup as your network grows and scales such that you from the beginning can achieve a totally

different world you don't have to start like this like really slow walk on like a you know traditional L1 TPS Journey if from the beginning it's just not even conceivable you could achieve that with your application use case so like the way I think about it sometimes and is like great I thought I have dial up I'm only going to do dial things on the internet and you know you could argue that like great if broad map like Broadband costs more money I'll start with dial up make sure that people like it and then go to broadband but if there's some things that never would look good on dial-up you got to just go right to broadband and so as I think about what hyper SDK Labs it's just a different trade-off

right maybe for like really high value D5 applications this is maybe not the best place to start if you really prefer maybe shared liquidity and shared security but if you're trying to just think outside the box and do something crazy that just would never make sense economically on an L1 where you have to share all your data or whatever I think there's a viable alternative here where you can Corral security or maybe at the security at the beginning is just not as important to you for some reason so I think there are trade-offs and the whole point of my Spiel I guess is that's okay like I don't think there has to be a one thing we all agree on right I guess you could also start out with like you know I don't know one

machine more or less like validating the subnet and then grow that over time if that's the the bottleneck but you start with a very heavy machine there already sort of in a way yeah I mean the point here is we're providing the tool to give the you know virtu the aspiring virtual machine Builder choices that's it all right cool the out Patrick thanks so much for coming on and and expanding on hyper SDK and Avalanche for us like probably for me at least one of the most deep engineering episodes I've done uh so it's super interesting and I think we hopefully we fared well um maybe yeah to before we wrap it up uh it would be great if you could like

again tell us where the hyper SDK is right now in development and you know how how people can get involved yeah great uh yeah thanks again for having me I I always have a blast talking about this uh tried to cut myself off at some points but you know we uh I love to have these comments of you know other people want to chat about you know whatever is going really deep uh whether that be one-on-one or whatever uh very interested in in doing that um but uh yeah so where we're going now is really starting that path to production hardening so right now as you mentioned it's Alpha level software so you know it's not audited yet I wouldn't say it's safe for production but it's really fun to experiment with so if you're a contributor it's a great time

to join uh the project and start contributing to it everything we do is fully open uh so if you want to start like earning uh you know contributions on different projects this is a great one to check out there's a lot of low light low-hating fruit of you know whether that be just simple testing or like kind of clean things up doing basic optimization work with memory work or you know speed of execution it's a great place to jump in um and then the other thing we're working on in association with the production hardening is adding a basic wasm executor as a what we call programs so although so it's great to always be extremely hyper optimized some of the most exciting things on blockchain come from having this like permissionless

deployment of programs on chain and so we want to offer that option as well to people or maybe you know the vast majority of your transactions are very specific to the hyper SDK and like implemented but you want to offer some ability to like stitch them together or do things with that guessing every possible way people may interact with your virtual machine so that's another thing that we're really working on now so yeah I mean fun time for me a lot of coding is what that means and hopefully uh you know the next few months uh will be entering the audit process for much of this uh because uh you know speed doesn't mean anything if it breaks or has massive bugs right so uh you know that's a a huge concern

all right then yeah Frederick again thanks so much for coming on and yeah hope we get to see some hyper performance hyper SDK subnets uh when we next talk in like a few months and yeah thank you for to our listeners as well