ZKPodcast: Anatomy of the Cosmos Stargate Upgrade

Transcript

[Music] welcome to xero knowledge i'm your host anna rose in this podcast we will be exploring the latest in zero knowledge research and the decentralized web as well as new paradigms that promise to change the way we interact and transact online in this week's episode james and i explore the cosmos stargate upgrade we do three interviews with three different participants in the upgrade that is zucky munyan tess reinersen and shahan cachadorian we talk about what the upgrade was all about how it went and what kind of

changes it brought to the cosmos tech stack but before we start in i want to remind you to subscribe to zk mesh newsletter for the latest in zk news research events and jobs and more the newsletter comes out monthly be sure to check it out i've added the link in the show notes i also want to say thank you to this week's sponsor leased authority leased authority is a security consulting company known for their dedication to pushing the limits on how to build privacy respecting solutions they are a team of security researchers cryptographers and open source developers passionate about advancing the security of projects in the blockchain cryptocurrency and defy space they're known for their audits of the eth 2.0 specification protocol labs gossip sub protocol mina protocol

stack's investor wallet and more they want to help teams improve the security of their protocol and use of cryptography including zero knowledge proofs from design to implementation their independent reviews improve the security of the technology by immediately helping developers of the projects along with their users and when the reports are published the broader community can benefit from this shared knowledge as well with over 50 completed public audits to date join the growing list of audited projects to do this book free consultation at leastauthority.com consult i've added the link to this in the show notes so thank you again least authority for supporting this podcast now here is james and my exploration of the stargate upgrade

[Music] so today james and i will be exploring the cosmos stargate upgrade and we're going to do this by interviewing three different people first off i want to say hi to james hey james hi anna and i want to introduce our first guest zucky welcome back to the show uh it's a pleasure to be here and zucky you've been on the show before we actually have an episode that i'm going to add in the show notes where we we had a full hour we got to talk about all the work that you've been doing actually at least up to that point but since you are somebody who wears many hats i feel like we should say that for this episode you're going to be wearing the iclusion hat as we talk about the stargate

upgrade so why don't you just introduce yourself and what iclusion is doing with stargate yeah so i'm jackie munion i build blockchain protocols i work on cosmos and cosmos related protocols and other protocols i'm a co-founder of a company called inclusion uh i co-founded it with myself christy polstem shelley stevens and tony archieri and we started this all the way back in 2017 basically with the idea of running a validator and trying to figure out about be a validator we do uh some combination of like new protocol development uh running validators um supporting cosmos in the cosmos ecosystem consulting for other

ecosystems and basically the only thing that we don't do that other people do in the space is we don't run a fund got it so many hats so let's kick this off what exactly is stargate what is this upgrade and how did you actually interface with it yeah so the history of stargate is cosmos underwent this like sort of significant reorganization in 2020 where like a lot of dev teams moved around a lot of people moved between orgs and we did game of zones which was an incentivized test net that was a follow-up to game of stakes which was the first incentivized test net um and after game of zones which was like sort of the big public

test of ibc i sort of had like checked back in into what was going on in cosmos software development and i had known that there were a lot of big changes that were planned and that there had been like a lot of engineering uh discussion and planning and work and i what i realized in the cosmos sdk process like in the process of doing game of zones is i was like oh there is no way to ship ibc i've always felt like a sort of like somewhat almost religious obligation to strip the cosmos white paper and i was like we can't ship ibc we can't ship the cosmos white paper until like without also bringing in all of these other changes and what i realized was those

changes were gonna have a really dramatic impact on exchanges and validators and wallets and all of that stuff was gonna be this big really painful upgrade that we had to do in order to ship the cosmos white paper and so i sort of decided that we were going to take this on that we were going to do this that like all of these upgrades were all going to get bundled together into this one big upgrade that also enabled ibc and that i was going to go and lead the charge on making sure that exchanges and wallets and validators were all ready for this big disruptive change which is basically the biggest and most disruptive change that has ever been done on a widely integrated blockchain in the history of this industry um and

that's what stargate was let's talk about this white paper kind of like what cosmos was and what it is now so you mentioned ibc i actually have done an entire episode on ibc with crisco's i'll add the link in the show notes to that and ibc as i've always understood it is the thing that allows for this interoperability in the cosmos ecosystem to actually exist before this point what was cosmos actually so cosmos was the first sort of bft fast finality proof of stake blockchain um it was the first framework for building blockchains at least one the first framework that was sort of widely adopted for like build your own blockchain so

people built their own blockchain cava terra band all built their own blockchains on the cosmos sdk and then we were this proof of stake bft consensus system that like showed that systems like this could exist and that that was cosmos before then the promise of cosmos and the white paper was this internet of blockchains right yep and it sounds like you're saying ibc and the stargate upgrade are the last big push to achieve that goal that you set out to do five years ago yeah so five years ago i you know helped with editing and writing and made comments on uh the original cosmos white paper and it was basically this idea of

interoperable blockchains and by stargate the stargate upgrade happened on the cosmos hub back in february completed the cosmos white paper like the cosmos white paper is done the software is shipped the system is running it's actually finished uh and it's been a big deal to for me personally to like have that burden uh lifted what does inclusion and yourself actually do in something like this what does it mean to do an upgrade from where you're sitting so there are more than 100 exchanges that integrate with cosmos there are probably about a dozen different independent wallets that integrate with cosmos and the cosmos hub there are block

explorers there's like a whole ecosystem of software that is not produced by let's say anyone who has who commits code to the cosmos repository or like goes to any software development meetings or is like in the discord channels and so there's this whole peripheral ecosystem that is really fundamental to like what makes a blockchain work why does a blockchain exist and the way in which cosmos sort of organically evolved in the ecosystem is there were dozens of different bespoke integrations between exchanges and wallets with cosmos software and what i realized when we were doing game of zones and getting ready to when

trying to figure out what it would be like to launch ivc we realized that like this was going to disrupt every single one of those integrations um yeah oh man and that that that that was that was you had this like sinking feeling in your stomach you're like um and so i came up with this brand of the stargate and it's the brand of the stargate is you know we're going through this thing there is a before and there's an after and once we're after like everything is different and nothing will ever be the same again and i i thought the name would like be helpful for convincing all of these different entities that they were all going to have to put hundreds of hours into this their integrations fixing the problems

and the disruptions with their software that this upgrade was going to cause and that did work so the the name was kind of a way to sell the idea to all of these exchanges and external developers who are not part of the core team yep so like essentially like you know you set up calls and slack channels and telegrams groups with the relevant technical person and the relevant technical content at all of these different organizations i have an entire telegram folder of just exchanges that i'm tech supporting with stargate i have does i have lots of slack channels i have all kinds of different ways of communicating with these folks um and you start and then you run test nets

and you you communicate clearly we published i think like 15 weeks of weekly updates on the stargate launch repository where we would uh just like update everyone and we would tell people like and then we would just get people to report back so one of the things stargate was all about was you know so the ibc feature is the feature for the token holders it's the one big thing that for the token holders what mostly started it was about was changing the way cosmos works from a developer point of view and one of the other things that we were doing is we maintained a backwards compatibility layer between the new way that transactions work and the new way the box works cosmos and the old way and basically a lot of this process was essentially

working with the developers of various exchanges on like a trial by fire basis to get them all to tell us what is broken about like what this new software and the compatibility layer it doesn't work for them and then going to the cosmos dev teams and being like okay like this thing is not working uh exchanges need it uh uh such and such large exchange needs it uh can we fix this bug get that bug out get a new test that launched get the exchange test against the new testament get to the next bug fix that down and the plan was basically that we were gonna land all of the exchanges and their integration issues and the software being qa'd and ready

and all of these pieces together on basically the same day and that's basically what we did this is kind of interesting for me because uh at suma we did a significant amount of work on a cosmos sdk module and we are currently in the process of upgrading that to the new version of the sdk yeah and just to take a relatively simple extension to the cosmos ecosystem and upgrade it is taking you know probably a few dozen hours yup it is not surprising to me to hear that exchanges are spending hundreds of hours on this because it is much more far-reaching through the tech stack than any other hard fork i've seen yep that is the reality of the situation

basically i probably hadn't really like looked very closely at was going on in the cosmos code base for like four or five months because i've been dealing with both like trying to sort of set strategic direction inside of tendermint and then leaving tendermint and then reorganizing the entire cosmos ecosystem um and trying to make that all that work and so game of zones happened i sort of started to look at like what is actually going on here um and i realized oh you guys have decided to break everything um and for all like good reasons that will make a huge difference in the future um and like all of this stuff is like reasonable and logical and makes a lot of sense and you know any point in time in which we'd

waited even longer to make these changes they would have been harder to do um so you know all in all it was like i'm not expressing remotely any regret into it but the stargate upgrade was this immense lift to try to get a very widely integrated very complex blockchain that has lots and lots of different integrations the other thing about cosmos right is it was cosmos never really participated in the listing process with uh you know the team or tendermint or no one ever really like went out and got cosmos listed on any exchange so every exchange permissionlessly decided to integrate cosmos and just every single exchange came up with their own completely unique integration which has been so much fun to support

[Laughter] oh my gosh this sounds like such a huge endeavor let's do something for our listeners though and and also for me actually would you define stargate as an upgrade like one upgrade or is it a set of upgrades like do you bundle this all into one thing and i guess the other question is is it over that was the reality of it is it was over there is a point in which like you go through the stargate right you're on one side it was uh you know the name has really has really turned out to work really really well like that visual metaphor of i'm on one side of the stargate the stargate opens i go through it and i'm on the other side

i don't know how many people remember the tv show it's on netflix now so there's like lots of people who can watch it yeah i mean i love that tv show uh uh so much growing up but like yeah this like metaphor of there's like a membrane and you're on the other side uh and everything is different on the other side is like extremely accurate so are we on the other side yeah we are on the other side today okay so let's let's go into this so back to that question of is it one upgrade or a series of upgrades yeah it was one upgrade it was one event that was the realization that like all of these changes that people were want that had made or were in the process of making had to be bundled together into one big painful change that all had to be

done at once because everything was so tightly coupled together that there was no other way so there were a number of major features that needed to be added or refactoring or things that needed to be changed and the decision was to bundle them all into one extremely painful upgrade and go through it all at once right yep what makes it painful i want to understand this like you describe like the exchanges having to put all of these hours in but like what does one of these exchanges actually do don't they just kind of like take new software that already exists and just like spin it up that is not in general what an exchange looks like uh integration looks like uh and i uh it makes sense to me that like very

few people are familiar with this so the first thing you can do is if you wanted to understand what an exchange integration looks like is you know coinbase has has done this rosetta standard right which is essentially attempting to like formalize and standardize the interface between a cryptocurrency and exchange and a blockchain piece of software and you know cosmos is working on being integrated into having a rosetta integration but if you look at all the exchanges where like the scope of what's in rosetta is the scope of what an exchange has to do they have to be able to like send and receive funds manage keys manage usually hot and cold wallets move funds back and forth between them many exchanges are integrated into

proof-of-stake systems so they need to be able to have the like staking rewards on bonding and as your blockchain becomes more featureful there are more opportunities for exchanges to integrate the features of your blockchain into their into their software and exchanges may choose to do anything from use the queries and rpc calls of your network uh that like your your node provides to interact with the network you use the api that you're notified to interactive network to like write thousands of lines of code directly in the cosmos sdk to interact with the network again every single exchange pretty much has a unique uh

version of this the world of all exchanges just used rosetta sounds like amazing and magical and fantastic um and would make things like something like the stargate upgrade a lot easier because there would only be one api surface area that you had to to like migrate over the upgrade to um but i did not get to experience that world i got to experience the world of everybody has a unique upgrade and there are over 100 exchanges that list atoms i have different relationships with them and i had to help them all with like random troubleshooting problems and i'm probably still helping a bunch of them today this is really interesting to me uh working on some of these core blockchain clients like geth and gaia

and you know cello i feel like we're targeting these at specific users people who are you know independent devs running them at home or in the cloud and exchanges are so far outside of that uh class of users they have completely different needs for a blockchain node so uh it sounds like the pain point here is that gaia itself is not serving the exchange's needs and they have all resolved that in different ways right and so you and the other you know gaia and cosmos developers have to go out to each exchange and help them you know update their bespoke setup yep ouch uh that sounds incredibly painful

yeah let's talk about like what's in it what is it doing i had chris goes on the show i think or about a year ago maybe maybe it was a year and a half ago where he talked about ibc so i know that ibc is a huge component ibc meaning like basically the ability for these zones to actually interoperate thus one of the keys for cosmos to be an internet of blocks exactly for it to become what it's sort of set out to be it's a huge thing but is there is there something else to this like are there other kind of key changes that would even be recognizable to an outside viewer yeah so most of these changes are designed to improve quality of life not really for like end

users of our software like atom holders etc like ibc is probably the most like adam holder facing feature of this whole system um there's been there are like a couple of other minor things that are like related to like how the community pool can work and stuff like that but like ibc is the big feature for for for token holders okay really what the focus of stargate was is this like totally reconceiving the developer experience and the node operator experience of running the uh cosmos software stack so virtually i mean this is this is very much in the weeds but when you write a blockchain

a big part of writing a blockchain is how data and transactions are actually written to the network and read from the network and written to disk what are the transactions that are actually in the blocks what do the blocks look like and virtually every blockchain protocol uh cosmos had an in-house homegrown solution to what did the transactions actually look like as bites on the wire what do the blocks actually look like and this homegrown solution was incredible pain point for developers and the goal like the biggest thing in stargate was we are going to reconceive this homegrown solution

into something that uses a standard format that is widely used in the entire computing industry and is no longer a unique thing to blockchains and it kind of represents the overall goal of the direction of the cosmos sort of developer community to make cosmos look less and less like a special unique blockchain snowflake um produced by some weirdos and more like a standard piece of internet infrastructure and that change that change was the thing that like is the reason why i had to talk to all of the exchanges and all the wallets and uh all of the custodians and and tell them because they had what they had done is they'd all figured out their own ways of

dealing with our like unique blockchain snowflake software got it and the other thing that we did is that was extremely painful was we changed the entire inner workings so every transaction that's on the network all different but we made maintained a backwards compatible interface for all of that that is like 99 backwards compatible um with the old version allowing people to sometimes not have to do all of this upgrade all at once but you know like what part of the reality is if you build something that is 99 compatible um you're going to run into the

situation where like the one percent that's not compatible is gonna break interfaces for someone um and then you're gonna have to deal with it getting into you know kind of how transactions and blocks are stored and transmitted between nodes i have a bit of a personal anecdote here early on in my cosmos sdk development experience we were at a hackathon trying to put together you know a very simple cosmos module and we kept running into an issue where a specific transaction would always fail and it wouldn't just fail during processing it would fail signature checks on the node and we couldn't figure out why because we were building it and signing it just the same as all the

others what it turned out is that the client was sending an empty list and when it went through the serialization and deserialization the node was receiving instead of the empty list uh nothing so some step in there was taking what i had sent and signed which was the empty list and replacing it with uh nothingness and that's why the signature was no longer working is because the serialization and deserialization actually changed what we had sent so i have two responses to this one is like this is weird edge cases um is why it's better to use internet standard software uh so we've

switched from an amino to protobuf which everyone uses but the nightmare of stargate is is that we have maintained a lot of amino backwards compatibility so there's actually a translation layer in the stargate software between the new protobuf stuff uh and the amino stuff that has to be one for one bug compatible with the old veto stuff and like somehow this works i'm so sorry [Laughter] and like really it goes to like aaron and the regen team for like pulling off this miracle it's pretty amazing that we this part of it went through the stargate um and we succeeded and it was like it was painful and it

was a lot of work and a lot of hours but also pretty amazing that like we were able to pull this off yeah definitely it's a really impressive bit of work and um i i have to say that the new protobuf stuff is much nicer to work with and i am a huge fan of outsourcing maintenance of this to google yes i kind of wonder with the stargate upgrade like so far i think of this as like it's focused mainly on the cosmos hub but is it also across all the zones that have been built using the cosmos sdk this is what i kind of want to understand like is it did they all have to upgrade something as well so in order for them to participate in ibc to start sending tokens

participate in the internet of blockchains they need to upgrade to stargate i see so that's that's one piece of the story a number of zones have either upgraded to stargate like akash has upgraded to stargate um or announced that they are going to upgrade to stargate um there's quite a lot of demand right now for engineers who can do stargate upgrades on cosmos modules so there's quite a lot of consulting going on doing that um like in in the the work that james is doing upgrading his uh his cosmos module to stargate so like in order to get the internet of blockchains you got to do start date and like by doing the cosmos and doing the cosmo stuff first we like paved the way right yeah every exchange that has an integration their integration with the

hub is how they was the basis for their integrations with all of the other chains got it the other thing about stargate is like really like once we went through this and we're not going back the the cosmos dev teams are only like really building stuff for the post stargate world if people want the latest improvements to tendermint in the cosmos stack they gotta upgrade to stargate too the other realities of it is right it's like in general like all of development slowed down across the cosmos ecosystem um because the stargate upgrade was ongoing uh and people either didn't want to build stuff knowing that they would have to do the stargate upgrade later or like didn't want to build stuff on top of half completed versions of stargate

which was probably not a bad bad decision for most people and so now that we're on the other side of stargate is actually a lot more scope for building stuff on cosmos it's exciting times in the cosmos ecosystem then zaki thank you so much for the interview and giving us your perspective on the stargate upgrade yeah thanks guys [Music] so now we're sitting with tess rainierson who's the vp of engineering at interchain game beja and product owner for tendermint core welcome to the show tess hi thank you so much i'd love to find out a little bit and also i think to give some context to our audience what exactly

do you do as vp of engineering at interchain and how do you exactly interface with the stargate upgrade yeah totally um that's a that's a good question and definitely merits a little bit of explanation so um at interchain gambeha we focus on basically like three pillars of core technology for cosmos one of them is tendermint core which is the consensus engine um one of them is ibc which you know everyone listening to this knows what ibc is and then there's also gaia which is like the the daemon that actually runs on the cosmos hub so it's like the software that kind of ties you know the cosmos sdk together for the cosmos hub specifically and i actually spend like

90 or 95 of my time focused on tendermint core so i'm you know i said i'm the product owner for it i'm the engineering manager for that team i also you know sort of do manage um the managers of the other teams that entertain gambija and sort of like work on general i don't know engineering management practice for lack of a better word for the company but but i actually really focus mostly on tenement core and that's like where my you know technical passions and interests lie for people a little less familiar with the architecture would you mind talking a little bit about the uh separation between tendermint core and gaia or the cosmos sdk yeah totally so i mean if you think of them as like an infrastructure stack basically which is

it's an imperfect analogy but there is sort of this dependency direction to them if you think of them as a stack tendermint core is at the bottom it is a algorithm for state machine replication usually when we say tendermint we're talking about like the algorithm which is you know this proof-of-stake consensus algorithm that also borrows heavily from academic distributed consensus algorithms and that's actually kind of why i like it it's like this kind of funny very cool like hybrid of sort of these ideas from blockchains and these like incentivized systems and then also this very academic like classic distributed systems thinking so that's one of the things i really like about it so it's this almost agnostic protocol for replicating state in this way you know you don't

necessarily even really need like a blockchain or a cryptocurrency to use something like tendermint so that's that's at the bottom and it just lets you again replicate state from one node to another in a network and then you have the cosmos sdk which actually provides like an application layer on top of tendermint so most people who are building you know cosmos blockchains are going to use the cosmos sdk um there are some other blockchain applications that use tendermint directly but uh you know for the sake of this conversation i'll focus on the cosmos side of things so cosmos is like one layer up and you know the analogy that people sometimes like to make with cosmos an aspirational analogy in many ways but people like to say

it's like rails for blockchains it's really meant to be this framework this tooling that makes it really easy for anyone to spin up a blockchain pretty quickly you know again a little bit aspirational but that's uh that's the goal and i think it's a a good one um and then gaia like i said it's it's the daemon so it's like literally like the software process that runs on the cosmos hub so if you are trying to join the cosmos hub you need to run an instance of gaia and gaia is built using the cosmos sdk so you have this like this like set of layers going up almost where like tendermints at the bottom and then there's the cosmos sdk and then there's gaia now this does beg the question like where does ibc fit into all of this

right i was just about to ask yeah so yeah it's the natural uh the natural thing to wonder next and so you know i'll i'll channel chris goes here for a moment he's sort of the original author of ibc and i'll point out that that ibc is really like a blockchain network agnostic protocol so when we talk about ibc as a protocol we're really talking about the specification the paper you know the protocol that has to be spoken and not the implementation but the way that ibc is implemented for the cosmos network is in the cosmos sdk so there's a module the cosmis sdk is comprised of all of

these modules that let you do different things whether that's staking or i think there's like accounts account management or i think there's even like different consensus modules if you don't want to use tendermint for some reason um you know breaks my heart but uh i understand um and anyway there's an ibc module within the cosmos sdk and that is where ibc for the cosmos hub is is implemented again that distinction is a little bit pedantic but i think chris would be upset with me if i didn't say it so i'll make that really clear i suspect he would i kind of want to go back i have just another question about gaia i just want to take a quick step back is that the client

or is the client in it somehow like is it is it a version of the sdk or is it a specific client software this is what i'm not saying um i think it depends what you mean by the client i guess i'm thinking always in like the ethereum context of like you're running a geth client you're running a node with this particular software i think it is if you want to both know more about guys specifically and also understand the comparisons to ethereum i know you're talking to shahan next and he both like comes from the ethereum world and works all the time on guys so he he's your guy there like he will he will he will tell you conclusively yeah sounds good okay can i can i take a shot

at an analogy here go for it please you know tendermint core is the engine you know at the center of this machine cosmos sdk is going to be how you uh you know get the gas in and the energy or movement out of the engine and then gaia is the actual like machine that we've built yeah i think that's a pretty a pretty good analogy it's it's not uh perfect needs work maybe by the end of this episode you'll have it yeah we can reshoot later take this part okay um so let's let's now kind of turn our attention to stargate and how like we learned just before

from zucky how kind of his perspective on what this upgrade looked like and the kinds of work he had to do but i'm really curious from where you're sitting like what did this upgrade actually mean yeah so you know i think especially coming from sort of like the tendermint core perspective like kind of the lower level core engineer side of things stargate really represented like the culmination of probably a year and a half of work to prepare the cosmos infrastructure for ibc so you know i think as some people will know the cosmos hub had launched in like early 2019 and it launched you know as many systems launched as

sort of like a like a first draft right there were a lot of things that were like everyone's best guess of how the system should work and how it should be laid out and what encoding algorithm it should use for things and how you know how to serialize data or what data structures should be used and um you know some of those things turned out to like not necessarily have been the right choice which is just how this goes that said you know because this is a blockchain it's actually very difficult sometimes to make these changes if we were working in a system that you know had mutability or didn't need to have this like long immutable consistent history things would be a little bit easier to change but with blockchains you know often you have to to treat these kinds of changes as huge

breaking changes and so that i think is kind of the reason that stargate became a whole thing right is like is like the core teams had to kind of rethink a few of the assumptions that turned out to like maybe not be as usable as they'd hoped or like the performance wasn't there and you know again this is very natural part of software evolution but i think one of the things that's like kind of novel for blockchains versus like more traditional software development is that you have to figure out things like how to get around this immutability issue and so i think that's kind of like i think it was it was actually it was zucky's idea to say you know we should really brand this launch because we're going to need so many people on board and you know all the clients are gonna break and all the

wallets are gonna break and every valid it's gonna be you know a huge thing for validators and i think a lot of that was really driven by this uh change to the serialization format so basically the first version of cosmos and tendermint and all of that used like an in-house encoding algorithm for serializing data and it just wasn't as performant or as usable as we wanted it to be and so um you know there was basically a huge migration process that took a really long time um and it took you know many smart people working for many many hours um to make the switch over and to do it in a way that wouldn't be breaking and so that i think you know again this is like

really really biased as someone who's like working on like kind of low level the stuff that's kind of at the bottom of the stack but to me that was actually like the motivating thing for the even the branding of it as stargate and and and all of that it was like this is the thing that's so breaking we know it's really important because this is like so foundational but it's gonna be so breaking we can't just like rip the band-aid off or like pull the rug out from everyone this is actually gonna become something where we need everyone to buy into it together and the amazing thing is that you know people like zucky and and i think chiefly zaki were able to go around and really rally that support from you know not only all of the development teams across the ecosystem but you know partners and wallets and

exchanges and do all of that work um so yeah i mean it's kind of like the most boring or kind of this this huge branded release um but from where i sit that that really was one of the biggest if not the biggest motivations for this whole thing okay i have a couple questions about this um first was the idea behind branding it that you know you couldn't rally companies and the ecosystem around a numbered release and so we had to make this something special that people would get behind you know i don't want to say it's like oh people couldn't get excited about a numbered release because that makes it sound like people are frivolous or something i think it's more that it's just confusing like it's it's better to have like

like something that's that's tangible and like frankly we had to change the numbered release that we thought it was going to be a couple times and and and also it was useful to have something that was like a like a moment that we could point to and we could say like this is the tendermint core version that's going to support stargate this is the cosmos sdk version that's going to support stargate and this is the gaia version those numbers are all different so if you can kind of package them all up and say this is you know this is the stargate release bundle effectively um you know this is the set of software that we will run when we all go through the stargate together you know if you can do that it's just an easier story to tell because it touches so many pieces it's

not just you know a couple releases here and there it's a you know ongoing project totally one one thing we didn't get a chance to talk about with zucky was kind of the because what we talked about mostly with him was that he was doing a lot of communicating exactly this thing you were saying of like sending the message out getting everybody together but i'm kind of curious on the timeline like was the work already done had a lot of the build been finished and then there was this period of communication or was it like communicated and then you had to work like i'm kind of curious how that happened there was some overlap so i joined the project in like late fall 2019 which is to say i haven't been with it for that long and you know at the time a lot of these

changes had actually already been decided but it wasn't really clear how they were going to actually end up on the hub so like people knew they wanted to change the serialization format or like they knew that you know there was a path forward for ibc or like there were a few major tenderness core features that like we knew we wanted to do so there kind of was a path there and the question big question mark was like what does it actually look like to get this onto the hub and we had been writing the software and doing the development for several months before the communications started but the comms phase that zucky sort of ran and managed definitely began before the software was finished so like he was working on rallying the troops

starting in like july or august of last year i think if he said something different he's he's gonna remember more clearly than i did but it was it was last summer and then you know at that time like the tendermint core team and again because it's the stack the tournament core team has to finish their work first right again you can you can start building on top of it but you can't you have to cut the releases in order so so the tenement core team at that time was like mostly done with the work we were like wrapping up a few you know outstanding things and really we were running like putting out what we call release candidates so like things that we think will be the software that will run on the hub for sort of the next major tenement release that process lasted a few months and we

had lots of people try the rcs out find problems and then we go back and fix them and so i think it was like the first week of november or maybe the end of october when we actually cut that final release for stargate was the game of zones kind of during that era or was it after or amazon's actually a little bit before okay um and the reason that we could do that i think was that um that was really testing like the ibc piece of it specifically which didn't depend on like fully ratified changes like all the way down the stack if that makes sense you could kind of test that piece a little bit in isolation and that's what game of this game of zones is all about um that said there

were so many later test nets that happened you know maybe with slightly smaller prizes or a little bit less advertised but like many many test nets were run you know leading up to stargate that zucky and and others other teams also managed and those were instrumental in in finding bugs and and helping us get stuff fixed so yeah interesting um you mentioned that uh you know there was a lot of people feeling like they would eventually move away from amino or you know people wanted to do this and i remember from being involved in cosmos a few years back that everybody disliked amino but nobody was willing to do anything about it would you say there was like a specific person or event

that you know turned this from mild distaste to actual action on a big scale um i don't know that there was any single person or event you know i will say that this work when i joined the project and i joined the project again to kind of like lead tendermint core when i joined the project that feeling was very very strong and the work had already begun to to remove amino and replace it with protocol buffers so that was not my decision you know i also stood by the team in that decision and i was like this is something we're going forward with this was not something that every single person in the the ecosystem wanted and there was a there was a tendermint fork briefly called tendermint classic that

had um amino in it still i i don't think that project is still under development but like i'll just be honest this is not like a universally beloved decision um but it was something the team really wanted to do and i know a lot of users really wanted it too and so you know i i think it was it was helpful to have someone who was at least nominally in a leadership position i was like brand new to the project so like what what did i know but like i was nominally in a leadership position and i said you know i really trust this team and i trust the way they want to execute on this and so i think that that helped you know around the same time that that decision really was ratified that was also the time that interchange gmbh was formed

and that was another thing too it's like it's like interchanging mbh you know it's it's it's a safe space and for for core engineering that's that's the joke and it's a it's a place where people can make decisions like that that might be a little bit politically unpopular but like technically correct um obviously i'm biased when i say that also but like that's that's really what it's meant to be it's like it's like we're kind of liberated from other initiatives that are happening in the ecosystem and we can really just try to focus on building like the best core technology that's going to serve as many people as possible that and that's that's the mandate and that's really why entertain gambeha exists that's a great mandate thanks what you just described you said this

there was this like very short fork but like was that fork when you did the stargate upgrade or was that fork when you did something else in the past because i thought it was all bundled into this one thing that was something that i won't beat around the bush here um this is something that jaquan the original author of tournament wanted to keep using amino and um so he forked the project you know i think at the point in which it became clear that this amino removal initiative was going to proceed and so i don't remember exactly when that happened i think it was maybe about a year ago i'm not i'm not totally sure there's no bad blood there or hard feelings this is just one of those things that happens right in blockchain development like sometimes you have a political

disagreement or like a technical disagreement and someone forks the the code base um and sometimes those things reconcile again later and sometimes they don't in this case you know i think from a community perspective there wasn't an enormous amount of contention and again i think it also really speaks to organizing skill and also the strength of the cosmos community that there wasn't any confusion there right people were like no we know what you want we know that we want to move forward with with stargate this is not a big a big issue in any way um but i i just mentioned this just to say you know like sometimes there are technical disagreements and and this is how this goes i think when you said fork i thought forked the ball oh but you just got it yeah the word fork is is

so overloaded overloaded yeah there's like yeah i'm just glad that i work in a system where we don't get the forks that are like an accident of consensus and these like probabilistic like proof-of-work type type chains so at least we don't have to worry about that kind of work no i just meant a fork of the of the repository a fork of the code of the project um got it are you worried about consensus failures between uh you know like the go implementation and rest one if that ever gets completed you know i think i think the rust implementation has been a great forcing factor for the whole project to get really really scrupulous about the specification and fortunately there's a lot of other reasons to also be scrupulous about the

specification i mean there's the usual like like the table stakes like if you have a good spec then you're gonna have better code like you know that's just table stakes but there's also all this work that informal systems is doing to actually formally specify tendermint and then like formally verify that specification um so there's lots of reasons to be you know to have an implementation and a spec that are very true to each other um and so for that reason because that work is happening i'm less worried about the breast implementation and the implementation diverging from each other you know it is a risk um but there's always risk in these consensus systems that something breaks consent like that is that is our one job right is preventing that

and so um you know it's something that we'll be careful about but not i'm not super worried about it at the moment interesting um it's a extremely difficult job uh one of the recent geth ones was caused by a shallow copy deep copy optimization yeah it's there's so many ways to screw things up yes there are one thing that i'm still curious about is like during this upgrade the actual upgrade time like what was that like in a way like what was that like in general like how how was that really rolled out and also what was it like for you what were you actually doing oh man so it was extremely stressful i'm not gonna

sugarcoat it i mean it was one of those things where like i knew that it was potentially going to be all hands on deck and that people are going to be standing by for it the scheduled upgrade time was also like 6 a.m my time which is just like a kind of a brutal i mean it's a global community it's a global project i think it was timed for the convenience of the exchanges in asia explicitly um for some reason i don't know all the details there but like you know it's a it's a political decision actually to decide when the upgrade goes um and i drew the short stick basically but um but so it's kind of this this tricky timing and you know i'll say from a tendermint core side it's like our software had been pretty much stabilized and

had been you know we cut the the stargate version of tendermint back in november and the upgrade was in february okay um and what was actually great about that is that it gave the software some time to mature and like i said there are non-cosmos users of tendermint and some of them actually started using you know the stargate version of tendermint before cosmos did and some of them found some bugs which was great and we were able to work with them to get those bugs fixed um so i'll give a particular shout out here to crypto.com their team surfaced uh security issue and helped us test it and they were just like fantastic partners so they were they were awesome anyway so you know some some things have been serviced but on the whole

it's like uh the software's been around for a few months like it seems pretty stable it's pretty okay and then at about 11 30 p.m the night before the 6 a.m upgrade i got a message from someone that said hey i submitted a security issue to your hacker one and it looks like it's stuck in triage i just wanted to let you know that my private keys are all over my logs for his validators like his consensus keys they'll let you like you know sign blocks and also like if you misuse them the validator gets punished we're being leaked all over his logs and logs you know sometimes get shipped off to log providers like a log is not a secure place and i was like oh oh god and what had happened was that

you know like i said the cosmos sdk is a downstream dependency of tendermint core and i'm gonna get a little bit detailed here but it's like it's like i think it'll be interesting to any programmer um what had happened is that the cosmos sdk changed their logging library um to something that when you passed in an object would use reflection to look at the like internals of the object while tendermint core was using a logging library that assumed that you'd call the tostring method on any object and you get the string representation so we had in the tenement core code base this particular private key type and the one that we were using is like the or the one that was a problem is like local key storage it's like no one

really should be using in production so like probably like it's not super secure anyway so probably not that big of a deal but just like a really bad vibe and look to have these keys here well so so so what had happened is that tendermint core expected you know this thing to be stringified and the sdk was actually looking at the object itself and the string the two-string method entendermint would strip out the private key material but if you look at the object itself you can get that private key material and so that's what had happened it was this discrepancy between the logging libraries which is like the most mundane thing you can think of but it had this really ugly effect like really it just like looked catastrophic even if again you probably shouldn't be using this type of key

this file storage for your keys in production you know we were like probably someone is you know if you tell if you tell programmers not to do something they're gonna do it and so basically you know with with about six or seven hours on the clock i was like all right we gotta we gotta figure this out and it wasn't obvious at that point when i saw this problem that the problem was the logging library thing it was just like something is wrong and so fortunately i was able to rope in a few folks you know um uh alexander bez who is known as bez he's a long time cosmos contributor who's working on tundra core now he's in new york so he wasn't asleep yet so i basically called him up and i was like what's going on and he and i looked

at it um and um you know he has worked on both the cosmos and the tenerman sides and so he was the one who who was able to diagnose it because he knew both code bases really well and then we like tried to cut a new release and there had been a go release that week which broke all of our release tooling so like where like our release tooling expected a different version of go than we were using because a new version of go had been released two days earlier and it was just this like series of unfortunate events basically that unfolded at like three or four in the morning like two hours before the release happened you know it's wild you can just you feel how much like as much as it's a technical thing there is kind of like

a right way that this works but it is this emotional thing as well where it's like if things like this happen it just throws off the entire process it sounds like you can't just push through that stuff because it will hurt it will do something totally and so like i was up really late alessio who works at tendermint inc it was up really late shahan was up super late well i guess it wasn't super late for him yet but he still had to get up really early the next day to deal with like all this stuff and so i basically got the release done at like 4 30 in the morning and then i went to bed and then i woke up at six to just like see if the release had started and i was like too tired to function so

i went back to sleep and then there was like a um we did a little uh launch party which was super fun but that was like a couple hours later i got up for that and then i went back to bed which is all to say that i pretty much slept through the actual stargate release so i i i um i had a lot of stuff to deal with right up until that point um and then it was actually amazing to look back on because the validators and the community members were so incredibly supportive of each other and of helping each other out and you know there were people from core teams who were standing by and helping helping out with things but there were a number of people who are community members and who are

running validators who are just like helping each other out answering questions i don't actually know the details of this again partly because i slept through it but there were some things that just took longer with the upgrade than expected and i think sean will be able to speak to that but you know people were really able to help each other out and say like oh this is what worked for me or like oh it took me two hours to get through this phase like just hang in there and that was really really amazing to see that's an incredible story and you know i have to say that i have literally had nightmares about dealing with the go module release [Laughter] so i am so sorry that you had to go through that you know at midnight right before a major loss yeah well the

the sort of it is funny because the the story is you know the go module system like blockchains has a certain aspect of immutability to it and so if you look at our change logs and our go modules and all of that you can actually kind of see the history of like there's a couple um release attempts that that didn't happen and so tendermint o346 and o347 are just like no op releases like they just don't actually exist they're they're artifacts of a failed build process um or failed release process the story you just told that was the actual night of the actual upgrade but the upgrade had also been sort of delayed that was a different thing right that was two weeks prior to that yeah so one of

the things that you know and i don't remember the exact cause of that delay but you know the way that these upgrades get scheduled is someone has to submit a governance proposal and then there's like a voting period for the governance proposal and then two weeks after the governance proposal goes through then the upgrade is actually like scheduled to happen basically and i think that the first governance proposal went out and even in that first proposal at some point while it was live and we were waiting for the release that was when that security issue that crypto.com found and helped the tenant corps team out with that was when that surfaced so there was like a

like a date scheduled and the thing about these uh these scheduled upgrades is that they point to a specific commit hash on gaia so everyone's committed to running like a specific version of the software and if you change it that's like that's like not part of the governance you know that's not what people voted for right um in the case of that particular tendermint core issue that i just described we actually were able to make the change for that in such a way that like you could still run the commit hash in the proposal and it would be compatible with the patched version and so we told people

you know you really should upgrade to the patch version and you know you you can run the one in the proposal if you want to verify that that's what's happening you can do that but we recommend you run this patched one so that kind of seemed fine and then i think there was another issue that emerged i think in cosmos sdk or guy itself i i don't remember the details of this one but that was one where we could not get around it in that way and so that was why it got rescheduled or it wasn't just that there was a bug it was that there was a bug that was where patching the bug was going to be incompatible with the um scheduled version got it and so it was at that point that we said okay we need to restart this process well this is such i mean thank you so

much for sharing this story of your experience kind of going through the stargate and going through this upgrade our next interview is with shahan and we hope that in that interview we can actually hear kind of a little bit more on that exact time of the upgrade and what's happened since thanks so much for your part of the story yeah my pleasure congrats thank you yeah yeah this is super super fun thank you so much for having me cool [Music] so i want to welcome our last guest of this three-part interview series shahan who's the cosmos hub lead welcome to the show shahan thank you for inviting me to our

podcast i'm very excited to chat with you today we're very excited to have you so we've had an interview with zucky who shared his perspective on what was happening like from the perspective of inclusion mainly we heard from tess about the actual like stargate launch day in some detail now what we want to do with this portion of the episode is to hear from you as the cosmos hub lead what it was like for you to go through it and also hear a little bit about what's happened since but before we start into that um maybe just say a few words about what you actually work on and what being the cosmos hub lead actually means sure uh as in my role as cosmos hub lead i'm

responsible for communicating and gaining agreement on the hub roadmap from the diverse dev teams and the validators that participate with the hub so i'm new to the role it's it was around three months four months when the stargate upgrade was taking place so i was quite new to the ecosystem and what that meant was communicating a lot with the dev teams at the different layers the tenderman team the cosmos sdk team and those who had been managing the gaia application over the years and trying to understand what exactly the upgrade required the upgrade itself was quite straightforward

on paper and the most complex equation of this upgrade was the communication with the validators with the dev teams making sure everything was in sync and making sure that everybody knew what to do when this stargate upgrade was quite intense and required the stopping of the chain and it was a coordinated effort you're pretty new to this role like you started and then very soon after we're faced with this upgrade yeah um what was that like when did you actually start my uh start date was end of november beginning of december wow uh the first uh two months of my onboarding was essentially

talk to as many people as i could and go through the stack prior to joining cosmos i was uh aware of the tendermint uh consensus algorithm so i had a strong understanding of that but it was the rest of the stack which was a very deep and very interesting scan through code at the same time this stargate upgrade was a migration of old quote to new code so it's very interesting exploration trying to figure out which parts were old which parts were new uh and and trying to answer the questions was very hard for me but essentially my focus was on the upgrade itself and the coordination around that with and

the communication making sure that all parties were aware of what their responsibilities were and making sure that if there were any issues they would be addressed by the right parties i mean that's a really interesting position to be in you've just joined the ecosystem you have almost no idea what's going on you're trying to catch up to speed really quickly and uh your job is to coordinate everyone all at once yeah did you know what you were getting into yeah um my background is from the ethereum land okay i have a lot of expertise with enterprise blockchains and ethereum stuff but what is different here is that the

cosmos ecosystem runs a different form of consensus algorithm so the way that the network upgrades is different than how things proceed in ethereum land and what i enjoy about this community is that everybody is on board and in tune and there is an on-chain governance mechanism that people participate in so there's a lot of activity and involvement from the validators in the development and the roadmap in agreeing with what's coming up with the hub so the social coordination mechanism and the on-chain governance they work very well together to signal what's coming up and introduces a fairly straightforward

process amongst the parties when you say you're the cosmos hub lead like we we learned from tess about the different layers of the stack and the fact that like gaia is sort of the and we were kind of confused about like whether you call that a client or if it's a interpretation i don't know and maybe you can explain a little bit exactly what is the cosmos hub and how should we understand that i had a great question uh this was probably a question i didn't even know to ask in my first month the cosmo stack as far as i can see is built up of the tendermint uh consensus layer which is essentially a way for validators to agree on

some data and the cosmos sdk is the layer on top which relies on tendermint to communicate the business somewhat business logic key stuff and then gaia imports the cosmos sdk and is actually the application or the client that runs the cosmos hub the current version of the cosmos hub so uh essentially tenderman is a consensus library that can operate for other business logic and cosmos sdk is typically the library that's used by partner chains in order to develop their own chains or zones but the hub also depends on it i guess like if you change

the cosmos sdk or add something to it the hub could potentially use that that's correct yeah when there is a change in the underlying libraries guy is essentially the downstream partner of these development teams and we do some coordination and uh when there is a new release guy also will tend to have a new release in the most recent weeks there have been a few uh there has been one upgrade uh security release that required some coordination in that manner and uh it's straightforward to do a release it just bumped the version and cut a new release right now a guy as a client or application has some functionality mainly for

on-chain governance for token transfers but what's coming up are some exciting modules like a bridge with ethereum and a decentralized exchange and including those modules into the hub will require increasing amounts amounts of coordination and that's when things will get interesting so the stargate upgrade as a pretty involved process was a good first step in that trial by fire training getting easier from here yeah trial by fire is probably the most hurt expression i've ever heard in the team yeah so now we understand the hub we understand how the hub interacts with the cosmos sdk but like what actually happened

during the stargate upgrade like from where you were sitting were you just like updating gaia the whole time or what was it exactly um so i was just sitting and watching really um so as you heard from tess that there was a security patch issued and guy applied an update that included that and uh once the release was available the the nodes were then informed that this is the version that they would have to update to the actual process by the validators would be they stop their client they export their state they migrate their state they install the latest version of the

client and then they start their client so just a few commands really but the process took some time and i bet if you have enough people they'll find every way to do that wrong somehow am i right yes of course starting from reading or not reading documentation yeah it's not supposed to read the manual like a vacuum manual i mean this is really it's very obvious that you'd have the validators doing the upgrading and i know that that's how it worked but it's so interesting that like when you think of this hub it's not it's not like you updated anything that's how i had asked it and sorry about that it's completely like

every single validator going through the process so for your your work then is really like preparing all of that documentation and being ready to answer all the questions that's right yeah and being aware of potentially some contingencies so if if the version had to change as it did from the original governance proposal uh the original proposal had a particular commit hash of gaia which then changed and the original proposal couldn't change so there needed to be some way to communicate the version to update to of course people some folks would be who who are die hard on chain governance supporters would be like well i didn't

vote for that version so i'm going to stick to this version and of course people are free to do what they want but essentially the upgrade was very smooth it took some time because the migration itself some of the steps took a few hours and it also required some time zone coordination because the validators are spread globally so probably after uh five hours or six hours there was a majority over fifty percent of the validators that had upgraded and i think it took uh around six and a half hours for the 66 percent mark to be passed and that's what you were looking for i guess that was the threshold of

this one that's right yeah yep exactly but what happened to the folks that didn't upgrade are they still not upgraded don't think there's a way to find out which validators are running an old version but the old if any validators uh that didn't upgrade they wouldn't connect to the network so they're just not earning anything that's right yeah okay yeah they would be they might notice yeah maybe yeah that's right uh can you talk a little bit more about what makes this upgrade different from like an ether fork the typical hard fork process is that we choose a flag height at which all of the nodes smoothly switch over um what specifically made you know stargate such a big deal that

we had to stop the chain for six hours yeah it's a good question it it starts to get into the more technical aspect the tenderman algorithm is a bft style algorithm byzantine fault tolerant style algorithm and what that means is that the validators are known and in order for the consensus to change or the state to be able to change via this hard fork the validators the majority of those validators need to conduct their upgrade in ethereum land a hard fork is essentially trying to convince the validators to update their client

and if they choose not to then they can continue a separate chain and they will have uh there will be a chain split and some validators will be on the old fork and some will be on the new fork but when you say validators you're do you mean like miners in current ethereum not validators and future ethereum that's right yeah okay got it i guess they're i mean they play the same role so that's right yeah and uh it's a bit interesting because uh staking is available in in hub but it runs a bft style so it's not it is a proof-of-stake system but not like ethereum which runs a different consensus mechanism essentially casper versus pbft yeah yeah that's

right when we spoke to tess she shared this story of like how kind of hour by hour this process was going and there was like a bit of a hiccup or there at least optically there was something that looked strange i mean she kind of mentioned you you were kind of in it with her or you were you're also online at this time what was it like from your perspective what was happening from my perspective i saw a lot of devs diving into code trying to resolve the issue and they got that figured out very quickly they are true experts they are very familiar with the sdk and tendermint stack and so from my perspective i was just really waiting and looking for the latest

release of the tenderman or cosmos sdk libraries once that release took place it was essentially a few steps to cut a new gaia release to bump the versions and then issue a new version after that it was essentially communicating that version to the validators so we have a verified validators channel on discord where validators were asked to keep watch on for events like these and once that took place we were essentially waiting for the moment time to stop the blockchain perhaps optimization we can do next time is for the stargate upgrade we had selected a

time based halt versus a block height based halt and we were trying to do that so that the time-based coordination would work out more easily so that we would know which time zones would be most affected or not and make sure there are enough people online because with a blog based uh halt it means that the time could drift so the side effect of choosing a time-based halt was that when the chain actually stopped there are a couple of validators that had already passed the time of halt and essentially there was this discussion of which block should be picked for the migration to occur at so there was a a very interesting

voting and social coordination of okay well uh there's this block that was past the hall tight and sunny pointed out very clearly okay any blocks after the hall height shouldn't be included in the migration so let's pick this number that was probably uh a key moment of the upgrade i i think this is such an interesting example of how difficult it is to even talk about time in a distributed system uh your software is running on a hundred nodes worldwide each of those computers is going to have their clock set slightly differently yeah yeah definitely um it was very interesting even with the halt height set to whatever it

was and the blocks still have a few seconds to get produced and it was very interesting to see that even with that a majority of validators did produce an additional block past the height the time okay i i was wondering about that like was anything lost in that upgrade like what could happen is is there like transactions that get lost sometimes or like what can happen that's a good question yes if there was a transaction that was issued in that last block uh when the when the hub was restarted those transactions would not be included in the in the migration wow can you imagine someone sent by accident tokens in a direction and then like magically they were back yeah that would be the best case

scenario of something like that but yeah yeah that's right uh well if that that was for a payment that was uh validated for only five seconds then yeah that would be a risk yeah what other things could have gone wrong i think one of the other challenges that some of the validators ran into was in order to have robust systems they would have a few nodes running and some nodes might be in front of their validator nodes so because of these architectural decisions that they've made and sometimes we have some clarity into the architectures trying to figure out which nodes should be upgraded first

which nodes don't have to be upgraded those types of instructions also had to be shared prior slash during the upgrade is this something where you had to do a little bit of a forensic examination to figure out what setup the validator was using and how to upgrade it safely we didn't do so much of that i think it was at least for the core devs it was fairly straightforward about what had to be upgraded at what point essentially if a node was acting as a validator it had to be updated but all other nodes did not necessarily have to be updated in that manner they didn't have to go through the export and migration uh they would just start with a new

binary and then continue and then they would sync on the new chain but given that with the current kind of security upgrade that happened recently and trying to improve the process and documentation for future upgrades there is a plan to have these playbooks for how to upgrade these validators what are common architectures that companies are using and providing more support as the number of validators increase and just giving them getting them used to this this whole experience it's fairly straightforward to run a validator in my opinion but of course i've not run one but there are some nuances involved like what happens when nodes fall out of sync

uh what kind of hardware requirements are needed what if we want systems to be more robust and of course the security aspect of key management in the future the aim of trying to do these upgrades as smoothly as possible is to try and integrate upcoming modules more regularly this stargate upgrade was probably a year and a half in the making involved many organizations and the future of the hub involves a similar set and a growing set there's development taking place these modules have to go through testing getting more awareness uh for the validators about what they need to do and how frequently they need

to do it would be very helpful this added functionality will increase the usage of the hub it'll make things a lot more exciting add feature [Laughter] so hopefully the future upgrades are not quite as painful as this that's right yeah we we are exploring different mechanisms for making these upgrades smoother and currently there are some mechanisms that try to automate some of this uh there is an upgrade module that allows somebody to issue a governance proposal which would update a client based on a particular commit hash there's also another mechanism which is

which is being used by the ibc module that's going to be in progress to being activated and what happens in that scenario is a module's code does get integrated into the cosmos stack either in the sdk or tenderman or guy eventually and once that code's present somebody can issue a governance proposal to issue a parameter change in order to activate that module so uh this way people have some awareness of and some mechanism to activate that module so what's happened since this upgrade we've kind of described it you also you just mentioned that the like ibc still needs to be activated

what does that actually mean and yeah what's happened since for you the ibc code essentially is part of the gaia stack now the gaia cosmos sdk stack which means that modules that rely on ibc will be able to use it and right now even though the code is part of gaiam what needed to happen was there needed to be an on-chain flag indicating to the validators that okay at this point in time ibc should be enabled for any modules that want it so essentially uh somebody submitted a governance per i think it was zaki and submitted a governance proposal onto the hub saying time to activate ibc

and people will have to vote on it and in two weeks of that vote the ibc module will look at that flag in the on-chain state and if that flag is set then ibc will be available very interesting uh so one of the outcomes of this big uh painful upgrade is that you've started building in more ways like this to upgrade specific parts of the system or almost turn on it's not even like you don't have to go through the process of upgrading anymore i guess right and that gives you a lot more flexibility for uh future upgrades to put them in the code base much sooner and then activate them when they're ready would would you expect that you

know this is the uh like long-term direction that cosmos upgradability is going to go it's more towards you know module-specific and you know very specific governance rather than big upgrades i think in terms of the stargate upgrade the majority of that those changes uh as i understand them were with respect to the underlying data format uh so it was an uh performance optimization it was uh other kinds of optimizations future upgrades could uh be as involved especially if state has to evolve in order to accommodate the new modules i wouldn't say that there's going to be one path for upgrades i think there are numerous ways for

upgrades to take place and they should be available depending on who's involved and what kind of coordination is required the coordinated hub upgrade we're trying to minimize those but at the same time if they do need to take place because of breaking changes we want them to take place much faster so trying to get the export faster and some of the updates that have been applied to the software since the upgrade makes that process much faster so it sounds like you know these big coordinated state upgrades are expensive but you know you're not scared of doing more of them uh yeah no i'm i'm not no i think these are

system level admin things that should be fairly well understood in particular we want the validators to be comfortable with upgrading we don't want them to think that oh i've i haven't touched this system in a year and a half i if i touch it now something's gonna happen we want them to feel confident in the process that of this coordinated mechanism and in some cases they're also uncoordinated mechanisms uh for instance i think it was a week or two ago there needed to be a security patch applied and it was required a 24-hour notice to the validators and that 24-hour notice was essentially within 24 hours a new version of guy will be released

please update your software and you don't have to export migrate you don't have to go through that process and it's uncoordinated so the validators don't have to turn off their software at the same time they can all do it at at their pace of course the sooner the better just to avoid the potential scenario where the chain holds in the scenario where the chain would halt if it had a majority of the validators would have had to update their client and then the network would come back up the chain would start back up you know we've talked a lot about adding modules to the hub and about this significant upgrade uh what kind of qa processes or what other

like processes do modules undergo in order to ensure that they're ready to run on a globally distributed consensus system a great question so uh we are working on a module readiness checklist that will help relay the information about the modules readiness to the community at large in addition to the technical qa that we would expect from anything being deployed on chain uh whether it's a third-party audit intensive performance testing uh there are other aspects such as who are the downstream partners impacted by this who are the upstream partners that this module is dependent on what kind of coordinations are required

as these modules get developed and we're working very hard to have this checklist in a very easy manner for module teams to fill out and this complements the technical checklist that some of the uh dev teams in the community are working on including region cool that's great well shahan thanks for coming on the show and sharing your experience of the stargate upgrade thank you very much for having me on here anna and james james i have a quick last question to you since we're wrapping up this episode like from what you've seen looking across different ecosystems over the years like how does this upgrade compare what what did you think about it now

also now that we've done these three interviews well you know outside of the major bitcoin and ethereum forks i think that this is by far the largest most distributed and best coordinated network upgrade that i've seen i i don't want to underestimate you know how much work goes into something like this and to talking to you know several dev teams and 100 validators is incredibly difficult sello you know my employer we're going to be undergoing a hard fork next month or uh you know late april early may and it is a uh you know extremely worrying

thing we're doing all of this testing and all of this qa and uh i am happy to know that it is possible and to see someone uh and you know teams that have gone through it with poise and come out the other side of the stargate huh come out the other side all right they are in good hands with you there james thank you thanks to both of you and to our previous guests for doing this episode and to our listeners thanks for [Music] listening