Juno chain halt inside story

Transcript

oh we're live we are live welcome to the interop uh the show which is all about understanding the decentralized economic networks that make up the interchange my hope is that my listeners will gain a better understanding of the networks that make up uh this the interchange sorry i had some uh i had echo in the background i was a little perturbed but yeah my go my hope is that my understanding my listeners will getting a better understanding of the topology of these now these networks the technologies that make them possible and the opportunities it provides to investors and developers and also sometimes we talk about drama that happens in the interchange and uh that's uh gonna be a good chunk of the conversation that we have today my name

is sebastian i'm here with jack zamplin who um [Music] um yeah i mean introduce yourself for those who don't uh who don't know you but like your your list of uh of um you know achievements and uh and sort of cred in the in the cosmos industry is like far too long for me to lift off the top of my head so um oh well sebastian that's flattering thank you i i really appreciate that um yeah my name is jack stamplin uh these days i'm sort of primarily the uh ceo of strangelove well no i guess i'm co-founder or shogun of technology you know it doesn't really matter uh strangelove is the the sort of company that i've centralized a lot of my

activities and um that started as a validator company with my friend tyler schmidt um and now all of my activity in sommelier in core one and sort of broadly in cosmos we we also were working with the icf to ship a number of features for the cosmos hub is is there uh we've got 12 employees now so um yeah i guess i've been the product guy in cosmos for a while that's kind of been my my thing yeah yeah um how do you how do you manage to have all these hats and still say stay sane and have a life assuming you have a life well i sleep i sleep eight hours a day which is i i think key

um and you know i i have a killer case of adhd that that keeps me switching between things continually so um yeah i think that those those are probably the two things oh cool um you know actually like it occurs to me that one like you've never been on epicenter i don't think you have i haven't no and we haven't actually had very many conversations like i know that we've met in real life a few times but it's yeah we haven't had really like a long-form conversation i think that was any longer than just like hey how you doing and like um yeah so um yeah it's good good to finally chat so um i i'm really excited as a long

time epicenter listener and you know i worked with sunny and sat with him while he's taped a number of those episodes love you guys podcast brian and the chorus one team obviously great friends um and yeah so i'm really excited to finally do this and i like this new format i am a huge live stream fan so uh this is great yeah this this is very intimidating for me so um the the whole live stream thing i'm not quite used to and and also like i don't know yeah i i think this is the format moving forward that i want to do with this because like stuff happens so quick in this ecosystem like i couldn't have recorded this episode and then like released it in two weeks from now it just wouldn't make any sense and so

it's more of a question of like getting used to getting used to doing content and releasing it right away and like being comfortable with it not being perfect and like the audio quality not being like super high def and you know like being mastered and all this stuff your studio setup looks fantastic by the way i i really like that um yeah well like my my uh my our audio engineer has been like really pushing me to have like really great audio audio quality for for all these years i i think i need to take a little bit of a step back for that in in in order to put some content out a little bit more frequently so we'll see how that goes um anyway so yeah give us a bit like uh give us a quick summary update um yeah yeah i want to i want to get that stuff

out of the way because there's lots to talk about regarding juno and um and then there's there's some other topics that i'd like to discuss with you but yeah what's going on with sommelier like so perhaps like first like what is some of the a for those yeah familiar with it and um where are you guys at in terms of like building that product yeah for sure so sommelier is an automated uh management for different d5 protocols um the first protocol that we're going to be launching with we've been through a few pivots on exactly what the first protocol we're going to be launching with uh is originally it was an impermanent loss for unison v2 then we were working on automated management for unisoft v3 positions which is a fascinating extremely interesting problem and

uh turns out automated rebalance is a guaranteed losing strategy um and we only found that out after uh sinking in of back testing uh and protocol work on that um so we have now pivoted uh the first seller that we're going to be launching um sellers are our version of vaults they are think of them as managed vaults and instead of being managed by like one key somewhere or this keeper network where you're trusting one keeper and you're incentivizing them not to steal all your money um it is managed by a full validator set and you have the same security as any other cosmos chain has which i think is a huge step up in terms of custody for defy applications that's kind of what i'm most excited about with

sommelier um launching a ton of these sellers and being able to make that sommelier chain a huge custodian for these different d5 protocols and the revenues for token holders insomnia will come from the revenues generated by those products the first one of those is going to be a seller on ave so this is uh users will interact with this solely through mainnet ethereum um they will you know go deposit into the sommelier contract on ethereum and then all of the management happens behind the scenes um and this is going to offer the highest stable coin yields available in ave and it will intelligently rebalance between them so very conservative strategy kind of easy to understand

with a really highly technical product where there's a cosmos chain behind it all kinds of stuff going on we thought that this would be the best way to launch it yeah that's interesting so i know that you there there was some live functionality i think at some point i tried to go on the website today like use it but it's it says now that it's launching soon so yeah has there been like a sort of pause in the product development in order to release some some new features like what's what's the status there yeah that that's a great question um you know our zachy and i's sort of development philosophy is just keep building and try to find customers for intermediate versions of

your product all along the way so there's a couple of big pieces to this we have the salinity smart contract architecture which is interesting and this whole big piece of thing there's many many billion dollar protocols built on top of solidity smart contract architecture so that that was one piece of what we wanted to do you know we have a team that's building this ave seller right now they're learning from and have worked on some of the best eat smart contracting protocols that's really cool there's this whole world of front-end development where you provide front-ends for these smart contracting experiences obviously when you have a experience that spans multiple chains that front-end development level of

difficulty becomes higher then we have the back end um sort of chain summary a chainside development that's been my specialty and we also have a data science team which is uh you know looking at the performance of some of these products that we're looking at over time and ensuring that they're going to be profitable for users so that's four different engineering disciplines at some and it's taken us a while to build up the team and build up the strength in all of those disciplines to be able to launch a final product but along the way we've been launching intermediate products the chain dev side that i've been a part of we launched the gravity bridge we've got a couple of people who are using our gravity bridge implementation um we worked very closely with the althea team to build the initial gravity bridge implementation

we've been very active on that side of things the front end team the first thing they built was the easiest way to add liquidity to unisoft v3 we took the product of our data science team which was producing some uh price predictions for different uh pools with uniswap to help users to help prevent users from losing money on unisoft v3 because it's very easy for your position to go out of range you've got essentially 100 and permanent loss at that point um and you're not making any trading fees so if you have some prior data about how prices have happened you can make some kind of intelligent guess about what they're going to look like in the future and the front end we had was just ethereum smart contract infrastructure with no management um and

the purpose of that was to sort of spike out on building uh those type of front ends and to help push the product along so right now everyone is kind of heads down integrating all of these threads we've been working on for the last year and a half-ish um and we're very close to launching a product we have a full rebrand coming out soon which is also why the site looks a little sparse but a lot of exciting things happening behind the scenes there it's taken a long time to kind of get all of those pieces to a point where we could integrate them and then also there's been you know i think if anyone's worked in a software company where you have a bunch of pieces and teams and you're bringing them together that's always a you know a challenging piece of the software

development life cycle that's going really well everyone's working really hard but the upshot of it is uh we don't really have anything out right now because all of the work that we had done prior we've stopped maintaining and we're focused on this ave seller kind of bringing that to market uh we should see some more information about that within the next month so what's that going to look like on on ave is that a specific ave v3 functionality because uh yeah in ave it's going to be on top of their stablecoin lending um so there's different yields for each of the different stable coins that you can lend on ave and this protocol will intelligently switch to the highest yield

and you would put stable coins into it and we just find the best yield in ave it's uh you know a very simple product very understandable um very like low risk in a lot of ways i think the ave the ave lending rate is like the risk-free return of crypto in a lot of ways so we just want to offer that to users in the best the best way to make money in that that type of market um we will be launching much higher beta products in the future but this is definitely a very conservative product we're launching with initially okay cool and and uh any any uh estimates on like when this uh new version will come out

uh yeah i think we're targeting currently the end of the month okay oh cool yeah the the front end is a very long way um we have the the new marketing site basically complete the strange love team has been working on the front-end side of this we've hired out a front end team we're doing a number of projects throughout the ecosystem sum is sort of the the biggest and best one that we're working on right now we're also working with the juno tools team and on a couple of other juno projects um and then the peggy jv team which is the sort of canonical song team is we've got folks working on the data science backend we've got folks working on the smart contract infrastructure

um and we've also got folks working on the backend protocol i work closely with that teeth as well awesome um so let's um let's maybe move to the topic that i think everybody is uh is dying to learn juno oh no no i was i was not talking about juno [Laughter] um yeah so i i actually i actually only really like i i've been off of uh i i had been kind of like off of twitter for the last couple of days like not paying much attention and yesterday i was sitting down with someone they're like juno is down i'm like what are you talking about and um

so yeah so so juno uh the chain has halted and uh it's been a couple of days now um so what what's the current state of the chain and when is it due to come back online and then you know we can get into like what happened and all the about five hours from now okay yeah so i i saw a wolf contract put out a tweet saying it was gonna be like nine o'clock utc or something like that i think it's yeah that nine o'clock utc today um we've got instructions up i i need to probably post some updates on that but so as to what happened um you know we'll be publishing a full post

boredom with all of this data but i'll just kind of talk through it here because it's deep in my mind um i woke up on tuesday morning and the chain had been down for a few hours so that that happened overnight pst sort of i guess that would be after noon in asia no that would be like morning in europe afternoon in asia overnight pst um the chain went down um you know initially we were not sure of exactly what caused it what we saw was each of the validators had a different view of exactly what the state was and we actually had 125 different forks of judo network so what just when you say the chain went

down like yeah what does that look like i mean like when you're when you're looking at the chain when you're validating the chain or where you when you're because i like i'm not validating juno i don't have like a juno node what does that actually look like from your it looks like a wide variety of different things what it looked like sometimes it looks like all of the nodes on the network panic and everything just stops um that is an awful situation but sometimes actually easier to recover from than what we dealt with which is all of the nodes stayed online they just couldn't disagree they just couldn't agree on what the next block was so what that ends up happening is they're just voting continually on what

the next block is and just disagreeing over and over again it's basically like the european parliament um and you know that is a really scary situation as a protocol developer because you're like uh okay so like why are they disagreeing and how do we find out what that is okay so so all the so there was like 120 some odd forks or like you know different um different versions of what the nodes thought the next block should be yeah each validator had a different view of what they thought the state would be which is actually just about the worst case scenario of this particular one you know in some cases it's like

40 of the nodes think one thing and then the rest of the network thinks another thing this is kind of classically what people think of as a fork but this was the scenario where not a single node in the network saw the same thing they were all seeing something different which is a very very hard thing to debug um so there were a couple of different options as to what that was and the juno team which is this kind of decentralized team some people have some formal positions but most people kind of you know working on projects on juno and contribute to the core so uh we had about 30 or 40 people in a telegram crew everyone really excited we had folks from sort of broadly within

the ecosystem joe bowman who you know was actively involved in debugging we had a couple of guys from the secret network who came over and helped out we had the tara team in the room as well to make sure that the um smart contract vulnerability if if we found a smart contract vulnerability or a cosmosome issue they were aware of it and able to patch it on terra so um that was sort of what happened and the two different things that we were looking at one was a difference in build method between validators and so we were reaching out to all the validators finding out exactly what version they were running we had this spreadsheet what was really wild about that one is that uh about one-third

slightly under one-third of validators built the wrong way and just over two-thirds of validators built the right way so as we were sort of going it looked like it was going to be really close and we had a lot of trouble ruling out that as the cause of the fork so like that line of thinking went for a long time and some of my tweets on the topic you'll see me talking about validator builds so at this point you didn't know yet what the issue was but you had validators recompiling the code to the to their nodes is that is that right when you say they weren't building right uh well we we hadn't asked them to recompile yet but we were just checking what versions and what how people had

built their binaries and there was a minor difference that could have made it made a big difference in the network um that would have like and this is not something that we had that y'all had like seen before like i mean this seems to me like a like an easy sort of hey let's just make sure that all the validators are running the same software yeah um yeah well there was a minor change in cause of wasm in the last version that changed how people had to build it some people built it the old way and some people built it the new way the documentation wasn't like screaming at them that that was a huge issue so uh that you know things that we can fix moving forward cleaner documentation on building that's an easy one

um then the other sort of area of investigation um i believe max who is a community manager at juno soon after the chain halt had been going through all of the transactions in the last block and identified a contract that had some very specific suspicious behavior so we had some folks dig into decompiling that contract working on figuring out how to trigger issues within the contract and uh after a lot of experimentation and a number of different folks sort of finding different pieces of it we were able to piece together that this contract um

was the the cause of the non-determinism and it was this issue that had been patched upstream um and had been fixed in terra and once we were able to reproduce it we were able to show that the upgrade that we had planned that would have happened like four hours after the crash um we were able to show that the upgrade would have fixed the issue so um at that point once we had root caused it that was last night well no that was a night and a half ago um once we had root caused it we started testing a couple of different upgrade scenarios um and then yesterday at around five o'clock

uh we finalized the test for the other upgrade scenario ran it ran successfully so we knew that the upgrade scenario we wanted was going to be successful that's when we pushed out the time and now we're sort of in the middle of organizing the validators to to get the restart going um we're going to have to dump state from the current network and restart from a new genesis however we're trying a new method where we're not going to actually increment the chain id we're just going to skip a block so that we avoid the double signed issue that happened on etmos and hopefully are able to retain all of the ibc clients and not have to deal with disruptive downtime on osmosis or well disruptive downtime in ibc essentially

because yeah the if we increment the chain id we're going to have to submit governance proposals on all of the counterparty chains to upgrade the clients over on that side which is massively painful let me go ahead and tell you it is yeah yeah okay so um so this is something that had already been identified as a vulnerability in cosmosome and have been patched by the terror team uh it so it gotten patched by ethan frey and the campio team who are the maintainers on cosmo yeah and then tara rushed out an emergency upgrade to address this yeah because tara i think has the most to lose yeah

yeah and and so they had patched all right they had already ran that patch and we're running the patched version of qasim wasm uh but juno had yet to implement that patch we were we we were bringing it in the loop or kayla upgrade so yeah you know it was we just okay you know i think that you know just to understand like the chronology of events here yeah okay no it's important and you know i think that like there's a huge vulnerability that can take down the network that we didn't patch in a sufficient amount of time and somebody ran funds through a mixer on ethereum and took down juno because they probably felt like it like we don't really know

why the attacker did it um because they had nothing to gain from this i mean there was no way that the attacker could steal funds it was just let's just throw let's just throw off this uh some people like breaking windows yeah some people like breaking windows like yeah it's it's hard to know what the um what the motivation is there okay and um so so the attacker um the yeah so the attacker ran a contract i want to talk about this contract a little bit um yeah the attacker ran a contract that provided a non-deterministic um result and then yeah that through so yeah can you explain like what happened

there so uh this is the stargate query vulnerability um console was a very flexible piece of software allows you to run queries against the underlying cosmos chain um so all all of the queries that a cosmos chain exposes you can pull in via a smart contract that's really cool that adds a lot of flexibility and developer support yeah one of those queries pulls info about the specific node that we're running on um and each node has a different identifier a specific key that they use to sign all their peer-to-peer uh information so if you query that it will return

something different for each node so the fix is disallowing the queries that are non-deterministic but so the user had a the hello world contract and they put the malicious code in one line it's a one-line change from the hello world contract so like when we were decompiling it was really hard to find what was going on um but the the secret network folks uh shout out from secret network who uh pulled out their cosmos and debugging tools and was able to find exactly what happened here um very exciting to see that yesterday because we kind of knew this was what had happened but we didn't have proof yet we were just able to like trigger it

we knew it was happening we didn't know exactly what it was um but yeah that that's it was a node info query that broke you know okay so essentially this contract was calling up information about the node that information is like user specified information or some sort of an id that's generated that isn't known by the network um and by doing so did it produce like a hash that was non-deterministic like a block hash is that uh well it's a it's a hash of the public key it's i believe it's an ed25519 public key and they were just hashing the public key uh from the node

so like each each node has a different public key yeah different result 125 forks there we go okay and so what the nodes were what the nodes were in disagreement about was like where where at which point does the node stop and say like i don't know what to do next yeah so uh the trigger for the contract was the attacker sent an update to a value in the contract state and updated it to the number 42. that 42 triggered the node to query for its node id and then say this is the state that we see and

that happened on block something something something 97 so block 97 went in and then on block 98 when the notes were trying to like add the next piece of state they all disagreed as to what happened and that was kind of how the how the vulnerability happened okay got it um yeah i think that's like that's super clear like i i yeah i i fully understand now like all the steps that need to happen for this to occur um real real bad um just just real quick in the comments i see andy mc is asking any other chains vulnerable to this attack i think all the other cosmo awesome chains have now patched this so uh that's good there was also a security update from

the confio team yesterday for an unrelated issue the juno restart will also include that security patch um in that uh that security patch has also uh gone out to terra and secret network and other cosmosome chains um so be assured the cosmos community does take security extremely seriously we work very hard to ensure that these chains are upgraded and each of these chains has kind of a separate uh security procedure so those updates happen at different times but this all those new vulnerabilities are going to be accounted for oh cool and thanks for pointing out that uh we can actually see the chat in this tool it's nice

it's cool i can actually put it up on the screen here yeah i feel like a real tv presenter it's it's fantastic the stream yard i love uh yeah i can't drive it but uh every time i get on with somebody and they are driving it it's it's a very nice thing uh amazing um but yeah back to backtracking at hand um so like can you tell me how how you guys like what was the process by which you guys um responded to this okay so like once you've figured out all right this is this is what what's happening um [Music] how long had it been like how long did

that take from the moment everybody realized oh we're not producing blocks to um like this is what's happening yeah um you know this is kind of interesting um i think juno is an extremely decentralized project like we have zero hierarchy within the developer team so when i woke up three hours afterwards you know like everyone had like grabbed something and was working on it and there was no coordination and um my role in juno has always been trying to break best practices from elsewhere in the cosmos and i was like who's leading incident response it was crickets and i was like is everyone okay if i lead incident

response everyone's like yeah yeah go for it so i kind of took on the coordinator role and i helped i tried to say like okay cool here's what i'm hearing from everyone here's these two different issues that could have potentially caused this who wants to work on first issue who wants to work on second issue go and then try to you know as a sort of incident lead you want to keep your eyes out see if there's any new information popping up help encourage give more resources to the teams that are sort of like at the tip of the mine shaft pushing forward the knowledge of what happened and then when one of those forks either bears fruit or

it shows out to be a dry well you try to redirect effort towards what is important and then just kind of keep everyone moving together um so that's kind of what i've been doing over the last couple of days and trying to help communicate to the broader community um about what's happening trying to be that central point of communication i think next time we need to do a little bit better job of using official channels to do that communication i was kind of doing it through my twitter account um which is fine and you know i've been retweeted by sort of the official twitter accounts as well as the other core devs in juno but uh one of the things that we need to do a little bit better is have one official communication channel during incident response

um that is definitely a best practice from sort of elsewhere in tech yeah no it seems like having some processes shorts in the other room uh having some pro yeah go ahead one second i need to go grab some sorry go go for it um all right so um well while you guys are just here with me uh uh well while jack is away um for a moment here i'm gonna try to fill some air um so yeah if you if you haven't yet subscribed to this podcast uh hit that subscribe button on youtube to make sure you get new episodes as they're released and if you like the

if you prefer just to listen to this you can listen to it it's on apple podcasts on spotify and everywhere else you get your podcast just search for the interop and of course you can go to the interop dot show to get um all the links to places where you can listen to this and jack's back hey i was just trying to fill some answers sorry about that no i i appreciate it it led me to the interop dot show which is a great great url by the way thanks um so uh yeah so having some processes around incidence response um i think you know is probably a good thing are are there um are there processes that

um that kind of emerge from this or are were processes already in place i i think i i heard somewhere i think it was on some other youtube channel and and jacob was talking about uh dave from osmosis having uh like i was looking for this document but apparently he like wrote up this kind of incident response um process um yeah can you tell us a little bit about like what what's that supposed to look like and what do we have in terms of instant response for sure you know uh the person who i will always sort of feel in debt to for this knowledge is uh somebody named jessie irwin she is a name and cosmos that is not heard that often

um jessie is now in a gorick leading security over there um but jesse worked at one password led security there led security and a couple of other tech companies uh prior to getting into blockchain so she comes out of the sort of traditional computer security world and she was one of the first people that i know who kind of came out of that traditional computer security world looked at this world of proof-of-stake blockchains just like whoa this is crazy we can't do a one-to-one grafting of sort of traditional processes onto this so like how do we think about this in this much more decentralized network where there's all of these different actors

things like responsible disclosure two different actors within the network very hard you know if your responsible disclosure includes 125 validators there's a lot of places where that information could leak and some of those validators might even potentially be malicious so how do you treat the validator set how do you think about doing incident response in a network where there's no single person who can actually say hey we're going to restart the network as an incident responder you kind of need to be a cheerleader and you know a diplomat in a lot of ways and work with all of these various actors in the network and i think jesse did a lot to sort of ingrain good habits and good

good ways of thinking about this stuff into cosmos in the work that you know dev and jesse and i worked out of the same office in berkeley um would love to see death document because i'm sure that he's taken the work that he's done at osmosis and done a great job of codifying that um but yeah i think we've got a lot of work to do to sort of broadly disburse these things throughout the ecosystem and as far as process juno did not have any process on this before we're setting up weekly security team meetings i think we'll probably do a vote for some folks to run a security council for juno to make sure that all dependencies are updated that anything that might be an issue gets gets dealt

with early as well as to provide a point of contact for outsiders looking to help help the change yeah yeah of course like having having a decentralized you know sort of development team and validator set and like a decentralized community i think provides a lot of value in terms of uh ensuring that uh well i mean i think like on the innovation side like it allows a lot of innovation to flow into a project but there are times i think where um having some degree of central command or um yeah like procedures and things like that will will certainly help and like in incidences is the time where that's

probably almost true um i mean i think people people look for a leader when things are hard and like this is not like as people who really value decentralization like that is an uncomfortable thing to say but it is kind of this inherent human desire and it's not bad you know you just need to like give it up at the end um and you know that's fine you know as long as there's like established procedure on this you know we have four-year elections in america where the president like gives it up and that's kind of the you know the the big the big scale of that but at these smaller scales we also need to establish those processes and procedures where you know we elect

somebody to do incident response that person does incident response and then they step back and like it's just as easy as that yeah as long as yeah i think that's i think that's right i think like in in in a time of crisis you need to be able to sort of elect people to to take take on those roles things that hap that needs to happen super quickly right like and there used to be a procedure by which that happens super quickly um so what's the upgrade path um you know and and i guess i guess i'm i'm curious about why uh upgrades are so hard or i guess like why chain restarts are so hard and i think evmos has

um some particularities that make it such that it's harder to do it than perhaps like a chain like juno but yeah what's the complexity in involved here chain restarts are hard just period i i don't think that no two are the same they all result from different things and what you really want is when the chain when you're bringing the chain back up you don't want any issues that is the like the worst nightmare is like have awful nightmare deal with that and then have your fix not work that's a real tough place to be in um so you know in my mind when a chain halts you want

to root closet you need to figure out what caused the halt because if you don't know what caused it and you don't know how to reproduce it you're almost guaranteed to walk into exactly the same issue when you try to restart the chain so um you know once you've root caused it then you need to figure out a fix and an upgrade path to that fix once you have those things figured out then you need to schedule with the validators time to do the upgrade or the fix and you need to have the documentation for that you know all of the artifacts like the genesis file or whatever it is steps to reproduce all of those artifacts as well as all of those upgrade steps

um and then there's the sort of upgrade coordination piece of it um any number of things could delay any of those steps and in the atmos thing in particular you know they had it was right after the network started so there was this whole debate do we want to restart the network or do we want to starting in from genesis um that was hard uh yeah with evmos yeah yeah yeah yeah for juno we didn't have that option yeah um you know and then in juno what we did have is like there was a lot of discussion around different upgrade paths if we increment the chain id which is sort of the traditional way of doing this in cosmos um we lose all of our ibc connections

and it leads to this period of you know two weeks on the hub three days on osmosis each chain has different governance periods and the amount of coordination needed to pass governance proposals on potentially 20 or 30 different chains is massive i've done the work to do that for a variety of reasons i i have like a run book and spreadsheets of different validators and like you know governance planning tooling for stuff like this um but so we could have done it um but you know we found a way that's not that and you know we have to test not only what is the what brought the chain down can we reproduce that we need to test the upgrade path so like all those things

take time and what takes the most time is coordination this is a truly global community you know one-third of the people working on this thing are asleep at any given time you know if people are sleeping eight hours um so you know there's a rotation of people kind of doing incident response like i got eight hours of sleep both of the nights of this incident and that's because i was able to pass it off to jake and jacob and demi and the other folks on the core team who you know are doing a great job pushing things forward uh so yeah network upgrades take time um there's a few steps and each of those steps kind of takes a variable amount of time depending on exactly what the issue is and it's just very very specific

i saw you tweet this thing where you had some machine with like a 500 like half a terabyte of ram and 28 like what was that all about like what were you doing here oh yeah yeah for sure uh we have that beast running right now andrew uh andrew golin from the strange love team is uh is running it um is it a machine that you have like in an office somewhere or is it is it a cloud google google cloud okay yeah yeah um yeah we are uh strangelove is the cloud only validator and that that's why we build and maintain horcrux which is i think the best and most secure way to run cloud validators we could talk about

that in a separate discussion oh that's cool yeah i once tried to launch a validator just for like a weekend project and like hit my head on a heart on the wall so hard so it's tough but horcrux is really cool um but what is that doing so when we're doing this upgrade we need to take the current state of the juno network export it into a file and then that file becomes the new genesis file for the restarted network um in order to test that what we did is we pulled that genesis file down and then we replaced the keys for the top two-thirds of validators by voting power

so that we could restart the network we basically impersonated those validators to start a new network to see if the genesis file worked if the fix worked to make sure that the upgrade path that we had because we're we're skipping uh block 98 um and going straight to block 99 because a lot of validators sign to block 98 and if we try to restart at block 98 that would potentially cause double sign issues and we might have the entire chain tombstone itself and then it would never restart and that would be so like when we're talking about like bad straight into more bad that would be bad straight into a lot more bad so this kind of automated testing framework allows us to test different scenarios

and to have very strong assurance that when we go out and do this live with 125 different actors each with their own setups it's going to work um and we we've been doing it strange love a lot of work on this sort of uh large-scale integration testing um and we're gonna be doing a lot more work on that coming up the purpose of this test framework is actually to test sort of the combination of all of the different ibc chains with all of the different relayer implementations um we were just able to quickly hack it to do this thing for juno okay yeah um and so what are the chances that uh is it a non-zero chance that uh that this will fail in a couple of hours that we'll run into some issues

yes [Laughter] what what's that what's that percentage i don't know one to two percent we'll say yeah let's i'll i'll be i'll be very conservative on that um you know there's like that test framework is incredible like the fact that we can pull down and start a hundred and like i think we started 28 different validators on that machine each with a full copy of the network state it is almost an exact replica of what's going to happen when the network starts up but instead of each of those validators being one docker container on one machine they are potentially you know four or five different nodes on 10 to 15

different cloud providers like there is just a tremendous amount of complexity with these live production systems that cannot be simulated in any time you're doing an operation like this there's a tremendous amount of risk um and you know i think the only way to deal with that risk is to just be honest about it and to try to do everything you can to test as many scenarios as possible in a reasonable amount of time and i think that what's kind of obvious when we're talking about this is that there's this trade-off between the amount of time that you can take to test versus like the need to get the network back online because as you mentioned this is a billion dollar network we have

a lot of people relying on it i just throw out that number i don't even know what it is but it's probably over roughly a billion i think at this point probably a billion and a half i don't know people can go check it you know check my numbers um but uh yeah we need to get this thing back online so like how do you balance the testing time versus getting the thing online and that's that's hard you know what i've what i try to do is just try to chart as middle of a path as i see and get reasonable assurance that things are going to work try to you know cross off as many unknowns as you can

and then once you're in a place where enough people on the engineering team were like this seems right this seems right you just go and that's kind of what happened last night after we root caused it and had tested the upgrade path you know everyone's kind of sitting around and they're like okay like what's next and that's when you throw a number out you're like we're gonna restart it here let's go and uh you just you just kind of do it um it's uh i like incident response it's uh easy and clarifying in a lot of ways you know you have this like very concrete problem you've got a ton of unknowns but you know you've got ways to figure those out you've got a team working with you everyone it's a it's a time-constrained situation

um it's a something that i've done a number of times in cosmos and so what what will be the are there any lasting repercussions here you think or are like if the upgrade works and you know validers start producing blocks this evening um is it just gonna be back to business as usual or or do you think there's lasting changes that this will um [Music] have like what will be lasting impact on juno and also i guess on other interchange blockchains i think that these types of crises are what makes chains strong you know

you have to get through some unexpected that's really hard when you're bringing a blockchain into maturity it is not easy you know we've had failed upgrades on the cosmos hub we've had tough production issues there too um bringing ibc to market not always the easiest people have lost funds we've had to recover clients like things are hard and these are sort of natural growing pains of a new network what lasting changes do i want to see after this juno thing i want to see regular security meetings from the core team and i think that that's kind of the biggest thing that we can do to address this exact issue which

is we didn't push a security update for a couple of weeks because we we had a scheduled upgrade we had all this prop 16 stuff going and there weren't enough people who were sitting in a room talking about the security of the chain every week to say this is real bad somebody with a test contract can take the network down we need to fix this um and you know that's what it takes it takes somebody being kind of like persistent about it and you know if we build that in from a structural standpoint where we've got people sitting down and talking about that every week and assessing these different vulnerabilities they come out then that process is easy you know vulnerability comes out security team talks about it security team says we

need an update we ship a small patch we notify validators privately 24 hours 12 to 24 hours prior to the upgrade we notify network participants of downtime network goes down briefly as validators upgrade network comes back up sometimes we can do this stuff without downtime um and then everything goes on as usual and then people forget about security um and then that's a danger in and of itself yeah yeah um well i mean i'm i'm glad that uh things are getting fixed uh promptly i guess like that this was resolved and that the you know the no funds were lost i think

um you know when when you hear blockchain hack usually you're you know the next question is like how much um and here at least at least no funds were stolen okay so some market cap was was kind of chopped off the top there but um i think it'll it'll it'll bounce back um yeah uh one thing that i do want to sort of point out about this it i think highlighted this property of ibc that people were not really quite aware of but didn't really understand when juno goes down no other cosmos chain is really affected yeah ibc isolates fault tolerance yeah

that is an incredibly important property you know when we're talking about some of these like roll up based systems where there's hundreds of chains rolling up into one central chain that central chain is a central point of failure full stop and if there is a production issue there it is critical for the entire ecosystem whereas in ibc with all of these independent validator sets they are truly sovereign and they don't rely on any of these other chains for core operations and like the only thing that goes down is the ibc connection to that chain it's a lot like the way computers and the way the internet works and it's highly fault tolerant as an architecture and i think that as time goes on more and more people are going to see this as a huge

differentiator for cosmos yeah you know that that that's actually a great segue into another topic i wanted to talk uh to speak with you about is you know like i think that one of the things that drew me to cosmos and that i really felt was an interesting sort of technical thesis and philosophical thesis was that change should be application specific like i still remember sunny giving a talk like in cancun in some small room at the hotel like and like talking about application specific chains and this is like blew my mind right and um and for a while i think a lot of chains were having that approach

now um many chains are [Music] enabling cosmosome um so of course like terra like juno is one of them ki is also probably going to i think i'm going to enable it at some point osmosis is enabling cosmosome and like who knows like i don't know like there's probably others that you're aware of um does the application specific blockchain thesis hold then because then you sort of end up in the same situation where if a contract goes down or if a contract has a vulnerability on a chain um you know we we end up in the same sort of

um situation where uh composability with other contracts break and when you have that intricate dependency between contracts on the same chain um you you fall uh you're victim to these sorts of um to these sorts of faults so where does cosmosm and the proliferation of say smart contract chains in the interchange fit in the application specific blockchain thesis yeah that's a that's that's a really good question i think first off i'm going to say no one's got the answer to this right now i'll tell you my opinion on it all right [Laughter] so

my opinion on this personally most of the chains enabling cosmos are doing it in a governance gated way i.e the only entity on the chain that has the ability to launch these smart contracts is governance and at that point it's not a smart contracting chain it is a different way to do application development on top of a cosmos chain are you building cosmos modules or are you writing in cosmosome because all this stuff interoperates so there's a lot of people who like the development experience offered by cosmosome it offers a lot of the same safety and guarantees that the cosmos chain development offers but it does it

in a more ergonomic way with some potentially less boilerplate now that's a thing um i think most people who are enabling cause of blossom are doing it for that reason um so that i think fits deeply into the application specific blockchain thesis and if you look at the way applications have developed on in the web 2 world you know they use all sorts of technologies in the back end to provide very specific experiences for end users so from that perspective i think this this definitely fits in with the application thesis now folks like tara and juno the two permissionless uh cosmos uh chains in the inner chain

um those are a bit of a different thing you know i think that juno if we want to think about it from that perspective juno's application is the experimentation that the chain enables we're going to be the first folks to ship interchange smart contracts we're doing some of the best work on interchange dows there's a number of things that are sort of bubbling up on juno that are going to end up either moving off onto their own chains or being broadly adopted throughout the interchange there is value in that and there's a huge community around that right now and that's what we're seeing in judo and judo is maybe the first not necessarily application-specific blockchain but it is a community-specific blockchain there's a

community of people who are passionate about this interchange technology want to be the first people to bring it to market and want to find the value in that and that's that's what juno is doing tara is a different deal terra is a stable coin tara's sole mission as stable quan will tell you over and over again is worldwide ust adoption if you're looking to drive stablecoin adoption there's a lot of things you got to do and something like anchor and then protocols built on top of that yield are key to that worldwide stablecoin adoption so why not create a permissionless smart contract thing on top of your network to help people build those things yeah drive adoption for your stablecoin and

you know i don't think that invalidates the application specific blockchain thesis and in fact a lot of those applications on tara mars protocol written by larry from delphi a great example they're going to be moving off onto their own chain but they're still going to rely on ust so in that same way where juno is this kind of community first blockchain tara also is this kind of community first blockchain but their community is based around this stable coin so yeah i think i think maybe application um will cease to be relevant in this context like maybe maybe it's ecosystem specific chain or like

yeah i mean perhaps like application i think has connotations of like very specific like one specific use case and in the case of juno all right like it's the application of smart contract development but um yeah i think the narrative is it application interesting it is an application uh yeah but it is an application but i think it it's not the it's not the thing that people think of when they think of an application like if you think of like a an app on your phone it's like that app does something right usually it like does a specific feature and um [Music] where it serves it serves a very um it serves a specific purpose i'd say um yeah i yeah and then and then the other

thing is you know as uh as cosmosome becomes more commonplace and uh more chains enable it what will be the interplay between then cosmos sdk modules and cosmosm and how do you communicate that effectively to new developers entering the space they're like hey we want to build we want to build something should we go build our own chain and use custom sdk modules and perhaps even develop our own modules or should we just you know build this on like smart contract chain x whether juno or fmos or algoric or some other chain that might end up you know

start competing with these chains i think that that's that's a really um this is going to be the battleground of the next few years i think celestia and the whole roll-up world is also going to be sort of a competitor here um shared security is coming out for the cosmos hub there's a number of different application development platforms that are out there for smart contract development and sort of blockchain use cases they're all going to compete against each other and if the web 2 world is any guide the things that are going to make a difference are size and ease of use yeah um in developer tooling um you know i cosmos has always kind of distinguished itself in developer

tooling that's one of the ways we've been able to grow um because we've competed head-to-head with ethereum and what they have that we don't is size you know that ethereum ecosystem is massive and you know for years i have thought that that is the competition you know cosmos is competing against ethereum so like when i started it was kind of a tall order and it felt kind of weird that we were like we're going to go compete with ethereum i was like okay so like how do we do that and it's like well you build the best in class developer experience developer tools for people to build their own blockchains and that's what we went out and built and that's how we were able to build this multi-billion dollar ecosystem um is there is there um i mean because i think one of the things

also that we are building as a as a as a community and sort of as a platform is um standards uh is is there um is there a pathway for things that get developed in cosmosome to that become sort of standard in cosmos and to then kind of move upstream at into the into the i see it as a upstream but yeah like become a cosmos sdk module where we sort of take things out of smart contracts and build them as modules that are kind of production-ready battle tested that people can use is that um does that make any sense or like i don't

know i mean it is i think that the the process that you were mentioning earlier in the call where all these chains are adopted cause of laws i'm like that's because it's becoming a standard part of the toolkit you know we're we're working on um uh the cosmos builders foundation where we're gonna ask all of the different cosmos chains to contribute to a foundation to help maintain the open source software cosmos and is going to be a part of that you know it's going to be the tender mint libs the tinderman and the tinderman stuff the cosmos sdk ibc not only the application layer but also the connection layer um and then cosmosome which is kind of a

key piece of that so um yeah we do anticipate that kind of becoming a standard part of the toolkit it offers a lot of uh a lot of affordances to application developers and you know was just talking to aaron and the regen team um i think that uh the cosmos sdk has been in this like period of transition for a while and it's kind of coming out of it in the application development experience there the shape of which we all kind of knew what it was two years ago but it's taken us a while to migrate there because we have a large number of production use cases that's going to become a lot easier too so you know i think that our aim is cosmos is to meet developers

where they are and to try to bring in anyone and whatever sort of speaks to them we want to have a way for them to do that within cosmos um so like that that's one piece of it but another piece of it is and i think that this is one that has resonated strongly with people and it goes back to that conversation about fault tolerance that we were having earlier um if you have your own validator set you control the consensus and you control everything that happens on that chain all the way down from the hardware up to whatever smart contracts are running on top of it and what we've been calling that for years is sovereignty and that sovereignty is important to a large number of entities and no matter what happens

i think that there's going to be a lot of applications that require and desire that sovereignty so you know we're seeing d5 protocols spinning off onto their own chains they want sovereignty um you know but i think that what this is gonna sort of chat what is going to challenge that and what is going to be an interesting sort of adoption battle over the next few years the battle between roll-ups and application-specific blockchains um because roll-ups do not necessarily offer sovereignty you know they they offer a lot of cryptographic security guarantees um but they do not offer the ability to control at the consensus level what's happening

on the chain you're delegating that work to somebody else what cosmos has shown like when we started cosmos everyone was like oh you're crazy if you think you can ask application developers to launch their own validator sets that's never going to happen we've got 45 cosmos chains running right now yeah there's there's a meme coin that runs its own validator set so like we've made that a lot easier so you know that that barrier to entry is not nearly as high as it used to be um and i think that you know it's gonna lower a lot more over the coming years for a variety of reasons um so anyway i think that that's kind of this interesting competition that's

going to be shaping up in the developer tool space of blockchain um yeah it does appear to be like yeah two two competing visions for how to scale that um that make very different assumptions about the sovereignty of the applications that are being built upon those chains and you know i don't even know where solana fits into this into this vision um uh because like i'm i'm not like super familiar with like how solana works under the hood i think like a lot of people are a little fuzzy on that but if you just sort of look at like the cosmos polka dot paradigm and then the you know ethereum evm

paradigm um there are two vastly different visions that uh i i don't think i i think we'll continue like i don't think one is going to stop existing i think they're just like both going to continue to exist and continue to serve applications and use cases that um that fit with each of the visions yeah i i have a very opinionated view on this go for it um i think that their visions both the ethereum vision and the solana vision is that there is going to be one layer one that gains a lion's share of the market and that is the only way it's going to work moving forward and what we're

seeing right now is a battle for which chain is going to win you know look at the multi-coin cap thesis for solana they lean hard into this solana it's the fastest it's the best nothing's better it's going to win 95 of all volumes settled on blockchains is going to come on solana everything else is going to roll up into it the eve people also chant the same thing this has been proven wrong over and over and over again in blockchain we are living in a multi-chain world and have been for a long time cosmos is the only major project that acknowledges that reality leans heavily into it and builds tooling for it this

is why i am confident that cosmos is going to win in the long term because this bit this multi-polar vision of many different entities running these things and being able to interoperate is obviously how it's going to work and people who think it's one chain to rule them all or trying to sell you their coins and like yeah yeah no i i i think i think you're i think i agree yeah i agree i agree with the with the thesis that there are multiple chains i do think that there's i don't think there's a clear winner i think that there's a space for ethereum and solana and sort of the cosmos vision to to

to um to coexist just like there's a space for like linux and mac os and and yeah windows to coexist you know i would point out that those things coexisting is the cosmos vision yeah exactly yeah and like you know the the solana vision is one really fast blockchain that settles 95 of all trades actions like i just think that that's kind of farcical yeah do we have time to talk about governance or uh are you oh definitely yeah okay i i got a few more minutes yeah um so yeah i mean like i haven't been the most active in governance i have not been the most you know the the

the upstanding uh you know community member participating in all the governance proposals and you know commenting on commonwealth and whatever you know i'll admit to that um i did not get the uh uh you know there's a lot of airdrops i didn't get because i hadn't participated in governance um but you would be both but um i i i have been paying more attention since well specifically since uh prop 16 on juno and i i i did i did chime in um quite a bit on the prop 188 this week the osmosis proposition and i feel like

there are a certain i feel like our governance systems on our crypto governance systems i think people thought that they were one thing and it turns out that they're actually not um at least that's the impression i i feel like people thought that hey governance on blockchain is actually going to be the thing that solves the problem of like all the problems that we have in our like societal governance problem processes right so like democracy but in fact they are just as vulnerable to things like misinformation and um yeah mostly misinformation um and uh

you know mob mentality and things like that so what you know like so norms go norms go a certain like distance norms go so far as to um create processes around which we can coalesce to create governance proposals that um are well thought out um are well debated et cetera but they only go so far in in that anybody can make a governance proposal um that can end up being harmful for the community yeah what's your what's your position on this and i'm kind of rambling here but like what's your point of view here and how do you think it improves how do you

think we improve this gov governance is one of those topics that is prone to a lot of rambling and i will be rambling in my answer so don't worry about it um you know i i think you're right like there's a lot of people who look to blockchain governance as a way to fix things that are wrong with the existing governance systems i am personally one of those people and having seen the sausage be made over and over again in a number of different governance rooms in proposals throughout the cosmos i'm still bullish on that um i think that the tools that we are building we are finding a way to do this governance of these truly worldwide platforms outside of the law essentially

i mean this people get uncomfortable when you talk about this but like it is the case you know what jurisdiction does the cosmos hub live in it doesn't so what laws is it covered by well it's not really so we as a community have to make our own rules we have to police ourselves and we have to do this in a highly adversarial and competitive environment and you mentioned some of the issues with governance and how governance can be attacked and you know how have i i think what we need to look at is how society's dealt with this in the past you know i think some of those things around norms you can think of as kind of a societal immune system and we see those societal immune systems

being built again and again within cosmos and on these chains and kind of continually evolving i think that you know this is all whenever we talk about governance and cosmos all those roads are going to lead back to prop 16 on juno um which is really a great way to approach a lot of these topics because there's not a clear right and wrong here there is and there are a lot of sort of fundamental questions that are being asked by the proposal is this person not necessarily somebody you would want on your cap table if you were starting a company yeah um did they get it by accident yeah

should we take it away from them are they dangerous these are all questions that are much harder to answer but what we are seeing is the community say these are norms that we value we're going to put our money in our votes behind that and we want to see changes based on those norms and that to me is fundamentally positive and fundamentally something that we are going to need to do as blockchain communities if we're going to police ourselves and is this the right policing action you know you can was iraq the right policing action i don't know i don't think anyone really knows but you know these are the types of things that we're

going to see played out over and over again in governance over the next few years and the way that we're going to ensure that those improvements are durable and safe is through improvements in governance tooling as mistakes get made and uh you know i think sonny would agree with this as well a lot more people need to read plato a lot more people need to go back and read aristotle a lot more people need to go read the classics about governance because we're repeating a ton of mistakes in blockchain over and over again yeah yeah i haven't i've not read plato or aristotle um maybe maybe that's next on my list um yeah i think i think norm's norm like i said earlier like i think

norms take us part way but in an you know that blossom governance isn't is i see it as like it's an adversarial space and um one needs to be fully conscious and aware of the of of the the yeah the characteristics of that adversarial space such that um we're able to anticipate the types of attacks on governance that could potentially cause a lot of harm to an ecosystem and find ways to mitigate some of those attacks by

perhaps codifying um some of those norms on chain you know at the end of the day at the end of the day the validators end up making the decisions uh the yeah there was a i don't know if you were on the osmosis uh community call yesterday but there um you know i was talking about uh headlight from key foundation was talking about the fact that france has once once you once you've um finished your term as president you you go into like this constitutional uh council it's sort of like the sages right and uh and you know in the blockchain space the validators end up taking on this role de facto um

[Music] and you know we're we're all sort of beholden to their good behavior there's no way around this um of course you know it's fundamentally baked into the system it's a representative it's yeah it's it's a it's a representative shareholder democracy yeah yeah yeah um what what what do you what do you what do you think of this idea of uh that your proposal should be on commonwealth for some amount of time because like you know what back in my day back in the early days of cosmos you know i seem to remember a time where you know proposals were um proposals were discussed on on the cosmos uh discussion board um you know a lot of time and and energy

we're going to flushing out proposals with the community before they were posted but this was a time where there were just a handful of people that were involved in these communities most of them were doxxed um you know we knew who we were dealing with and you know had met a lot of the folks that we were talking to on these forums as the systems grow and um and become um yeah more where there are more participants people with with with incentives that are disallowed that are not so aligned um will governance continue to work at scale i think that's a great question

i think it's going to be a competition point between different chains but also i just i kind of want to get to this like sort of idealized version of governance i have been one of the people who's pushed back against this specifically over and over again this idea that like we need to have this thing on commonwealth and if you don't have that then like no governance proposal for you you know what none of that stuff matters what matters is can you get the votes on chain and there's a lot of paths to that and the reality is as long as those paths are open they will be used whether by good actors or by bad actors i thought up a proposal on the cosmos hub last year interchange routing i wrote some code i talked to about 20 validators and i got

it passed was 60 of stake voting we had 99 approval that never went up on commonwealth there were a lot of people were real pissed to me about that that i believe was for the good of the network and it was great marketing for cosmos and we shipped some of the first interchange routing technology that's cool so you know what i'm trying to do is try to you know with these systems as they are they are very imperfect we do need to change them quite a bit but if we ignore the reality of the systems we have built we put ourselves in a very perilous position because we avoid thinking about the things that we don't want to think about

and we think that the world fits into this nice little box that we've created when in fact it does not yeah there are many ways to attack governments and we should and will see those over and over again and what we need to do is build systems that are resilient against that and that is our mission here well you know what i think that's a good note to end on um thanks thanks for being a part of the interrupt and for being on our our first uh i think really well put together live stream i'm really glad well thank you i appreciate you having me uh so yeah um and uh yeah hopefully we yeah i think we should yeah we should have these conversations more often so please come back

happy to come back would love to you know some upgrade coming up with with new stuff there a lot of exciting stuff going on at strange love cosmos builders foundation hopefully we'll have an announcement on that soon so yeah definitely we'll have many many more things to talk about the next time next time i come on all right cool thanks a lot sebastian thank you have a great one you do