Solana's Outage Lasted 17 Hours. What Does This Mean for Decentralization? - Ep. 195

Transcript

hi all before we begin two quick announcements first thanks to those of you who have filled out the unchanged survey if you haven't yet please give us your thoughts on how unchained is doing and what it could be doing better plus two lucky survey respondents will receive a btc candle which is scented with satoshi wood musk musk tulip bulbs and finite minerals head to surveymonkey.com r unchained 2021 to fill out the survey today again that's surveymonkey.com r unchained 2021 second as i mentioned earlier i'm now writing a facebook bulletin newsletter joining the likes of malcolm gladwell malala yousafzai and adam grant among others

this week's edition includes a sidebar to the interview here in unconfirmed and it's a conversation with a solana validator about his experience of the outage and what he thinks solana could do better going forward if you're not already signed up subscribe at laurashin.bulletin.com you'll also find all the links to the weekly news recap there which i send out every week as part of the facebook bulletin subscription again you can sign up at laurashin.bulletin.com and now on to the show hi everyone welcome to unconfirmed the show that reveals how the marquee names and crypto are reacting to the week's top headlines gets the inside skip on

what they see on the horizon i'm your host laura shin a journalist with over two decades of experience i started covering crypto six years ago and as a senior editor at forbes was the first mainstream media reporter to cover cryptocurrency full-time this is the september 17 2021 episode of unconfirmed looking for nfts that are useful and fun try so rare the largest nft based fantasy game you can collect trade and compete with officially licensed digital cards of soccer players from over 160 clubs on sewer that's s o r a r e dot com looking for crypto market data that meets institutional standards digital asset research delivers curated and vetted crypto market data get crypto

pricing and verified volume data crypto asset reference data and token and blockchain event tracking learn more at digitalassetresearch.com the crypto.com app pays you up to 8.5 interest on your bitcoin get twenty five dollars when you download the krypto.com app with code laura the link is in the description today's guest is aiden mott research analyst at missouri welcome aiden hi laura thank you for having me on so this was quite the remarkable week for solana which is that fast rising blockchain that many view as a competitor to ethereum tell us what happened yeah so on september

14th tuesday of this week around 7 50 in the morning eastern standard time validators started noticing that their current node current validator blocks were misaligned and they really saw a large inflow of transactions later to realize to be stemming from the great protocol ido that was hosted on radium ido initial dex offered similar to icos in the past on radium these tokens were allotted to be sold at a specific price at a specific time

people interested in participating in this auction knew what the token standard was they had the address and they were able to see ahead of time what parameters they would need to send a transaction to purchase these tokens and people use that opportunity to set up automated transactions from bots which led to hundreds of thousands of transactions all hitting these validators at the same time these validators saw this influx uh as it was happening the this led to uh forking across the validators there was instability across the chain uh there was the last set block that all validators could agree on was falling

behind where other validators were the issue of this stems in the fact that solana processes in parallel transactions as well as messages related to critical consensus uh transactions and critical consensus message messages within the same block so as this queue of transactions builds up messages about consensus were being put further down in the queue and they weren't able to be propagated across validators and as this queue builds further and further up you saw what they call a deadlock with these parallel information flows being clogged and the network got to the point where block production

halted completely and so what became called the outage actually lasted for 17 hours and i was curious can you put that length of time for the next block to be created into context in terms of what has happened on other chains that maybe have you know been around a bit longer than solana yeah uh one example actually that solana labs uh ceo anatoly mentioned on twitter is that bitcoin recently when the hash rate fell off significantly primarily in the west china specifically miners went offline you saw this

rapid change to the difficulty on the network due to that hash rate decrease which led to about two hours i believe 120 minutes between block times uh it's not the an exact equivalency bitcoin in all senses was still operating as it was supposed to that's a natural reaction to what we call a difficulty adjustment but solana itself had actually experienced an outage of around six hours on december 4th 2020 and actually even sooner on september 2nd of 2021 the network experienced general instability due to a very similar situation that happened with the

star atlas and polis ido on radium so the length of the outage was the longest seen on solana generally anytime you have a blockchain down for that amount of time where there's actual economic activity like these decentralized applications it's the impact the potential impact becomes bigger the longer the outages occurs as you have this kind of disconnect between this chain and the ecosystem of other chains and applications you have linked through there so 18 hours is definitely a significant amount of time for a network to be offline but it's not outside of the realm of kind of not a normal course of action for solana but it's not

something that was completely unexpected or out of surprise on people's minds and what they consider acceptable within the solana network oh it was it is not considered out of bounds for what they consider acceptable in the sense of by the if you're looking at it as a sense of the actual technical consensus mechanism it would be i would say that solana's tried to and uh solana team members try to position this as something that should not necessarily be expected but they view it as something as part of calling it the solana mainnet beta it's it's not that the the

technical part failed the social consensus in a sense of what people are willing to accept as this still being the solana network i would say didn't necessarily violate those standards okay yeah i mean the one thing i would say about anatoly's comparison to when the bitcoin hash rate dropped is that since the bitcoin block is only mined on average every 10 minutes you know by that measure i guess you could say it was maybe just like a dozen blocks behind of where it would have been whereas obviously here you know because one is a much faster black blockchain that you know when you if you're using 17 hours as you know what the outage was then

then it's many many many more blocks that were missed so um anyway what was the community reaction to what happened here i separated into the community of a community within kind of not necessarily all engineers or validators themselves but people who are actively participating on the solana network whether that be application developers whether that be validators or kind of these tangential infrastructure providers i think it was it's definitely very concerning in the sense of is this kind of a root is the root cause something that'll be able to be patched as the application stands today or is there fundamental

issues with how the consensus mechanism and specific how transactions and consensus is propagated across the network i think that validators quickly realized that a restart would be a solution that would get the network running again and as i mentioned before they've done this in december so it's something that they're in a sense prepared for they the restarts was part of the process of the solana test net um tds or tour day solana i believe is what the test net has been called so this is something that once identified they knew what the solution was at least in the short term

in the longer term i think there's going to be a lot of discussion and a lot of research and ideating on if this is a problem that can be solved with these iterative patches or releases from the solana labs oh okay but in the moment in order to get it back up and running what did they do at that time so for the solana network to restart there's a threshold of active stake on the network that needs to be running the updated code to restart the network compared to december the number of validators has increased almost twofold i believe it was around 400 active validators in december this most recent one on september 14th it was

closer to 600 validators so while it's not the number of validators but the stake or the amount of tokens they're holding as validators that is important for reaching that threshold validators were had to coordinate across time zones primarily in the solana mainnet beta validators discord channel to one alert everyone that this problem's happening they needed to wait uh probably i believe it was two to three hours before the official restart instructions were shared from the solana labs team uh in that meantime there really wasn't a whole lot for validators to do at that time

because they didn't have the updated code ready to go to restart their validators once validators did have that code that was shared by the salon labs team they were then able to start restarting their nodes getting their nodes back online and as that happened you saw over the next 10 or so 10 to 12 hours you saw uh state continually continuously increase as more validators were participating in this restart and when it did hit that 80 threshold that's when the network was able to restart again from the snapshot taken of the network before the outage to decide what the last finalized block was to restart the network from there all right so in a moment we're going to

talk about what this means for solana and for the space in general but first a quick word from the sponsors who make this show possible do you love sports collectibles or fantasy sports so rare is blending this together to create an entirely new gaming experience powered by its community so rare cards are officially licensed nfts from over 160 clubs including real madrid paris saint-germain and liverpool and built on ethereum you truly own your collectibles they are productive gaming assets that will generate rewards if you're a good fantasy player join so rare and connect with your favorite teams live the game with passion and earn weekly prizes with over 10 million users crypto.com is

the easiest place to buy and sell over 90 cryptocurrencies grow your crypto with crypto.com earn which pays up to 8.5 interest on your bitcoin and 14 interest on your staple coins when it's time to spend your crypto nothing beats the crypto.com visa card which pays you up to eight percent back instantly and gives you 100 rebates for your netflix spotify and amazon prime subscriptions download the crypt.com app now and get 25 dollars by using the code laura the link is in the description back to my conversation with aiden mott so during this time what was the communication from the solana foundation and or

ceo anatoly yakovenko who you mentioned earlier in terms of their explanation for this outage yeah so the first official communication from the solana team came at about 8 30 on tuesday september 14th which noted that there was instability of the mainnet cluster this was the first indication from the solana team that there was in fact instability issues on the network and shortly after around 45 minutes the solana official communications team through twitter announced that validators were proposing to restart the network but it would would not actually restart

for around another 16 hours after that anatoly uh was quite active in communications at this time on twitter i would say in general anatoly is a pretty well he expresses solana's visit how he sees solada in the crypto ecosystem very well he has a sense of kind of altruism around how he sees his network uh not his network but uh the solana network uh relative to these other smart contract platforms and i think we saw a little bit of i would consider kind of reactive information oh he had a reactive reaction i guess you could say

to what he saw as people may be pointing out attempting to uh to meet not i can't think of the word at the moment i think he became pretty defensive and i think that's understandable and kind of this situation where you have all of these different validators that you're trying to coordinate your it's a multi-billion dollar network and you're trying to fix the problem as fast as possible but there's constraints that don't make that possible such as the coordination across time zones hundreds of different validators all trying to communicate through discord and other messaging platforms like that i would say relative to other uh incidents

performance related incidents to salata this is definitely the one that's captured the attention of in the mind share of the industry for the media for the community members so i think looking back it it's going to be important how going forward the solana team addresses this issue uh from a communication standpoint from a technical standpoint this reaction will kind of define what the short-term future of solata looks like and how solana is viewed relative to other networks and other smart contract platforms in the industry yeah i think you know i noticed kind of as time went on and well frankly after

they got the network up and running again he was tweeting things like sometimes a block takes 17 hours or um there was a twitter user who kind of made fun of um some i guess older solana marketing that called solana unstoppable and his response to that was the humans is what makes it unstoppable so um you know i think he's already kind of trying to uh you know put his spin on what happened there but you know at this moment so after that patch that they made and then the new uh restart which is essentially a hard fork um at this moment is selena still vulnerable to that kind of attack yeah so i would say at this current point

the patch the the simplest explanation i give for what i can give for what the patch did was it prioritizes consensus messages within this kind of within this parallel threading on these blocks so this avoids the issue where consensus mech messages aren't prioritized to be sent and propagated in the face of these excess transactions is the ddos of the network and this particular issue was something that had actually been patched partially following the star atlas and polis ido and i think an equivalency i'd heard was it was four of like the six major problems

identified from that incident had been patched by the time that the great protocol ideal happened which triggered the outage so there was outstanding uh known bugs that were then patched following the restart it's difficult to say exactly if solana is vulnerable to this specific attack again with where you have in it's i'd say it's almost a misnomer to call it an attack if these network these actors weren't trying to take the salon network offline they were trying to participate in this initial dex offering so if in fact in the future there was malicious entities attempting to do this it's hard to say if there's not a

similar attack vectors yeah yeah actually right when i said the word attack you know i meant it was more like an attack of enthusiasm or something like that rather than something malicious um however you know it is worth i think pointing out that i i guess because transactions on solana are you know quite inexpensive that that you know is another factor here and that is what makes this sort of thing possible um so you know i did see a lot of conversation on twitter about what was going down and neha narula of the mit digital currency initiative asked on twitter serious question if your validator community can easily coordinate for a

network restart what's to prevent them from coordinating to block transactions or contracts when compelled to by the sec and i wondered what you thought this solana community's response would be to that question yeah so i think the solano respond the solana community response to that would be that there wasn't an immediate available solution to restart the network the fact that they did have to communicate to all validators over this span of 18 hours to this communication flow as well as having to create this code to send it out to have validators do it themselves is something that i think they would

point to to say that this isn't centralized if it was centralized there would be a single entity that would be able to restart it arbitrarily or be able to influence validator or coerce validators into running this code without validator participation i'm not sure if that's a necessarily a very strong argument but i'd also say that the solana community is perfectly not perfectly accepted perfectly fine with but there understand that there is a level of centralization within the solana network and that's one of the trade-offs that they're willing to accept compared to other networks some networks like bitcoin that's something

that wouldn't be accepted on a consensus or a social level block times aren't nearly as important or transactions per second are for other networks uh solana has really built itself as this platform for fast transactions a big market they're trying to appeal to is high frequency traders that's something i hear a lot when talking about what's the target audience for solana and i think there is a level of acceptance in acknowledgment of that trade-off um so i would say it's difficult to assess the centralization based solely on the validator restart but it is definitely a strong point of

centralization compared to other networks uh there isn't any kind of diversity of which clients these validators are running compared to say the ethereum network they're all running the same client from release from solana labs and any kind of control over that specific vector is definitely what i'd consider a very centralized aspect of the network and do you think that they're going to address that because i did see comparisons on twitter to the dos attacks on ethereum in the fall of 2016 and really the saving grace at that time was the fact that there were multiple clients so do you know if the community does plan to go that route

specifically i do not know uh of any plans myself it would make sense it's almost like the natural logical progression of a network once you kind of stop being reliant on this single on this single code base and you have this diversity and you have the ability for validators to choose what they're running you have this kind of voluntary uh validators can choose to opt in or opt out of which code base they're running so it would make sense i haven't heard any specific plans bad at this time okay and just last question i go back to this question about centralization um you know when you talked about how the sonic community does seem comfortable with the level of centralization that

their network does have at this point in time you know obviously all of this is happening at the same time that we have these conversations you know in congress and with the chair of the sec talking about uh how they would regulate defy and chairman genzler basically kind of implying that things in d5 aren't necessarily as decentralized as you would think and so i just wonder um has that come up at all where they kind of recognize that there are people watching and who the typical way to enforce is through centralized entities so i don't know if you've seen any commentary on that i haven't seen a ton of commentary on that specifically my guess would be that

wasn't necessarily the forethought uh at the forethought of what a lot of people were focusing on the time in this kind of crisis situation but i think it is something that is especially important moving forward giving the regulatory climate as you mentioned and i think it's important for the developers on these networks to be very explicit in acknowledging these points of centralization one is information flows are always good it allows the their users to understand what trade-offs they're accepting but it also allows them to identify issues or specific points of centralization that

independent developers can work to build on so i don't think them not addressing it would fix that problem if it is something that would come under a regulatory purview and i understand that they might not be willing to advertise specific points of centralization uh as they in this climate might be seen as asking for the sec to come investigate something specifically but i think it's something that definitely needs to be addressed from the highest level solana labs and anatoly and anyone participating on the network who has a high level of influence to acknowledge what their influence is and where they see the direction of the network going and laying out a road map in the

that the community could have input on on where you're going to get to this point where you can say it is not centralized in those specific manner okay well we will have to see where this all goes all right well thank you so much for coming on unconfirmed this has been a fascinating conversation thank you i enjoyed speaking with you as well don't forget next up is the weekly news recap stick around for this week in crypto after this short break does your firm need rigorously vetted crypto market data that's aligned with the latest regulatory standards since 2017 digital asset research has delivered high quality crypto data to institutional clients like ftse russell and bloomberg

digital asset research offers clean crypto asset prices and verifiable volume data that's calculated from highly vetted sources crypto asset reference data and an events calendar that tracks token and blockchain events like hard forks soft forks and client and application updates crypto data from digital asset research is available through refinitive or directly at digitalassetresearch.com [Music] thanks for tuning in to this week's news recap first headline fcc chair gary gensler believes crypto exchanges need to register with the sec in a speech before the senate banking committee u.s securities and exchange commission chair gary gensler reiterated

his belief that crypto exchanges need to register as securities exchanges in his opening statement he remarked quote many platforms have dozens or hundreds of tokens on them while each token's legal status depends on its own facts and circumstances the probability is quite remote that with 50 100 or a thousand tokens any given platform has zero securities in response to a question from senator elizabeth warren regarding coinbase gensler specifically singled out the exchange lamenting the fact that it had not registered with the sec quote even though they have dozens of tokens that might be securities gensler's comment comes shortly after coinbase ceo brian armstrong took to twitter to accuse the sec and genzler in

particular of what he called sketchy behavior regarding the regulation of different crypto products notably the sec threatened to sue coinbase if the exchange released its lend product which offers users yield in return for users taking certain crypto assets with the company in a conversation with the block gensler also spoke on lending products this week saying quote if you're investing on a centralized exchange or a centralized lending platform you no longer own your token you've transferred ownership to the platform all you have is a counterparty risk and that platform might be saying as many of them do will give you a four percent or seven percent return if you stake your coins with us or you actually transfer ownership and we the platform will stake your tokens

that takes on all the indicia of what congress is trying to protect under the securities laws next headline digital assets may soon be subjected to the wash sale rule the house ways and means committee which is in charge of any legislation dealing with taxation approved draft legislation that would apply wash sale rules to digital assets the draft comes as part of the build back better reconciliation package they could see an increase of 3.5 trillion dollars in taxes to offset the cost of future spending according to the block the current language is likely to remain intact as house democrats are determined to push the budget bill through to the senate at the moment wash sale rules only apply to securities not property which is how

the internal revenue service classifies crypto that means that currently crypto can serve as a loophole for tax loss harvesting traders can sell losses and almost immediately buy back at the same position while retaining the tax write-off however if this version of the bill were to pass crypto holders would be held accountable to standard wash sale rules meaning that crypto owners would have to wait 30 days after selling a coin at a loss to purchase the coin back additionally tokens sold and purchased within 30 days would not be deductible as a capital loss if adopted crypto trades would be subject to wash sale rules after december 31st 2021 next headline

openc exec resigns after using insider info to purchase nfts openc's head of product nate chastain was caught by nfts based on insider information and has resigned according to a statement issued by the company yesterday we learned that one of our employees purchased items that they knew were set to display on our front page before they appeared there publicly said ceo devin finzer who also announced the largest nft marketplace is commissioning a third party review the revelations began tuesday when twitter user zoowoo tv accused trusting of purchasing certain nft projects just before they would show up on the front page meaning he nabbed them before the prices picked and could net easy profits

the next day opens he acknowledged that allegations were true calling chastain's actions quote incredibly disappointing although the technical definition of insider trading does not apply to crypto assets openc treated chasteen's actions as similar to front-running and according to the block legal experts say that such behavior could invite regulatory scrutiny the nft marketplace immediately instituted new guidelines for bidding employees from one buying or selling from collections or creators while openc was featuring or promoting them and two from using confidential information to buy or sell nfts whether they are in openc or not next headline litecoin and walmart the partnership that never was

a false press release published early monday morning announced that walmart partnered with litecoin to accept cryptocurrency as payment the news was quickly picked up by various mainstream media outlets such as reuters zero hedge bloomberg and cnbc along with multiple crypto publications like coindesk decrypts the litecoin twitter handle and according to my editorial assistant the unchained uh twitter handle who uh he actually deleted the tweet in under 30 seconds and says thank you very much litecoin's price immediately jumped on the news reaching 231 dollars at its peak good for a 30 increase in price before dipping back down to 170 in about 90 minutes in a statement the walmart

superstore later clarified its non-partnership walmart was the subject of a fake news release issued on monday september 13th that falsely stated walmart announced a partnership with litecoin walmart had no knowledge of the press release issued by globe newswire it is incorrect walmart has no relationship with litecoin next headline coinbase futures trading could be coming soon coinbase announced its filing with the national futures association to register as a futures commission merchant or fcm the exchange explained the move on twitter as part of its goal to grow the crypto economy coinbase tweeted this is the next step to broaden our offerings and offer futures and derivatives trading on our platforms for exchanges like ftx and binance

both of which are increasing their presence in the us derivatives market volume is sizeably larger than spot volume an example of a derivative is the futures contract which allows customers to buy and sell contracts speculating on the price of specific cryptocurrencies on a particular future date to sell derivatives in the u.s businesses must be approved by the us commodity futures trading commission or cftc the regulator in charge of all derivatives products however typically the first step in the process is gaining approval from the nfa which then handles the registration process speaking of coinbase the exchange announced plans to sell 1.5 billion dollars of debt offerings the company will use the influx of cash for general purposes such as product development and

future acquisitions next headline finance ceo believes centralization is key according to binance ceo chiang pang zhao the exchange of scrapping plans to decentralize its business model xiao explained in an interview with the south china morning post quote four years ago when we started we wanted to embrace the decentralized model so we wanted to have decentralized teams everywhere but we do run one centralized exchange which is the biggest part of our business now we have come to realize that for the regulators we need to be centralized zhao believes that regulators view binance's current status of not having a central headquarters as dodgy to win back regulators the ceo explained

we need to be centralized xiao's comments come after a slew of regulatory warnings from multiple jurisdictions including the us uk hong kong japan and more next headline google is going with the flow on wednesday technology giant google announced a partnership with dapper labs the company behind the popular nft project's cryptokitties and nba top shot according to forbes it is a multi-year deal with google cloud planning to act as a network operator and providing infrastructure to help scale flow the blockchain upon which cryptokitties and nba top shot exist the new relationship should allow developers to access nodes more efficiently quote it's really about helping them with rapid and sustainable growth said

google cloud north america vp janet kennedy blockchain technology is becoming more and more mainstream so companies like dapper need scalable secure infrastructure to grow their business and even more importantly support their networks the new partnership with flo is not google's first foray into blockchain since last year the tech giant has been a member of the governing council of hedera hashgraph a high throughput distributed ledger interestingly the hedera governing council made headlines this week too by announcing a 5 billion ecosystem fund on thursday which will provide grants to accelerate the development of hedera's network along with funding new partnerships and initiatives time for fun bits time magazine celebrates vitalik buterin

time magazine named ethereum co-founder vitalik bruterin as one of the top 100 most influential people of 2021. in the profile reddit co-founder alexis ohanian described vitalik as quote a builder's builder ohanian went on to add quote no one person could have possibly come up with all the uses for ethereum but it did take one person's idea to get it started from there a new world has opened up and given rise to new ways of leveraging blockchain technology baton lake is not the lone member of the cryptocurrency industry included in times list tesla ceo elon musk president of el salvador nyu bukelli and nvidia ceo jensen huang also made the cut all right thanks for tuning in to learn

more about aiden and the solana outage be sure to check out the links in the show notes unconfirmed is produced by me laura shin without from anthony yoon mark murdock and daniel ness thanks for listening [Music] [Music] you