The 2016 Shanghai Attacks: History and Technical Deep Dive

Transcript

I'm Hudson and I'm here today to talk about the 2016 Shanghai attacks uh a history and a technical Deep dive on this uh you're going to see some chat logs that have never been released before uh you're going to see some insights that may have been muddied at the time from everyone freaking out or just being so long ago so let's get started so why am I speaking on this who who am I why am I so handsome so I worked at the ethereum foundation from 2016 to 2021 did a lot of protocol coordination stuff with uh EIP editing cordev calls devop security excuse me uh you name it and now I'm at polygon Labs after hitting up

a few other places like flash Bots and zcash uh making sure that people know about the incredible stuff we're doing with uh eliminating L2 fragmentation and ZK so I want to do this talk because I think history is important and it provides lessons that get forgotten about over time I'm going to kill this microphone um so the story is really interesting and uh I have an important lesson at the end there will be an end reveal all right so this is when I joined the ethereum foundation Bright Young full of energy you know just so excited but uh this was what June 2016 and then uh September 2016 was

Devcon 2 so Devcon yearly ethereum conference uh that's why a lot of us are here this week in Bangkok um just to put in perspective how things were back then proof of stake we were working on it researching it we wanted it to happen but we were far away from answering even the most basic research questions around how to deal with it um the price of eth was $13 uh that's going to be really funny when you see how much the attacker spent that's coming up um we're still using proof of work like I mentioned and there were four clients that could SN to M it again all execution layer clients we didn't have consensus layer thank you ah great service

here yeah so the clients were gu written and Go parody written in Rust uh ethereum J written in Java and then ath AKA ethereum Plus+ the C++ client which was mainly used for testing uh arguably vitalic OG client py theum could kind of snc to mainnet but not really uh that was kind of we were over that by then so Devcon 2 was happening in Shanghai it was going to be a think two or three day conference I was co-running it uh with a number of other people at the ethereum foundation under my role of devops events and things like that so it's 3:00 a.m. and devc con's going to have their first onstage speech with vitalic buan

in about six hours and I get a call the call comes from a phone that is in the toilet because I guess in Shanghai like we had the phone in the room and the phone in the toilets and I was so disoriented I jumped out of bed and ran to the bathroom and picked up the bathroom phone um and George halum who used to be the pr director for the ethereum foundation was on the line and said hey wake everyone up we're under attack so then I do a deep sigh I had not gotten much sleep anyway uh it was a very stressful Devcon uh operation there so this is the start of ethereum being under attack on September 18th um

there were some people this was in Shanghai China there were people mainly in the European time zones that had not gone they couldn't go because a family or things like that one of those people was Gustav Simonson he used to be the head of security of the ethereum Foundation likely 2014 2015 till 2016 2017 and I said alert gu nodes reported running uh out of memory and crashing on this block uh and then UDA Steiner who's now I believe still the CEO of parody Technologies uh or not parody uh yeah might still be parody Technologies she was also one of the security people at the ethereum foundation uh Nick Johnson was just awake for some reason Nick Johnson's the creator of ens but was on

the GU team at the time having just joined fairly recently in the year before um so they're like okay get everyone together can we get a hold of everybody so according to there's actually a like a hard Fork like celebration call that happened I think it was for like Denon or something where vitalic recalled this story and he said that he got a call at like 4 in the morning or 5 in the morning and it was me just going like we're under we're under attack get up and get down to like this weird hallway area um and he just like sighed and then went out there so I start like I don't say I shouldn't say I start freaking out I start going quickly um and trying to

figure out what like who I can wake up so the the funny the funny stories with this are we're in different hotels around the Hyatt in Shanghai which is like one of the main ones and where the conference was at and like I didn't know this was like 2016 so it wasn't like you could everyone was on Skype like all the logs you're going to see today are Skype or getter so like you could call on Skype but people didn't really answer Skype calls after a certain time because they do not Disturbed so anyways I go down to the registration desk and I'm like just look frantic and I'm like hey I need to wake up this person this person this person so they give me two of the names hotel security always just gives you what they want if like what you want if they you're just like

sounding like fast enough but uh one of the names they said they're not staying here and I said yes they're staying here I saw them here and then I stopped for a second and I said that's not their real name they've never used their real name with anyone have they so uh could not find the Anon developer on one of the client teams uh that was great um so anyways let's dig in a little bit to what actually happened so I can't pronounce that like fart knos something like that uh some people think it means go home this is a a transaction that went to the attack contract that brought gu down with memory errors so we're not sure what that means to this day we really don't

know who did this or why or things like that but that's just one of the things that at first really hit Reddit hard uh cuz people like used it more back then as like oh wait someone's like doing this on purpose it's not just like a bug and gu so one other funny thing that comes up every once in a while as me and Peter will joke about how I woke him up in the middle of the night so Marius is aord Dev with gu but he wasn't around in 2016 but Peter was the co-lead of gu um and so yeah like we we we like to joke on Twitter a lot about the shangh high attex because it really was probably other than the Dow hack one of the most stressful things to happen within ethereum core development work this is a picture the only one that

I'm aware of of the uh cevs and others in the spare room that we just found we literally just found like a big hallway in the Hyatt and then eventually was were allowed to go into into this spare room to work um for the people in here top left uh is uh was that going to be left for you yeah top left is Nick Johnson and then Piper Mariam um Martin send and uh Peter uh zagi zagi yeah all right so we're going to go over the first attack it's a transaction spam attack about a mispriced op code and we're going to go through the fastest lesson in op codes ever all right right you're making a

contract it's solidity it's Viper it's llll if you're a really cool guy you compile it down when you compile it down it gets into the thing that the uh you know evm code and the EV ethereum virtual machine can actually read that's on the left that's code generation from what you is a human or typing human readable computer readable let's go back over to the right now we're actually in the evm we have this code that we just sent in that's readable by the evm and we have a virtual machine that's running it and the that's running the virtual machine or the runtime process in this case would be gu or parody then lower than that's the hardware level so that's like the base level of like that part now we're into like how do we know what when we send in a smart

contract and it says like 1+ one or a plus b equals c how do we know like how much gas to allocate how do we know you know like what's going to happen to this so code in ethereum is non deterministic and the reason for that is whenever you run a plus b equals c in your smart contract every other node on the network has to run that so you can't have an infinite Loop or else it'll it'll just like go forever and all the nodes that are trying to run this are going to like run out of memory so you also have op codes and the op codes here on the left are like regular ones they're really simple to understand like add multiply

subra ract and then you have ones that are a little more complicated the reason that we have these op codes um are to have like things built in that most people can use or extrapolate into bigger and better programs you want to use um we also have Dynamic op codes these are ones that are like a little bit more complicated they're things that like may be useful a lot of the time like doing a Shaw Three hash I might need to take a word and make it into a one-way unique function so depending on the word size depending on a lot of things you have to get that gas calculation right and doing that is a very hard problem that's actually if you dig very deep into it what I believe ultimately killed ewm which was an early attempt to have an alternative to the

evm built on wasm years and years ago so looking at this you know the gas for an addition it's three like that's fine like it's super easy for Shaw 3 and for E it's going to vary by a lot of different things and if you misprice it you can cause a a flaw where an attacker can go in and just click sha three sh three sh three sh three and the contract really cheaply and it just take up a ton of memory uh in the system and this doesn't have to do with block gas limits or things like that it's there's going to be a block gas limit regardless of what you're putting in there what this has to do with is estimating based on what a general purpose computer that is

going to be running this or a general purpose computer running a node is what can that handle what can that handle before you're out of memory so here's the actual attack method gu clients were trying to process that block and it would crash they could uh Spam The Chain the attacker could with an EXT code hash op code to seriously degrade Network perform performance um it was underpriced compared to other um IO intensive operations so what I mean by that is you have uh gu at the time using level DB as their underlying database storage layer uh and you have parody using I believe rocksdb so databases at the time weren't EXA and still to this day for the most

part aren't really maintained well if they're open source because like they don't get enough funding Etc and they also aren't used to dealing with a system like a blockchain that has to hit it so many times so when you're doing this and you're like uh exe hash code like it overwhelms the hard drive it runs out of memory it crashes so if it's a if it's underpriced then they're able to do that for like no money and overwhelm the client there's a memory error it crashes but the parody client who was running on Rock DB stayed alive it like literally saved the network here was the temp solution so I'm sitting in the hallway and there's

like the core devs and we're all sitting there in a circle and like I think a few of them because like you know they were going to a foreign country didn't even bring their laptops because they didn't think they'd have to deal with something like this so I had this just piece of [ __ ] tashiba laptop I bought as a backup for Devcon that I gave to like I think it was Peter or Martin and and they were like trying to syn a note on it over the worst internet you've ever seen at a Hyatt in Shanghai so like that wasn't working there were people outside too trying to sink but it was just really hard to get data but what we did know was okay if they're calling that op code over and over again what can we do to stop this now parody is keeping up but

for how long so what we decided to do was lower the uh Target gas limit and this would be more of a lowering the floor of the gas limit uh specifically so as we are doing that there has to be a coordination effort so I'm leading that part at the time I was in contact with every major mining pool and the ecosystem uh and a lot of major Miners and on top of that in Shanghai uh some of the biggest miners including F2 pool and eth fans which then turned into spark pool were there in person they're in Shanghai they were at Devcon so we literally just called uh xen from uh us like eth fans in the middle of the night and he's just tired as hell and like

drives up or rides up to the Hyatt and comes over there and it's like me a few Chinese Miners and they're like calling all their other miners and we send this out uh to change their setting on their node the thing with changing the gas like the max block gas limit is there is a special formula built in where the network looks at all the miners that are putting out uh signals putting out blocks and they say what what do you want your gas price to be and if enough of them agree they lower it one uh one over 1024th us gradually over time so amazingly within about five hours or less we lowered the gas price to like

something incredibly low it was like 4 million no it was less than that it was like 400,000 it was something like super low so now when the attacker tried to call that same contract it wouldn't fit in the block like it literally just wouldn't fit in the block and if they modified it to try to hit that op code over and over again in a new contract it wouldn't do as much damage because you're limited by that block gas size and by the contracts you can put in there um it also increased the cash size on these uh mining nodes so that's another thing that kind of helped deal with this along the way so this is the temp solution so it's September 22nd it continues I'm still in Shanghai devc con's over um and uh let's see I say attention multiple

reports of death failing is this an issue um and parody struggling as well so here's the second attack method so the same attacker set up a separate set of contracts and this was another type of attack and this one actually did take down some pity nodes if I recall but um at at the on the initial denial of service attacks that were happening with the EXT code hash uh they exploited another flaw where they could cheaply create a large number of empty accounts in state um that might be a little foreign because that no longer can happen and I'm going to explain in the next slide why um this was done repeatedly using the suicide op code now it's called self-destruct we don't use

that as the original name from the yellow paper um and uh the op code's ether transfer function poked new accounts into existence over and over again even though there was nothing in those accounts the state kept up with them which was really weird so now uh bro is making 19 million empty accounts and so suddenly State bloats like and uh yeah it was it was kind of like a oh wait are we going to be run out of this space eventually if he keeps doing this so that's the state blo attack so empty account versus non-existent account I actually forgot about this till the other day this is one those things that reminded me when like there's some weird thing in like JavaScript where it's like

why is this equal to zero but then this does know like it's the same kind of thing so I'm glad we fixed it um an empty account is an account with a zero balance nson Co code empty accounts are like the same thing but the exception is uh it has a few gas calculations and they can be easily turned into non-empty accounts by sending an amount of ether to them so like basically whenever you try to make that account even if you don't do anything in there it like cost number one computation but that's actually less important than the actual like memory on and storage on disk so yeah why can't we have 19 million empty accounts let's have an account party

well it waste hard drive space increases sync time if you've ever synced a node and looked at a graph there's going to be this funny time in 2016 where it's like ah and then like it's done and you'll see why it goes back down in a second and yeah it slows down transaction proper uh stuff because a lot of our way we do our storage stuff within the client level is through Ram so we um set up uh two hard Forks that dealt with these two problems the first one dealt with the problem uh when it came to like the immediate effects which was the uh mispriced op codes and then the second one was cleanup we need

to clean clean up all those you know empty accounts there's no reason to do it um so yeah the process of writing testing and coordinating these eips uh was done actually fairly quickly under the circumstances um Tangerine whistle was first and it did it uh repriced a lot of op codes and there were a few Corner cases of other op op codes that we discovered and then patched um and then spous Dragon uh went in and like made it so that if you touched an empty account on chain using a script and I'm not sure who ran the script uh it would go away so you're basically cleaning up State at that point um the attacker used 300 ether during the attacks uh worth an estimated $4,000 at the time uh but now

is worth $875,000 so [ __ ] you so who came up with spous dragon and tangerine whistle uh this has been discussed a little bit on Twitter this is uh a getter chat that I discovered um where we have uh Powell who uh does a lot of the stuff around at the time testing I believe they do uh a few other research things now at the EF uh Christian W reasner obviously um Christian's like let's name it 150 and 158 which are the eips involved and then Martin is like Tangerine whistle spous Dragon let's make t-shirts it's an icon I'm like I'm like what but like yeah no I'm all over it though that was pretty

cool because I mean hard Fork 2 the ethering I don't know who came up with that I didn't scroll up but like that would have been cool as well so what did we learn multic client blockchain networks are the most resilient uh so uh ethereum didn't stop that entire time it was under attack didn't halt I could always put a transaction in and it would go through the fact that we had two clients this is like what a year after launching I guess 2016 about a year and a half they're both running on the network as an attack crashed one node and the other stayed on that's the importance of separating your specification from your client um Bitcoin has their specification

intrinsically linked to their sole reference client so I always say Bitcoin specification is their reference client ethereum specification is in GitHub and we make clients and that's incredibly incredibly important so my spicy take is bitcoin's going down well for ethereum would go down purely because they don't have a multi- Noe setup uh for the ones in there the well um actually yes there's like JavaScript and node implementations of Bitcoin core however there's so few people running that and also so few people running that with the ability to connect to Strat miners that like I don't really think that that's going to do anything if like that were to

happen um and I believe that's my last slide I have time for a question if anyone has one yeah yeah we go cool so I guess um talking about like uh your last slide and you're like it's multichain client or multi client chains are more resilient um I guess the question safety versus liveness though because guess like for this particular example when the bug was a crash bug you were able to preserve liveness of the

network but if it was for instance a fork bug would it not be actually less resilient to safety because for instance if it was a fork bug then par might take one chain versus the other one versus if you all running one client then you'd Fork together such that uh that's a great question so it depends on your blockchain I feel like at least back then Bitcoin and ethereum were really all about like liveness because it was like decentralized staying alive staying alive like the BS but at the same time um I do see your point Forks in ethereum have a lot of I guess uh cultural around like manual intervention from the community so you roll back the chain or whatever that

like uh and not like the Dow Fork but like you basically have the longest one and then you figure out how to deal with that so it's kind of a thing where that wouldn't that happened in Bitcoin too like like Bitcoin had a thing in 20133 where they had a fork uh because they had a bug in like 0.7 or 0.8 or whatever and um it caused them to split so all the miners in about an hour got into one room and uh they said okay this one's the bad one this one's the good one we're taking this one so Min or two mined on that fork did not get their rewards however the transactions did stay from what I understand and I don't know how they did that um lastly there's like like salana which I feel like you

know obviously are very different in terms of what they prioritize compared to ethereum uh but you make a good point it's what the community wants and what the blockchain is kind of initially or culturally designed for uh where that could be a problem I just trust ethereum's like resilience as a community and the protocol devs and the processes we've had and the you know hardening we've done to be able to deal with something like that and maintain liveness yeah sure oh yeah he had one uh sorry maybe m is actually a question if you can go back um I'm curious about the script that would uh close those accounts yes can you explain

it more yes okay so um I think the ethereum foundation may have run it I don't know but uh you basically have uh all those accounts the hard Fork made it so that if you were to send a specially crafted transaction and you said hi I'm touching that account it goes away so the script itself I don't know where that script is and I don't know who did it or who launched it I'm going to guess the EF because it probably cost a lot of money or at the time a good amount of money to like run that through 19 million accounts like concurs but yeah thank you all so so so much [Applause]