ByBit Hack Deep Dive

Transcript

Hello and welcome everyone. My name is Peter Robinson and today we've got our pseudo anonymous um hack analyzer security expert um AYC and um they're going to give us a really detailed look at the bybit hack and other things related to that. Um so over to you AYC. Thanks Peter. Thanks for having me. Thanks for everyone for joining and um and everyone in the future on the uh YouTube streams. Um I was keen to do a deep dive of the buy bit hack uh especially um looking back to the last talk that I did around X. Um both of them are exchanges uh but very different

in the in the nature of the hack. So um this is a deep dive of uh of Bybit. Um just to introduce myself um I've been around for a little while. Um, I'm mainly focused on, you know, threat actors and the way that they compromise companies. I was EVM pill in 2022. Uh, went down auditing, fuzzing, building things with Rust, um, related to the blockchain. Um, I've mentioned before, but my PFP is is by an artist named Kevin Abos. Uh, opening email attachments can ruin your day. I think opening Telegram chats can probably ruin your day as well. So, um, and we'll go through that today. Um, you can hit me up on Twitter or you can see me in the ETH Melbourne Telegram. So, let's just cast our minds

back a little bit to Wizer X. It was an Indian exchange compromised uh in uh in 2024. Um, and it taught us a lot about how exchanges should approach the use of multi-IG wallets and I suppose um fired a shot in terms of uh nation states going after crypto. Now, most of this audience would know that nation state adversaries um target crypto, but I'm not sure all of crypto uh knows knows that that's the case. But certainly in terms of crypto exchanges, crypto protocols, even um crypto gaming uh uh um Lazarus Group um or DPRK have chased all of these kind of uh targets. I think

the other thing that Wizir um um taught us, you know, with so many so many funds lost around 250 million was that JavaScript and blind signing just don't work together. Okay, they're a dominated strategy and we had to be concerned about browser extensions, browsers placed in debugging mode, the supply chain in relation to JavaScript because JavaScript was queuing up the transactions that were then signed uh by hardware wallets. I think the other thing that came out strongly from from Wizir and and other compromises was that we don't want to sign or sign transactions where we read our email.

So, uh most of these hacks are conducted where the victim is in Telegram, they're in LinkedIn, they're in email, uh they're opening up attachments, they're executing code, and that's just a recipe for disaster. I think we also saw that there were some particular dangers around safe wallet and why it was important uh to not blind sign and be able to be exposed to the EIP712 data. And I'll go a little bit um further into that in this talk as well. But you probably remember from the last talk that there was a difference between safe wallet doing a call and safe wallet doing a delegate call and that was um based on the operation. I think since then it's also been really important for high-v value

multi-IG wallets to independently generate the safe wallet um hashes which includes a safe transaction hash and a data hash independently of the safe UI. So, if you haven't seen Pete Cavasio's save transactions hashes util, it's kind of emerged as a little bit of a standard uh for how to compute all of those hashes independently so that you can then verify them in the safe UI and also verify them in the hardware wallet. And I'll talk a little bit more about that as well. And from my last talk, we also mentioned safe transaction guards.

They're a smart contract you can deploy on EVM blockchains that can check transactions or safe wallet transactions before and after uh the transaction takes place and they allow you to implement a number of guarantees on chain. And if you're interested, you can always go back and um and visit the last talk. The links will be in the slides um when you get them. Uh, as I go through the talk, I'll probably stop on each one of these kind of yellow um markers and we can we can just pause for questions if there's any and I'll try and scoop up any that's kind of popped up in the chat. So, what's the TLDDR of the bybit hack? uh a safe um a safe developer, not a B by individual but but a safe uh um employee, a developer was social

engineered uh it's believed via Telegram where they executed a backdoor container uh believed to have root privileges given the adversary uh the ability to deliver and execute malware and steal AWS credentials. So the parts in the square brackets here are from a slowmist report that came out um a little bit before the buy bit hack. So uh everything that's in the kind of curly braces is what's come out subsequently. So uh the developer was socially engineered. They executed a container. That container had a um Monte Carlo simulation in it. Um and I'll go a little bit more into detail this in a moment. Uh ultimately it was a ruse to

to execute um code a Python code that then either dropped plot twist or or a mythic agent that then um sought to steal AWS credentials. With those AWS credentials, uh the adversary was able to um move laterally, find an S3 bucket where the safe wallet app was being served from and then, you know, um swap out uh the the a particular JavaScript file with a backdoor JavaScript file that specifically target uh targeted by bit um the bybit multisc and um we'll go through the attack, but ultimately um $1.5 billion worth of uh funds was stolen. So when you think about it, it's very it's very similar to Wazir in terms

of the Wizer X in terms of the web 3 part um but very different in terms of um of what happened prior to it. And if you just look at the timeline, I think if you look from the second line, developer compromise on the 4th of February um and then the exploit transaction happens um a fair bit later almost um you know uh over two weeks later. So you can what I wanted to show with this slide is that uh is that trader trader the actor in this case was inside the safe environment for a very very long amount of time um scooping up uh AWS session credentials uh as they went and eventually found a way to um to compromise the Bybit multisig wallet and I suppose we we'd say that they were

there undetected. So in terms of targeting uh trader trader has a long history uh going back to um 2022 and even earlier than that um a lot of the uh you know kind of interbank um heists uh you know kind of pre-blockchain um are kind of attributed to trader trader as well but just to give you some of the dates here like um um you know the American cyber defense agency was talking about them in uh in 2022 there a really good blo blog post that came out talking about the kind of tactics that they're using now in 2023 and this is the kind of uh share a GitHub repository um have a reason to

recruit you or need help fixing a bug uh get you to fork the repo and execute it. So it was happening with npm in in uh in 2023. um GitHub confirmed that um uh later on in 2023 around about the time that JumpCloud was exploited and then from there uh we saw um uh other kind of like um really great researchers talking about these kind of tactics and uh then from there we see that npm was used, Python has been used, Docker has been used, LinkedIn has been used as a vehicle and but you know most of this activity happens in Telegram um between uh between the threat actor and um and uh and the developer or the per

or the victim in this case. And just to give you a rundown um if you uh so I've kind of left Wizir and Radiant um Capital out of this uh and just kind of looked at what was you know um Wizer X and Radiant Capital are attributed to a slightly different um North Korean threat actor but trader trader is kind of responsible for a bunch of these and they're generally a little bit more complex. they often um revolve around this theft of credentials and then l you know a lot more lateral movement than probably what we saw in Wizer X and uh and um and Radiant Capital. But probably some interesting ones are JumpCloud in 2023. Uh that was a supply chain vector.

So they they're compromising JumpCloud to then target customers. They never released the customer list of who the five customers were that were compromised that were targeted by uh Trader Trader when they when they compromised JumpCloud, but you can probably see some proximity in dates there, right? Like uh in July, JumpCloud was popped. Um um soon after that, there was multiple um crypto project uh crypto payments protocols um that were compromised. um stake is uh uh pops up as a kind of um crypto gambling um online, you know, kind of uh crypto casino and legitimate casino. And then from there, there's probably two interesting ones. DMM uh was compromised via uh their you know via a third party that a supplier which

was GKO. So a GKO engineer was compromised, executed code, a session was stolen, and then uh some impersonation was conducted um um inside of chat programs that then meant that um that DMM sent uh 308 million um worth of uh Bitcoin. DMM subsequently folded uh they they didn't have the liquidity to kind of keep going after that. And then um not not long after that I suppose the next year we see buy bit but via safe right so um you can kind of see that these um this actor doesn't mind looking for weak points in the supply chain and then um you know compromising a developer or engineer and then via that

um compromise uh moving laterally and and uh and then achieving their goal. Any questions so far? There's nothing in the chat. Right. So, we've got a motivated thread actor. Um, let's let's kind of go through uh, you know, the the the kind of kill chain. So, um, the reason why I've got this I've got a slowmisted um, blog post on the on the right of the slide and um, that was a really detailed explanation of some past attacks that would that were conducted but were thwarted. So, they they were stopped.

So, this is kind of like pre-bybit bit. Uh, there was a number of um um kind of forensics that they um you know, they went into a bunch of forensics on a on on a couple of different hacks, but it looks like most of those were stopped. So, um but what you can infer from that blog post is that almost the identical um um TTPs were used against by bit. So, what I've done is is where there's square brackets, it's from, you know, it's assumed from the blog post. Um if not, I've noted it. So uh like I mentioned before a, um developer was enticed to uh fork and run a docker container which um looked like a Monte Carlo simulation.

The container was deliberately backdoored using a um YAML loader called unsafe loader. And unless you were like a pyaml expert, there's no way you would have picked this out. Uh but basically what the unsafe loader does is allows YAML that contains specific Python YAML tags to execute Python inside of the container. So if you didn't know that YAML tags could be executed, I didn't when I was first kind of going through this, they can. And by default inside of Pyamel, they do. Uh and um that that is because the default loader is the unsafe loader. So any anything that can be returned from get stock price which was

registered um early in February uh any crafted YAML that was returned from that would be executed inside the container. Interestingly if you do go through that slowm blog um blog post there was another domain called get stockpric.info which was registered a month earlier. So, uh, this looks like a recycle campaign from some of the, uh, uh, interactions that slowmist had been, um, uh, finding as part of their investigation. Recycled and then some of these, um, um, almost near identical TTPs were used, um, were used against bybit. It's assumed that the Docker container um, had a Docker compose file uh, where privilege was set to true. And uh, that allows the Docker container to

execute with the users's privileges. And generally um you know if the engineer is admin or or or root on the Mac uh they'll have full access to to to to the box with restrictions. Um but uh it's then likely that they executed either um a a stager like plot twist which is um a pretty unknown um um piece of malware from from trader trader where probably the use of like plot twist is malware attributed to trader trader. Yeah. Uh mythic Poseidon is an open source framework and I've linked to it there.

It's written in Golang. Um now in the mandant report for the safe wallet, you know what they have released um from the mandant report uh they do talk about mythic Poseidon network indicators and mythic Poseidon is a really interesting uh uh is a really interesting project and probably worthwhile checking out at some stage anyway. Um but all I wanted to say there is um a noted open source framework. Um now uh post compromise with rce they've been able to find uh longived um AWS credentials but the AWS credentials the longived ones uh wouldn't be able to be used without MFA.

So if you're familiar with AWS credentials uh anytime you see AKIA um that is a longive credential right not a shortlive token. Um so to use that longive credential they needed to have um the uh developers MFA device which they didn't have. So they tried to register one and that failed. Uh so because they couldn't use the long live credentials they basically just watched the developer accessing AWS over a period of time and uh that developer was uh um requesting short-term credentials which are saved to a slightly different place. Right? So the thread actor here is on ExpressVPN. They've got they've got

persistence on the developer machine uh based on the malware that's been executed after the container uh was compromised. And then they're probably looking in a in a location uh like the AWS CLI where um short-term keys are written or STS uh sessions are written and they can grab those by default every 12 hours. So unless uh safe had done something to reduce that session time uh then as the developer accessing AWS using short-term um sessions those short-term sessions are written to the C and then they can be scooped up by the adversary and then the adversary can access the AWS API from the ExpressVPN.

Uh they don't um by default there's no restriction on where those uh on where those keys can be used from. And like we saw before in the timeline, they've been able to maintain persistence for a couple of weeks um without detection. Just want to make sure my screen has advanced. Has that has my screen advanced? Uh yes. Yeah. Just so you know, off to the top right. Whoop. Um yep. We've changed screens to remove loader alias. But off to the top right of your slides, it was saying something about an additional screen. Oh, okay. You're saying that additional display detected.

Yeah, let me just um let me just kill that and restart. I think something's happened with Zoom, but uh hopefully that's um that puts us back in action. Oh, okay. So um one thing that was interesting when I was researching this uh was um the bante who's really well known in in crypto circles had actually submitted a pull request to pyaml uh to change all of the safe loaders back uh or the unsafe loaders back to safe loaders. So I've kind of included that PR in as well u um bante doing the lord's work.

So once the uh once the adversary has those credentials uh we they can move laterally and they ultimately found an S3 bucket that had built artifacts in there and those build artifacts were being used to serve the safe web UI and uh this file app-52 ultimately was served as app.safe.global as a chunk of the page. Um the adversary makes a copy of this JavaScript and then um and then goes about uh patching or modifying uh two two kind of functionalities in there which was sign transaction and execute transaction. Um this would have uh

changed the hash of the file uh um and uh you can kind of see the changes here. So if I just um show the first uh the first patching of sign transaction, you can kind of see what they're doing where uh if and I'll explain this in just a little bit, but uh if you look at that line let WA equals um some addresses, there's an address there that ends in FCF4. So what they're basically doing is saying if those if the target address is within uh those addresses and the operation is set to zero then copy the transaction but swap out the two field the operation field the data field the

value field and the safe transaction gas field. So those two wallets that are in that array let WA equals the address ending in FCF4 and the address ending in D141. The first wallet is the buy bit multisig. The second wallet is a test wallet. So they actually um tested these transactions uh or tested these back doors to make sure that they worked. Um as you can see from that uh way back machine, there was two snapshots taken on uh on the 19th. The first snapshot uh at 11:19 is has not been modified. The second snapshot at uh at 5:29 in the afternoon had been modified. Um as mentioned the sign transaction and the

exec transaction had been modified uh in a similar way. Um so so the JavaScript backd dooror had been created. Anyone that was browsing the uh safe wallet uh web app would have received the um vulnerable JavaScript. But as you can see from from the way that they constructed it, uh there's it only would have ever targeted the um the by big buy bit multi-se. I think the important part is um is that uh the transaction that the um buy bit treasury operators are signing is not the transaction that's shown in their UI. So uh they would have seen the original transaction which was

a transfer of ETH. they wouldn't have seen the this uh cloning of the transaction data and they would have been asked to sign something was that was totally different to what was being displayed in their UI. So we expect in the UI uh that it would have been an ETH transfer using operation zero. Um and uh what they were signing was uh you know the execution of a delegate call using operation one uh with some specific call data that you can that you can see um in the uh in the images on the right. And this is starting with you can see the line DA equals 0x A9059. And we'll go into into that a little bit deeper in a moment.

So the attack transaction I think a couple of things to call out first is that the transaction was actually submitted by the buy bit exploiter. So not submitted by one of the signers which you'd normally expect when interacting with a safe wallet. So it means that at some point the uh the attacker has captured all of the signed data and then from there decided to play the transaction or submit the transaction themselves using their using their own EOA. Uh this kind of brings up a question as to whether they needed to patch exec transaction or whether they could have just patched signed transaction. Uh but I suppose um that's a kind of like com you know a discussion we can have it with safe at some point. Right now it's not known. All we know is that the buy bit exploiter is the one that submits

the signed data. And you can see here the transaction is to a specific address. Now that is an attacker contract. uh the data here is made to look like a transfer. So you can see the function selector uh is is what we would normally expect of a transfer of address and amount. And if you go and use something like himal uh to decode uh you know that entire call data that's inside of that data field, you'll see that it's basically a transfer to an address. And interestingly and weirdly, the value is zero, right? Um, normally you would expect uh, you know, like a ERC20 uh, transfer, we would expect the value to

have something. Um, but let's go a little bit deeper. You can also see an address laying in there and that's another attacker contract. So basically what it's doing is calling one attack contract and it's calling a transfer function using delegate call and it's passing a parameter which is another address and we'll see how they interact. uh a little bit lower down in that transaction data, you'll see that they've submitted all the signes. So, this is what they would have scooped up from uh the patching of the signed transaction which was in the modified JavaScript. And what I'd assume here is that all um three signers from the quorum signed the transaction and

submitted it back to safe. And it's likely that the adversary uh grabbed that signed uh transaction data uh from a safe wallet API and then has played the transaction themselves rather than waiting for uh the final execution from one of the individuals. If we go a little bit um uh if we go into the state um the most important thing when looking at safe wallets in in terms of their storage is that slot zero is the implementation. So, safe wallets deploy a proxy. That pro proxy points to a generic uh well a versioned implementation contract from from safe. And what you can see here is that whatever happened in this

transaction modified slot zero from being from pointing to a legitimate safe wallet implementation to pointing to an attacker control contract. And interestingly, if you look at the addresses, you'll see that it's been swapped out for the parameter that was placed in the call data. Um, and I suppose we try and work out how that's happened. Again, remember uh that um as as far as we know right now, there's been a transfer call which is obviously not a transfer call uh but parameters been passed which is the parameter of a contract. Um so we can we can kind of dive into the attack contract one.

So this is the contract that is that is here and if you go and click on that in in etherscan you'll you'll see something like this right um it's a pretty small contract uh if you go and decompile the bite code then you'll see something like this where there's a transfer function you give it an address and uh interestingly all it does is take that address and store it to slot zero so if you see a little bit above uh uh slot zero uh is defined and then the the address is taken straight out of the parameter and pointed to here. And notice there's no access control here.

So um uh so it's important that the attacker gets it right which um I don't think they were going to mess about in this case. Uh what would have been stored um to slot zero is what was passed as a parameter which we saw before. Uh the value for um the UN256 value is actually never used. Um, so that's why it didn't need to be set. Now, if we look at that second attacker contract, it's a little bit more complicated. Um, and this is the one that has been stored as the implementation now for uh for the safe wallet proxy. And if we deo decompile the bite code and if I make that look a little bit nicer, you'll see it's very

very similar to wiza where there's a function to transfer ETH that's inside of the uh inside of the um proxy contract the safe wallet the buy bit safe wallet and then there's another one uh which will um transfer ERC20s. So this is very very similar to uh the ultimate attack contract uh in in Wizer X where it had functions to sweep ETH and had functions to sweep ERC20s. Um very very similarly uh once the delegate call is made to the first attack contract the implementation is swapped out with this contract and then any subsequent call uh to the proxy um we'll use the storage of the proxy but the

functionality of this implementation which means that the attacker can move uh any of the ETH or any of the ERC20s that sit inside uh of the um of the bybit um safe wallet which is a proxy Right. Now note here lots of access control. So uh this is um all of these functions are kind of gated um by looking at uh the the address and this is what would be known as the bybit exploiter on uh on etherscan. The adversary kept the the same address for deploying the contracts for calling the contracts and for um and for kind of protecting um the calls to this uh to this contract.

Once that implementation is changed over uh or the slot zero is rewritten with a different implementation, it's it's game over, right? There's no way that bybit um can get back into the contract. There's no ACL's or any other functionality that can be leveraged. Uh there's literally nothing they can do u once that slot zero is swapped out. Any questions? So far, we covered some pretty detailed stuff. So, uh I might just quickly bring up the chat. No, nothing in chat, right? I'm happy to continue.

Okay, so I think firstly um let's talk a little bit about the game, right? Um I think a lot of crypto projects, protocols, exchanges uh may not realize that they're in a game, right? There's a there's a nation state actor that is working 24 by7 365. It's on LinkedIn. It's on Telegram. It's it's uh in your email. It's trying to get you to um to execute code. It's trying to compromise your projects. So whether whether you like it or not, you're in a game, right? And and ultimately the way that this works in the security world is it's your team versus their team. Uh and there's no option not to play. So uh if you can if you can keep the level of your ab

abstraction uh you know all the way up in terms of the all you do is execute transactions on the blockchain and you craft the call data um um in a in a kind of artisal way then then potentially you can avoid playing some of this game. But as soon as you kind of delve into what we would call web two primitives or or web two security you're forced to play. So now that you're forced to play, you really have to think about a strategy to win. And um the strategy to win has to be like something that wins the majority of the time, right? So um I'm not sure that, you know, looking at a number of these protocols and and looking at by bit, looking at safe, I'm not sure that they quite had this kind of like um idea

in their heads uh that that they are involved and interacting with threat actors. they are likely to be compromised and they really had to develop strategies and mitigating and detective controls that kind of solved here and I'm not talking about securing everything I'm kind of thinking about well you know by bit treasury operators were compromised at the same time safe was compromised but they don't both need to to to secure at at the same level and I'll go a little bit more in into the detail of that I think as well what surprised me about by bit is that after every hack you have the opportunity to learn and there was definitely o opportunities that by both by bit and safe should have taken but didn't uh when they saw those previous hacks

especially with Xerx and Radiant Capital and I think lastly if you don't have a team if you don't have a security team if you um if you think that you're going to go up against um a nation state adversary you're likely to lose um but even small teams can think in graphs and my advice would be not to try and secure across an infinite horizontal space. Um, but rather really tune into the the the tools um the techniques, the tactics and the procedures uh that your uh adversary use and uh and focus your energy there to start. And I'll I'll go through some of those different mitigating controls later on in the in the talk.

So you kind of move from being totally confused um about you know what you're trying to do, what you're trying to secure um to being a little bit more confident um that you can choose the the landscape on which you're going to meet the thread actor. So let's go through a little bit about you know um by bit and and what they could have done in relation to this hack. So firstly, you know, Bybit needed to react to Azerex and Radiant in particular um because those those hacks had a lot of uh had a lot of information um based on what they could learn. But really what they had to do was they had to swap out all of their blind signing hardware wallets uh for wallets that supported EIP712 signatures immediately, right? Like there was no option once

they saw Wizier uh that they could have avoided this. Um there was the combination of uh tainted JavaScript in in the case of Wizer X and Radiant Capital. It's thought to have been uh via um uh malware that then um used browser extensions or backdoor browser extensions. Uh so they knew that JavaScript could be modified either on the endpoint or potentially uh as part of their supply chain and they should have um gone and looked at their blind signing uh hardware wallets and changed them out. And I wrote a tool a little while ago called secure decoder. And what it basically does is walks through a a um all safe wallet transactions for a particular address and and extracts the signers. So who actually signed

those and um on the signing transaction of the the attack transaction, there were three addresses. The last one is is Ben himself. If you go back and look at all of the different events that happened uh using something like Rindexer, uh you can see that although Wizer X happened 285 days ago, uh all of those wallets pre-existed that attack. So none of those wallets were swapped out after they saw Wizir. Um which is which is kind of like a miss, right? like is uh what we would expect to have seen is that Wazir happened 285 days ago and then hopefully uh you know not long after the Wizir attack we see some of these wallets start to change

over as they move from Ledger Nano's to ledger flexes or treases uh that can support the EIP 712 standard. So that was that was definitely a miss on on on bits on by bit's case and you can kind of prove that on chain. Um and what that would have meant is that they were operating with something you instead of something like this. Uh they would have been operating probably with something more like this. Um and uh this image here uh was taken from um from uh from some uh excellent research. Uh I didn't produce that myself. So I just wanted to make sure you can see the source down the bottom.

So moving on, other things that Bybit could have done is they were working incredibly high value, right? So this this wallet was had 1.5 billion billion dollars worth of funds in it. Uh they should have really been independently generating uh their safe transaction hashes and we mentioned a utility to do that before, but there's multiple of these that that do exist. Uh this is probably one of the ones that's uh more popular. And what that allows you to do is with the given or with the next nons you can uh uh with the call data you can generate all of the transactions that should pop up in the wallet so that you can verify them in the safe wallet UI and you can verify them in the console and you can verify them in the hardware wallet. So it gives you like an additional an additional kind of u um

confidence that what you're signing on the hardware wallet is actually what's been produced by the CLI tool independently and also via the safe wallet. And uh what you probably um if you've played around in this space, what you'll notice is that different wallet and wallet connection software combinations show different things when you're signing uh safe wallet transactions. So if you go and take a ledger flex, a treasure 1 uh and a Ledger NOS and then you use MetaMask or Ledger Direct or Wallet Connect, um you get slightly different variations. And uh some of the treasury operators that I work with had already done this kind of 3x3 matrix where they had already evaluated five wallets. They had

evaluated those five wallets using MetaMask, led Ledger Direct and Wallet Connect. And they had also evaluated those on mobile and on desktop. So they had a really really good idea of um what's going to be shown up in what wallet interface and which were the best ones to use. But if you haven't had the time to kind of go and do that kind of exercise, then I definitely recommend you go and check out Patrick Collins's uh an analysis. Basically, if you if you follow this tweet, um they've got a spreadsheet uh and a video as well where they kind of go through the interface of each of these. And my expectation when you're talking about billions of dollars is is that is that um the uh the exchange would have done that that they would have uh spent a little bit of time

understanding that they needed to be on the best uh um combination of connection software, hardware, wallet and um uh and device. You know, I think last and we mentioned this in the last talk as well is there is an onchain mechanism called a safeguard. basically a contract that you can deploy and link into these safe multisig wallets that can check the transaction details, can check the outcomes of the transaction in terms of the the um changed state and revert if there's if there's particular problems. So, uh a lot of people don't know about these. Um so, if engineers don't know about them, then treasury operators are unlikely to know about them as well. Um and they also you know I also saw some

conversation online that they couldn't have stopped this hack but they definitely could right like uh both in Wizer X and Bybit um there was very limited use in Wizer X there was no use of operation equal to one ever in um and I believe that that was the same as Bybit as well by had only done ETH transfers using uh using Safe Wallet and had only done one ERC20 transfer days before the hack which was for uh Mantle ETH. Um, so you definitely can build guard contracts, link them into safe wallets and apply a policy uh for some protection. And I think uh exchanges should either um um fund and audit uh you know a guards that work really

really well for their cold wallets or go and look at some of the existing um um you know uh safe or guard standards and try and see whether one of them works for them. Um couple other things that we discussed in the last presentation is there's you know although bybit was a supply chain hack where safe wallet was popped and the javascript modified upstream there's still the threat that that is posed by attacks like wizerx and radiant that uh that these signers also could have been compromised via their endpoints and via their browser. So still looking for dedicated signing devices like a Chrome um um you know Chrome OS or um starting to trend towards mobile where the user

interface works. Um and the exact same reasoning that I used last time is that zero day bugs for something like iOS are so much higher than um zero day bugs for Windows. uh you're much you know you're much more likely um to be compromised on a general purpose operating system where you read your email and you you do telegram than you are to be uh executing off of off of a mobile stack with signed code and signed um signed bootloadaders etc. Um, so there should be this trend towards moving to mobile and and if that's um is is a kind of deal breakaker in terms of UX, then you probably should start to think about um uh the use of Chromebooks um where you've you know you're still using um an enterprise

browser policy and I'll talk a little bit about that in a moment. I think as well uh you know any web3 company where signers are performing signing actions uh via their browser uh have to be looking at an enterprise browser and what that allows them to do is to ensure that they're protecting at least one browser right so there's real differences between Chrome and Firefox in terms of how they protect their cookies and how they protect their sessions. Uh there's also a real difference in um how they can protect their browser plugins. Uh what browser plugins are allowed to load up on what particular sites. And I've mentioned a couple of different controls here, but

uh you know the idea of linking uh the enterprise browser via MDM uh to some kind of posture assessment is really important. Uh we want to make sure that browsers can never be placed in developer mode. We want to make sure that browser extensions can only be installed from the Chrome web store for example and not from uh sideloaded from disk as we as was happening prior to uh involving some attacks prior to Azia. Um there's a huge number of browser extensions that can access all sorts of data. Um and uh like we mentioned before uh we want to have some kind of policy for what browsers load uh sorry what browser extensions load on particular websites. Um so I think all I'm saying here is the threat of you know

JavaScript being tainted by browser extension on the endpoint doesn't go away uh even though um by bit is an example of a kind of supply chain where the JavaScript was modified uh upstream. So, you know, safe probably has a lot more to think about in terms of this attack. And I'll probably, you know, pick out these points. Uh um you know, I won't go through everything, but they did come out and say that they were going to kind of require multiple reviewers, right? Which was interesting. Um it was something that I probably would expect straight away, right? But what it kind of said was that although production should have been immutable and build artifacts shouldn't have been able to be accessed by engineers at any time uh that they were focusing a little

bit more on the on on on the overall process which is fine but um I suppose what I would expect is is firstly that you know engineers can't merge directly into into main right they have to go through some kind of review process and also I expect that the SSH keys uh ephemeral, so they're time limited and signed by a CA. Um although the AWS session was stolen in this case, there's there's no reason why if the engineer um uh had it that they could have they could have stolen other things like SSH keys that might have given them commit ability inside of repos. Now, why would they in this case it was much cleaner to just look for an S3 bucket and modify

the source there than trying to kind of um trick a bunch of developers by submitting a um a a PR or piggybacking an existing PR with code modifications that back door the code. So, you can kind of see why the adversary went straight for the S3 bucket, but it doesn't mean that those GitHub credentials weren't there and couldn't have been stolen. The idea of having these um signed periodically just means that these SH SSH keys don't last forever. And if you haven't um looked at signed commits, I think sign commits are an additional protection uh if the private key material for this for the signing of the commit is stored on something like a UB key. GitHub also has a a a um a feature called GitHub artifacts which allows you

to um uh sign all of the built artifacts and then you can go back later and and uh and query that uh that add a station and that's really really handy to kind of say well is everything in my S3 bucket what I built from when I built uh uh when I kind of uh created this build using um GitHub actions. you can go back and um compare artifacts to each other. I think as well what I would expect from any kind of uh you know company involved in really really u um valuable supply chains is that production is just treated as immutable and there's no way to access production unless you've got some kind of ticket

and you've got just in time admin escalation. Um so the idea uh is that production should be immutable that the engineer should never have had the ability to read write access to to to a to a bucket that has built artifacts and that can be generally achieved by uh engineers only have readonly policy and modifying production via build pipelines and not uh not directly. uh and if they do need to break glass and get in and do some kind of function directly uh then they should have to elevate their privileges or add additional role policies to their AWS profiles by going through some kind of uh internal system that can that can boost their privileges. Another really interesting technology to look at is subresource integrity. And um it's a it

would have uh allowed well in this case what it does is basically allow you to hash your artifacts um that are being loaded um in JavaScript and if the brow if the hash of the JavaScript doesn't match the hash that's been um described in the HTML then the browser won't load it. So, one way that safe could have mitigated this uh attack is if they had have um through their build pipeline have produced hashes for all of the JavaScript that they were loading into the to the application. they could have specified those hashes uh and as soon as that hash had changed on the um chunk related to uh the patching of sign transaction and execute transaction then

the browser wouldn't have would have noticed that the hash was it was different to the hash that was being loaded and um and wouldn't have loaded it. So, uh, for those of you that are kind of looking at these, um, um, mitigations, sub resource integrity is a really good one for, um, for protecting against, um, JavaScript being tainted somewhere in the supply chain. And for those of you who aren't familiar, there's a there's a great framework called the Salsa framework, uh, that talks about different levels uh, that you can achieve uh, of supply chain security. And it's a really good read.

If you go and Google Salsa, you can go and read about um all the different attack techniques across supply chains and also um what you can do to try and mitigate some of those. I think in terms of AWS, it sounds like um from what safe is saying that they were pretty underprepared. Um there's no definitive statement uh uh that they didn't have cloud trail cloud trail enabled, but let's assume that they did. Um even if you don't have cloud trail enabled um uh you can ask AWS for emergency access and they can give you access to to kind of like historical cloud trail for a period of time uh as a kind of um um you know mulligan. Uh uh the other thing that we

note is that they they either didn't have guard duty enabled or they did have guard duty enabled and and they weren't monitoring events. Guard Duty is like a cloud intrusion prevention system where AWS has a uh um does a bunch of different monitoring and one of the actual detections that they have that you that is enabled and you can enable as well is to is to generate an alert if someone comes through with a browser agent of Ki Linux and interestingly the trader trader actor uh was coming through with a with a Ki Linux user agent when they were accessing the AWS API. So, this would have been going off um um quite consistently during that activity and something that uh that

you'd normally have turned on and if that that kind of alert popped up, you'd either know that you're being redteamed or you'd know that uh you're under some kind of um um um you know, you're potentially under some kind of attack, right? Like it's a good indicator. I think I think the other thing we mentioned was readwrite roles versus readonly roles and and just in time approval on endpoints though there's the ability to monitor longive session keys AQIA uh or AKIA and also um short-term session keys which are um you know the uh the Asia prefix uh they can be monitored on disk and you can look at what kind of processes are accessing them and you can set up detections to say well uh why is this strange process accessing uh you know my aws directory Um, there's also tools like AWS Vault

which allow you to uh push all of these kind of credentials into kind of like a a mini um a mini kind of password vault and not have them sitting around on disk. Um, in general, you either need to fight like hell to protect the endpoint so that all of these important sessions and cookies can't be accessed or you need to try and push all these sessions and cookies and important information into more secure alternatives that the adversary doesn't have access to. So, we mentioned signing of the SSH key before and moving the SSH key to something like a UB key. So, it's not sitting on it's not sitting on disk.

We've talked about AWS longived and shortlived credentials. Uh, ultimately, they're going to be um um accessed and and used against u um uh used against you. So, you try and move them to more secure uh alternatives, right? And I'll talk a little bit more about some other options there as well um in a in a subsequent slide. Um, in terms of session theft, like that's ultimately what we're talking about here, right? Like the the remote code, you know, the the actor has remote code execution. They go and steal a credential. They then create a session using that credential which is, you know, off the original box that um that they compromised. This should be like not that it's table stakes, but this is

something that all blue teams look for, right? So often what we do is we go and restrict AWS API access to a particular cider block or a particular VPC or VPC endpoint. If we don't mitigate this way, we generally detect when sessions uh are are being you know uh are used or AWS API is being called from outside of our traditional cider blocks that we know about. Um that's if you know that's if you're working in that kind of model where people are accessing via an office or or or or things like that. the the idea that ExpressVPN was used, you know, most teams will take uh the majority of their logging data and look for these kind of um VPN or or VPN usage. So, you

know, I'd kind of expect that to be picked up. But, but the you know, more of a gold standard is to try and restrict these endpoints. And if you are using a zero trust model where people don't exist in an office but they um can travel around where you know from home work from home work from airports where from wherever then you try and get some kind of um virtual VPN that that does uh go outbound from a particular cider block or you start to look at app binding or session pinning and you might have heard of things like Zcala and other technologies identityware proxy proxies where um you can make sure that even though uh all of these different users are on are are all over the world, they still kind of come out from the same kind of virtual network or cider

block. That allows you to ensure that sessions stolen from endpoints can't be, you know, unilaterally unilaterally used against the AWS API uh without some kind of um without some kind of mitigation. What it does force though is if you do go and implement some of these kind of mitigating controls, the attacker is forced to proxy back through the machine. So, uh, once you know what we've seen here with Trader Trader is they take the session and they move it to a different box and then they access via ExpressVPN. If that wasn't available to them, then they likely would have just tunnneled through the original box that they compromised. Uh, and the mythic agent uh that they uh that they um used in this case does support socks

proxying. So they they potentially could have done this, but u me personally, I don't mind the attacker being forced to proxy back through the the same compromised machine because if you've got all your controls set up correctly, um you you you've got more of a chance to pick them up than you would um um potentially if they had migrate that session off the box. We talked about moving the AWS credentials. Um we talked about the fact um that we want some kind of cloud trail and guard duty logging and almost like a minimum for most blue teams as well is they'll set up detections that monitor sessions and look for movement of the IP address the ASN the user agent or browser attributes uh and they'll look at them generally

within Windows. So the minimum will be they look every hour for a change in those uh uh for a change in those u um details or maybe even um shorter windows. So you can at least get some kind of detection in relation to um to VPN usage endpoint devices like in both uh in all of the safe reporting there's been no mention of hashes. Uh there's been talk about um you would expect uh the the thread actor at trader trader to have cleaned up after themselves, but I'm kind of assuming that there was no endpoint detection response software on these on the developers machines. Um and and that's kind of like table stakes,

right? So you would go and grab Crowd Strike, you go and grab Sentinel One. Uh you install those and and hope uh that they give you protection against the the type of uh um the type of ex, you know, kind of uh uh exploits that um that the actor is using. In in any case, you'd want all of these kind of cloud and endpoint logs coming back to some kind of operation platform where you can write detections. One thing that myself and some others have been looking at lately for for some organizations we work with is uh the Santa project and this was a project that does like binary uh or app whitelisting and what we found is it also not only does uh binary whitelisting and and deny listing for

Mac it also does file authorization. So there was a bunch of detections that we had like uh no one can access Chrome cookies outside of Chrome processes, right? Um so if anyone else tried to log, you know, tried to access the uh the Chrome cookie store and decrypt it using something out of the keychain, then at least we we uh we would detect it, but we couldn't prevent it. But using a tool like Santa, we would be able to to ship a policy down via MDM to the to the endpoint and say that there's only um two specific signed processes. um that are MAC binaries that are allowed to access the the Google Chrome uh cookie store and and no one else is allowed to access it. And similarly with things like shortlived and long live credentials, we're looking at policies

that allow us to restrict what binaries or what processes can access those. Um so I would encourage you to go back and look at u um Santa and look at it's now been forked into a project under North Pole Security. Um neither of these are really good for Google uh because if you don't Google them right um you're going to find so many pages just related to to to the real Santa. Um but um definitely worthwhile in terms of looking at binary and file authorization. And in that way if you haven't moved all of your important credentials off disk and into more secure alternatives, you can at least start to restrict what signed binaries can access uh these files um and their particularly sensitive stores.

Uh so this is a slide I had in the in the original one. I don't need really need to rehash it but um we kind of mentioned that if you're in the game there's a price you have to pay right like endpoint detection and response is one of them. Um you know that kind of software running on the endpoint and also cloud um you know cloud logging like cloud trailers is super super important. um you have to have these to to to be able to monitor the effectiveness of your mitigating controls, but also uh you got to be able to be able to, you know, write detections that that alert you and uh and tell you when threat actor activity is happening. So, um but I won't I won't labor on those. You can probably read through that yourselves. So, that that's a huge amount of content. uh a lot of more

in-depth security um talk, but um hopefully I was able to keep you with me the whole time and um and happy to answer any questions that you may have. Thank you. That was um really good. And I'll wait and see if we've got some questions from the audience. and if not um yeah can I say that the

idea that um people could modify the hosted JavaScript code without any audit trail or you know like that just seems crazy. Um yeah that yeah I think there's look they haven't released the full mandant report. It would be super interesting if they did. Um I think by bit are like have been a lot more transparent. Um uh but I I would hope that both are released at some stage but I could understand why safe safe may not choose to release anything other than excerpts.

um you know 2 weeks inside of an environment is a long is a long time and I would expect that there was a huge number of API calls outside of just looking for that S3 bucket and I think the S3 bucket just ended up being the the kind of um lowest cost um easiest attack point. Yeah. Well, actually I was that was another thing that you said. So you said that 2 weeks was a long time and yet often I thought um attack groups are within organizations for months sometimes years before they're detected and like you know the Chinese military when they hacked RSA back in 201213 I think it was worked out they were inside

for 6 months or something you know that and that you know that that's a organ a company built around security and yet. Um, so I thought would have thought 2 weeks is like no time at all. Yeah. Like definitely in those attacks that I showed for trader trader, a lot of it is like four weeks, 6 weeks. They're very patient. Um, in this case, they got to their goal within two weeks. Like maybe it's just something of um of my like defender bias is anything longer than than an hour I'm kind of upset with. So uh it it may just be like a high bar for uh for not wanting adversaries inside your environment. Um but uh but yeah look two weeks to achieve their goal is

is pretty good. Remember they also tested everything. So they deployed all their contracts. They tested everything with the modified JavaScript. Um and then then they after test they went through and and did the production run. Yeah. Yeah. Yeah. Yeah. Oh good to see they test um got to look at the positives. Wonder if they did fuzz testing and static analysis, etc. Um, yeah. All right. What about other people? It's easy. Uh, what other questions do we have from the audience? Um, if you like, you can type them into um the chat and I can read them out.

So there's no questions I think. So um I I think it was really good insight. Um are you going to talk about the recovery process anything or Yeah. Are you talking about like post incident response like uh what you would expect in in a compromise like this? Yeah. Yeah. Anything anything to recover that will be really good. Yeah. So I think um I think in terms of recovery there's you know uh all of the artifacts that are potentially modified by the by the thread actor. So there's the endpoint and um looking at forensic imaging or uh so generally what you would go is is through a warm forensics round. So we try and grab some things

off the box from browsers uh from from some uh you know uh last modified files etc. Then a cold forensics phase. So definitely with that box uh I would have looked at imaging the disk and then going back through that image and looking at everything the thread actor may have accessed. Um the docker container uh looks like it may have been um uh already deleted. There's a cleanup phase that um that this this thread actor does. So you may not be able to pull all of those uh all of those artifacts back out. And similarly with the cloud, you'd be looking at you'd be looking through cloud trail with a with a fine tooth comb to look at every single API call and why there were successes and why there were failures um

at each one of those points. Um and then from there you would be able to reconstruct the majority of the adversaries movements except for in this case some of the assumptions um that that have been made. So if you were buy bit though, all right, so we've just been hacked for 1.5 billion and I guess there's another, you know, pile of money to go as well. Would you race out and buy some new ledger wallets, new laptops, and start from a clean, you know, just assume everything's compromised quickly? start from a clean slate and transfer control of the multisig wallet to new addresses

immediately or how would you do it? That that's a really good question. Um I I think if I was buy bit I would be looking at three things. So absolutely number one is a phased roll out of anything that does blind signing for an EIP 71 712 compatible hardware wallet. So, uh, I would go and create a a test multisig and I would go and grab three or four of the top wallets that are out there, Ledger Flex, um, some of the treasures, uh, and then test transactions against the, um, against the, uh, safe wallet on on one of the, uh, uh, EVM test networks and just make sure that I'm I'm happy that the independent creation of hashes using something like safe transactions util

um, and uh, that the UI and the wallet all give me the information I'm looking for. So what what I would expect on chain is to see them removing or adding new signers and removing old signers and that those old signers should be the ledger stock. Um after that I would definitely be looking at enterprise enterprise browser policies and uh what extensions can be loaded on what websites or move towards something like Chrome OS with with a very similar enterprise browser policy as a dedicated signer. So, you get a lot of benefits to having Chrome OS or or iOS as a dedicated signing device. Um, if you're going to use Chrome OS, I would still advise that you have a Chrome Enterprise browser policy so you can uh have your

security team apply particular restrictions and controls which help you. So, number one, change the signer stock. Number two, get some control around the browser so that we don't have backdoor um browser extensions. And the separation of the signing machines means that I'm not talking on Telegram, LinkedIn, uh, email and executing PDFs and and executable code where I'm also signing transactions. So, you get a bunch of benefits there. For for someone like Bybit, I would definitely look at implementing guards. And uh, that's going to be a slightly longer process because you've got to go and evaluate the guards that are out there. You've got to see whether they work for you and then you've got to audit them and then and then you've got to implement them.

But once implemented, you have an onchain protection that even if even if your other controls fail that you can't you wouldn't have a scenario where there was a delegate call on that wallet um and you lost control of it. So they're probably the three biggest. Um but I think all of these organizations need to number one decide that they're in the game, right? Like and and this is where I think they've been kind of sitting on the sidelines and uh if they didn't know now they know like you you are in this game whether you like it or not. So that means you need a team. You need an effective security team. You need to start walking down this road. Um otherwise these these protocols and companies going to continue to be picked off.

Yeah. Yeah. I I think it'll be I know at Immutable we've talked about how it would be just so devastating to put in years and years and years of effort and then to have it all wiped out in minutes. That would be just Yeah. And so Absolutely. And you saw that with DMM. Uh we spoke about that that Japanese exchange 308 million. They're they're now not in business. Uh you know, by bit were were really really good in their response and were able to shore up liquidity through through loans really really quickly. Uh so look, I think I think in a lot of ways by bit was like a nextgen response in terms of how their executive handled things. Uh but you you see it right there um that one one is now dead and one is one is um just

alive, right? but lost a lot of pain. Yeah. Yeah. Yeah. So, what do you think in terms of the money that is being taken by you know the Barbie is trying to track them? Uh they're hoping that if they end up in some of the centralized exchange they should be able to get it back but majority of them you know it's taking a different channel this money is going to be sitting there you know what are your thoughts around that line? Yeah, like I think once like certainly in Wizer X case and then also in Bybit once they're in ETH um uh then it's it's really really difficult to start thinking about uh how this money would be you know um

recovered or or or um or traced like I think they've got pretty efficient laundering networks uh you know via different um blockchains back to China uh OTC and then back to North Korea. Um I don't focus a lot on the auditing because by that time I kind of think uh sorry I don't focus a lot on the um on the on the laundering part. There's other people that do that way better but um I thought what was interesting in this case was there was a lot of fast action by different centralized exchanges and even USDC uh uh finally came to the party and I think froze some funds which was which was rare for them.

So, I think that is interesting that there was this kind of um um grouping together of of um of people that really wanted to help out. Again, I think the way that Bybit went around building the website to kind of take on Lazarus, try and um um um freeze funds and recover some of this money. I think they actually did some of the best um that I've seen. So, um you know um well done. Yeah, they have a free API that tells you these are all the addresses that are associated with the you know they keep updating. So anybody is is looking into that they can fetch the API and get over the addresses. Yeah, look I would still expect them to get well over what 80% of of funds converted to to to to

um to their program. Like I know that Mantle Eath got frozen from memory as well. Like they were able to recover the Mantle Eath and and that from memory was quite sizable. Um, so yeah, I think it was a great job and that that's one of these like really interesting stories about Bybit is that the the the incident response is kind of open source and that never happens in web 2, right? Normally what happens in web 2 is a company gets compromised, they go insula, they really close down and they really protect what they talk about publicly. But what what you found with by bit is Ben was straight on the streams and I think um uh he was talking on spaces. Uh and I think that gave people a lot of

confidence that he was sticking around that the company was sticking around and then everyone I think you know a lot of web web3 security researchers and um and people at exchanges all decided to to lean in and help freeze and track and and find funds and I actually think that's um a really important story. Yeah. Yeah. I I think it is quite important as well for tracing the money and then blocking it up you know making that recovering it is important as well because that sort of stops people from even if they steal it is is very hard to get it out right but yeah happening here it starts to push on some of the tenants that we love about

blockchain right like um you know and I can see you know you're nodding I'm nodding right like we we of like we're now talking about um we always wanted censorship resistance um but in this case uh I think the market will find uh that will match the level of censorship to the level of risk that people are prepared to take and um and eventually these things will level out. Yeah. Yeah. I'm also pretty picky on one of the thing is privacy know privacy there is lot of privacy preserving systems uh zero knowledge things are very useful for laundering monies and things there's a whole different type of area but uh but

couple of tenants of blockchain become really challenging to uh to real world applications yeah I think you're right um this is this is not a problem like I would expect that in six months we have this talk again, right? It'll be another uh very big hack. Um and we you know uh you know I don't want to be pessimistic but um unless teams in the web 3 really start to um prioritize uh their actions against this threat actor and start to play the game, we're likely to see more of this happen. Um this adversary is very very good and we're only looking at one of their one of their teams, right?

like we we we we're not even talking about a bunch of their other teams that are acting in the crypto space as well. Okay, why don't we flip to the last slides? Sounds good. Okay, the merch store is still open. Uh next slide, please. Um yes. So um we've got um next week or in two weeks time we've got a talk that's going to talk about the intersection of blockchain and AI. After that we've got um Ratan who's coming from OP succinct who's going to tell us about how um you can use ZK with a OP rollup and uh optimistic rollup. So

that'll be a good deep dive. And then on in June, I was going to talk about um the um EOF um but that's being delayed beyond Fuksaka um hard fork. So I'm going to and there are now competing um visions for how that's really going to work. So I'm going to push that out until October. And so I'm going to talk about how to do web 3 game design and implementation in mid June. Uh last slide, please. Thank you. All right. So, if you're here today, this will be on YouTube, including the um a link to the slides. Um join the meetup if you're watching this on YouTube and you want to be able to ask tricky questions live.

And um there's a Slack workspace and sometimes there is example code. Um and so that's all. So, thank you AYC. A enlightening and detailed talk. Um thank you again. Thanks Peter. Thanks for having me. Yep. All right. Talk to everyone later. Um, bye-bye.