The $1.5B Problem: How Exchanges Can Build Safer Cold Storage Infrastructure on Ethereum

Transcript

Hello. Uh, my name is Benjamin Samuels and today we're going to be talking about how to build uh, safer gold storage infrastructure on Ethereum. Before we jump in, a little bit about me. My name is Benjamin Samuels. I'm the director of engineering at Trail of Bits. Uh if you're not familiar with Trail Bits, our goal is to help developers write safer, bug-free software. Uh you might know us for our audits. We also do early stage consulting uh helping developers build security into their protocol from the very beginning. Uh in fact, the content in this talk is based off one of our consulting engagements uh and some of the things we learned from it. Uh we also maintain a bunch of tools for building safer software. We maintain slither, slither mutate, uh medusa, akidna and a whole bunch of other tools.

And if you want to follow us on Twitter, you can take a picture of that and go to our Twitter list. But let's jump in. Um, let's talk about cold storage from the very beginning. Where did this idea come from? Why are we doing it? And I think the best way to do this is to take a step back into 2012. Uh, and ask ourselves if we were in charge of building an exchange in such a way that we had to secure like a hundred billion dollars worth of Bitcoin, how would we do it? How would we structure the wallets? What would we do with the technology that we actually had available back then? And the model that most people arrived at was to have like hot wallets and cold wallets, which is something we're all familiar with. Um, so cold wallets, the idea is that you would have this wallet

where the keys are kept off any kind of digital media. You'd have them either written down, uh, you'd have them engraved on something physical. Uh, they wouldn't be on any computer. And that's where you would store the majority of your funds. Uh, and then you'd have hot wallets. And hot wallets are your operational wallets. That's where you would have users deposit funds into. That's where you take funds out of uh your exchange to like withdraw. Um, and the idea is that if your hot wallet gets compromised, uh it's not a big deal because that's a small percentage of your total assets under management. You still have your cold wallet and your cold wallet is still secure. And so if you wanted to go like the extra mile, uh you could add a multi- signature wallet to your co wallet. Uh, and the way you would do this on Bitcoin back then was

something called op check multisig, which is like a Bitcoin script command to set up a multisig. Uh, it would only work up to three keys and still only I think it I believe it only still works up to three keys for this day. But, um, it's still fundamentally the same security model. So, you'd have three keys that you keep offline at all times. And with a majority, like two out of three of the keys, you'd be able to sign a transaction to withdraw from the COB wallet. Uh, one of the key problems with this model is that if your keys are compromised in any way, let's say someone breaks into your building, they steal two of the two of the three keys and they're able to sign a transaction.

That's it for your cold storage wallet. All those funds can just be withdrawn. Uh, so let's move forward and see how this model has evolved over time because what we really care about today, we care about Ethereum, we don't care about Bitcoin quite as much. Uh, on the right side, we have hardware wallets. We didn't have hardware wallets like 15 years ago. Uh this is a fantastic development. It's really great because now you have a way to store your keys uh in like an HSM like device uh where it's very hard or even impossible to excfiltrate the keys out of it. However, the downside is they don't actually tell you what you're signing because Ethereum transactions can be so much more complicated. You can program your smart contracts. They can do all these different things. Um there's no way for the for the harbor wall to actually

figure out what you're signing. In a lot of cases, they can tell you if you're doing like a basic transfer, approve, or if you're using a big protocol like unis swap or something, they tell you roughly what's going to happen, but they don't have a way to model like a more detailed transaction. But on the bright side, now we have a bunch of uh actually good well audited multi-IG implementations available. Uh we've got NOS safe, got all kinds of other stuff. However, um the way that most people use these multisigs requires you to go through someone else's website. So if you want to use Nosisafe the primary way to do it is to go to the Nosisafe website uh propose a transaction approve it or use their API to propose and approve transactions and sign them and that has supply chain

risks. Uh we saw that in the by bit attack. Um in that case uh the Nosis front end was actually compromised and uh the transaction that they saw in their user interface was different than the transaction that was actually being signed on their computer and they had no way to figure out that that was happening. But on the bright side, you can now program your wallet to do almost anything. Um back in the Bitcoin days, there was there is a a language for scripting in Bitcoin. It's called Bitcoin script. And it's very limited in what you can do. And now with Ethereum, you got solidity. You can do whatever you want. You can add all kinds of different policies and do whatever you want. However, uh attackers can also use this functionality against you. So if you don't think very

uh uh if you just let your wallet do anything, you don't really apply any kinds of policies, this can increase your risk. Um so you've got like in by in the bybit attack, there's like a delegate call. So you could call into another contract with the same storage and you could do basically anything. And that's what the attackers leverage to steal the funds. And so the idea that I want to take is what if we were to actually actually able to program our wallets to protect the funds in the first place because it feels like uh it feels like this model isn't actually as secure as something that we might have had 10 or 15 years ago with that more basic multisig model.

Um is there a way to actually take this and transport it into like you know 2025 and actually have a more secure solution than we did back then? Uh, so here's our our two wallet models. The first one is the one you just saw, the um the Bitcoin multisig model. If you lose access to those keys, it's game over. They can just spend the funds. And what I'm proposing is in the Ethereum model, when you have your multisig, uh, just because your multisig is compromised doesn't necessarily mean you lose access to your funds. uh you could set up like wallet rules, you could set up usage policies, you could set up all these different logic that you can do in an Ethereum contract to constrain how those funds are actually used.

So let's approach this from a design exercise perspective. Let's design a cold storage solution for Ethereum from the bottom up. Uh like from the absolute fundamentals, let's not borrow ideas from uh previous iterations. So, uh, we're going to put together some design goals. Um, I also wrote a couple of blogs about stuff like this a couple weeks ago that we're going to borrow ideas from. The first is called the custodial stablecoin wreck test. And this blog post is about custodial stable coins, but the threat model for custodial stable coins is very similar to the threat model for a custody provider or cold storage or like an exchange. Uh and we also have a blog

post about maturing your smart contracts uh beyond private key risk. And a lot of those ideas we're going to draw from that to try to constrain the damage that can happen uh when a private key is compromised. And so our three design goals uh the most important one is we want to make sure that a complete compromise of the multisig does not actually lead to a loss of funds. This is the most important one. This is what the talk's all about. Uh the second one is we want to make sure the transaction call data can be thoroughly validated using a permanently offline air gap computer. Uh and we'll talk about about why that's important in a bit. And the third is a we want to make sure that our wallet implements a very simple state machine that can be easily tested, easily validated, and we know that its

code is going to be correct. Because the one thing we don't want to do is we don't want to spend all the effort building like this wallet, this extra infrastructure, these extra smart contracts and accidentally increase the risk to our system by adding those smart contracts. So our first goal is we're going to try and make sure that complete compromise multisig does not lead to loss of funds. How are we going to do that? We're going to try to constrain what the wallet can actually do. Uh if you take a look at the wallet that was compromised in the bybit attack. It implemented these two functions, there's an execute call and an execute delegate call. These are the exact opposite of what we want to do. We want to be able to constrain what the contract can do to the highest extent possible. Uh you

don't want these functions to be on your wallet contract that actually stores your funds. So what are we gonna do? We're gonna start from a new contract bottom up. We're going to have one. We're going to start from one basic fund. One basic function called transfer out. This is a very simple function. It just takes a a address that you're going to send the funds to. It's going to take a token that you're going to transfer an amount and we constrain it so that only the multisig can call this contract. And this is just like the first basic implementation. It's very simple. Uh but it's already way better than our initial situation where you could basically call and do anything you wanted. Uh so the next step is you can take some of the off-chain rules that you would normally apply before signing a transaction and

reimplement them onchain. So what do I mean by that? Um in a cob wallet it's pretty obvious that you're not just going to be transferring those funds anywhere. There's going to be a rule that you can only transfer funds from your cob wallet to a hot wallet. And that's a very simple constraint that we can actually implement on chain. So we'll add these two functions. We'll add one to add a hot wallet adds adds a hot wallet and remove hot wallet removes a hot wallet and constrain these so only the multisig could call it. Is that all we have to do? Uh not necessarily.

There's actually a little bit of a bug in here because if your multisig is completely compromised so the attacker can sign whatever they want. All they have to do is call add hot wallet with their malicious wallet. Uh call transfer out and transfer the funds to their wallet and then they're done. So what we need to do is we need to implement something called the principle of lease privilege. And the idea behind this is your system has a bunch of different principles that access it and need to do something. And you want to make sure that the access that those principles have is as small as it can possibly be and have those principles still be able to do what they want to do. Uh, and the way that we're going to do that here is we're going to take the only multi-IG role and split it up. So instead of one

only having one multi-IG, we now have two multi-IGs. So we've got two multisigs here. One is a cold wallet configuration multisig and one is an operation multiig. The configuration multisig is only responsible for adding and removing hot wallets. And the operations multisig can only transfer funds to those hot wallets. And by splitting it up this way, we've made it so that if you were to compromise the operations multisig in isolation, the worst possible thing you could do is transfer funds to a hot wallet, like just drain the cob wallet into one of your hot wallets. And if the attacker hasn't compromised that hot wallet, then the funds are technically still safe. They're still within your system. The attacker would now have to figure out how do I compromise this hot

wallet and move the funds out of it. If they compromise the configuration multisig, the worst possible thing that they could do is add their malicious wallet or remove one of your valid wallets and cause like a you know some kind of problem with your operations. But what's important here is at this point you would have to conf to compromise both of these multisigs to be able to steal the funds and drain them out of your system. So this is where we stand right now. Um we've got our two multisigs. We've got our wallet contract. got a bunch of functions that they call and they only send the funds to one of your hot wallets that you've configured. Now, there's some additional rules that we can implement that you might be already enforcing in your off-chain uh like

signing software. Uh the first one is a rate limit. Uh so the idea behind a rate limit is if we have a situation where your operations wallet or operations multisig is compromised and a hot wallet is compromised, you don't want the attacker to be able to transfer the funds to a hot wallet which is relatively low security. Like this is a key that's running on a machine in your infrastructure and transfer the funds to that hot wallet and just remove them and transfer them out to a malicious wallet. And so the way we do that is with a rate limiter. Um, and the idea here is that if if this potential killchain was to actually happen, um, they would be constrained in how fast they'd be able

to exfiltrate the funds through that hot wallet. Um, there's a really good implementation for rate limiters and chain link CCIP in rate liiter.soul. Uh, it's relatively simple. Highly recommend it. Um, if you're even designing a protocol that doesn't have to do with this, I highly recommend looking at rate limiters. uh because if you do get hacked, a rate limiter can be the difference between uh like a complete compromise of your protocol and only losing a couple hundred thousand dollars. And we can make this a teensy bit even better. And that's by adding time locks.

So the time locks in the system uh are just an extra layer of protection to give your instant response team a chance to respond when something bad happens. Um so what do I mean by that? So if your one of your multisigs doesn't matter which one is compromised and it starts sending transactions to the chain those transactions are those transactions are executed immediately which is not good because your instant response team is only going to know that those transactions are happening when they see them executed on chain and actually happening. And by adding a a time log contract between those multi-IGs in the wallet contract, you can give your incident response team a chance to actually respond to the incident and stop the transactions

before they're executed on chain. Um, and it doesn't have to be 24 hours. I just selected 24 hours because most people like, you know, a 24-hour lag is enough time for their instant respon 24-hour lag is enough time for their instant response team to respond. But it's like if you have a 24-hour SOC and like you have really good alerting and really good monitoring, like it could be one hour, it could be four hours. Uh it could be whatever amount of time is valid for your instant response process. Uh and there's actually alternate versions of this entire scheme for cold storage and like you know custody solutions uh that the Seal Alliance is working on. I don't know if you're familiar with them. It's a project run

by Sam Sun. But um their idea is to primarily operate through time time lock contracts. So instead of just being able to send transactions to Nosis safe um each of your signers would submit something to this time lock and then once it hits a certain threshold uh you know it'd be stuck in this time lock for a certain amount of time and you'd have a certain period to respond and cancel the transaction. But the time locks are fantastic. They're very important and I highly recommend them. So let's talk about the next design goal. Um so that we want to make sure that transaction call data can be thoroughly validated using a computer that's offline.

Uh the reason why we want this is we want to make sure that there isn't a supply chain dependency. Uh the problem with Ethereum transactions is they're very challenging to validate if your contract can do anything. So that example before when we were talking about like it could call like execute delegate call could execute a call to anything. There's no way an offline computer can ever validate that. And so fortunately, our system's relatively simple, so we don't have to worry about that. Um, if you're using like a Nosis safe, I recommend using safe tx hashes util. Uh, but the chances are you're probably going to have to modify that for your cold storage solution. Um, but highly, highly recommend this.

And finally, we need a simple state machine that's ideal for fuzzing and formal verification. This is the last and probably the most important piece of this. Um the the problem I've seen it when we've recommended this to clients is they're worried about the smart contracts. They see a bunch of smart contract hacks, you know, in the news all over the place. They think if I was to like add these rules through a smart contract, isn't that going to make me more exposed to risk instead of less exposed to risk? And that's true. It does. It does make them more exposed to risk. The key is we want to make sure that the contract is very simple. It's very easy to test and it's thoroughly tested. What kind of techniques do we actually need to do that?

The first and most important one is no upgradeable contracts. Upgradeable contracts can be nice in other contexts. You don't want them here. Um the reason why you don't want upgradeable contracts is one they make things more complicated and two uh they add a central point to failure because you need a way to authorize a contract upgrade. And if your contract upgrade is authorized by one of your multisigs and then one of your multisigs is compromised now all of the protections that we just put in place don't even matter. Uh so if you need some kind of like upgrade upgradability mechanism instead of using like a proxy or something provide a way to migrate from one contract to another and make sure that it requires both of the multisigs to be involved for that to happen. The

next is 100% unit test coverage. This one's like baseline. You should have this at this point for basically anything. Um, however, I should note that like 100% unit test coverage doesn't actually mean your test suite is good because you can get 100% coverage just by running every single line, but that doesn't mean that every single line is actually tested. Um, and for that you have to do something called mutation testing, which we support with uh running slither mutate. If you have slither installed, you have slither mutate, I highly recommend it. Um, what mutation testing is is, uh, it's a tool that will basically take all your code and modify it. So, like change pluses to minuses, it'll comment things out and run your test suite. And if your test suite still passes, uh, then there's

something wrong with your tests. You need to add an additional test to verify something uh, in your code. And I highly recommend this for figuring out whether your test suite is actually good or not. And last is fuzzing and formal verification. Uh so these two tools fuzzing and formal verification uh I think they've become a little bit less effective over time as smart contracts have become more and more complicated. Uh but if we keep a relatively simple machine relatively simple state machine relatively simple like functions uh you get a lot more mileage out of these tools than you would otherwise. So I highly highly recommend them. Uh and when you have a simple like system like this, you can write much more thorough uh properties and fuzzing uh invariance.

All right, where we've landed. So we've designed this system. Uh a multi-IG compromise does not lead to loss of funds if it's only one multi-IG. Uh our transactions can be validated offline. Our simple our system is simple and thoroughly tested. Um there's also a couple of important takeaways from this talk that I want to bring up. Um the first is that there needs to be more effort spent building smart wallets for custody providers. I feel like a lot of the effort has been towards how do we make a smart wallet for users when really one of the big use cases here is going to be smart wallets for custody providers. No one really thinks about this and custody providers in the first place are already nervous about this idea of running smart contracts. Um I mean you can the other takeaway is you

can actually build this stuff resistant to multi-IG compromise. That's fantastic. And last most important thing is that this is only possible for ecosystems with really robust development tooling. Um so like Ethereum is great because you've got formal verification, you've got fuzzing, you've got unit testing, you've got uh mutation testing. If this was any other ecosystem, I wouldn't recommend it because there's no way to test your contracts and figure out if you actually have the level of security required to do this safely. Cool. And uh that's uh kind of where we landed. Uh does anyone have any questions?

[Applause] Yeah. Um, due to time constraints, if you we invite you to take questions outside if that's possible. We'll take questions outside. Thank you very much. Thank you. Thanks for the great talk, Benjamin. [Music]