The Death of the __gap: Custom Storage Layouts - Eric Marti Haynes

Transcript

All right. Hello everyone. Uh, my name is Eric and today I'm going to talk about the death of the gap and custom storage layouts. Okay, so a little bit more about me. Uh, for those that don't know me, uh, my name is Eric. I work at Nethermind as a blockchain engineer and tokenization tech lead. And yeah, my focus is on developing smart contracts mainly in Solidity. And for the past few years, it's been mostly around tokenization. So, uh let's start with upgradeable smart contracts. Something that many of you here should be familiar with. Um we have a few issues with them, right? And the main problem is if you want to have

the logic of a contract changed, it's very complicated because a smart contract by default is immutable, right? Um there's ways to get around it. Basically, we have a solution where we can use an upgradeable proxy in order to keep the data in the same place, but have the logic at a different address that can then be replaced over time. And the benefit of this is you have one address, it stays the same, but you're able to update the logic over time pointing at uh different implementations. So um I wanted to give an example that is uh a bit practical. Let's say not super practical. This is a bug counter

but it's not super abstract either. So in this case we just have a contract uh that's called bug counter and it inherits from arthropod counter because arthropods are bugs, right? Uh it has a warm variable because worms are bugs. Um and yeah that's all it does basically. You have a few view functions uh like the public uh variable worms that is uh a view function and you then have a initializer disable constructor pretty standard for a upgradeable simple contract and then an only owner function um to update the value for worms and then basically a bugs function that returns the total amount of bugs. Okay.

Um here we can see as well arthropod counter which is the contract that the bug counter is inheriting. Uh and this is quite similar and here we have insects basically uh and we're able to update the amount of insects and then return the total amount of arthropods. Insects are arthropods. Okay, pretty cool. So uh this is what we have in storage right now, right? We have insects. We have worms. Insects are stored slot zero. Worms are at storage slot one. And we have a value of seven insects and three worms. This is pretty standard for a simple solidity contract.

But now what happens if we want to upgrade this and suddenly we realize there's more types of bugs. We actually have software bugs, right? This is a kind of bug. Um and so basically we decide to add uh a variable for it, but we add it before uh worms. Um, but then we also add a function to update the amount of software bugs and we also add them to the counter function. Well, this is what happens to our storage, right? Uh, this is not great. Suddenly, worms have been replaced with software bugs and it says there's three, but that's clearly not the case. Now, we have zero worms. Uh, so it's all messed up. Of course, to avoid this, what we have to do is put the variable after uh worms.

uh and in this case as we can see here it gets appended at the end and we don't run into these issues. So this is already something a bit tricky like dealing with storage. Uh but what happens if in the arthropod counter contract we now want to add arachnids which are of course a kind of arthropod um and we want to add as well a function to update the amount and we want to add them to the counter. Well, now our contract is storage is messed up again because aragnets have been added after insects and then worms have been pushed down pushing down software bugs and now again everything is messed up.

So uh this is an example of a bot smart contract upgrade let's say um we have to be very care careful about upgrades um from lesser to more severity there's some things that uh some impacts it can have right the simplest is basically uh an increased additional time and resource just to deal with upgrades double check everything make sure we're not going to run into any issues um but of course then we can have quite more complex issues like for example we can have our uh smart contract storage is all messed up to the point where there's a denial of service. It's not working properly anymore and users are not able to use it or in the worst case scenario we can have an exploit and a loss of funds of users which obviously is is is the worst case scenario here. Um an

example for this is audience. audience is basically like uh web 3 approached to Spotify, let's say, and in 2022 after an upgrade, uh it lost $6 million of its native token after a malicious governance proposal was passed due to a messed up uh smart contract upgrade where the storage was layout was impacted. So this is just one example, but it's quite clear. So uh let's talk about storage gaps because this is one possible solution to the problem. Uh but as we can see and in the title of the talk uh storage gaps are are dying right and we're going to see why. So basically they were introduced around 2019 by the open

zeppelin team and their purpose is basically avoiding slot collisions like we've seen and the way they do this is by reserving uh empty slots. In this case, usually it's 50 slots uh by using large arrays that are added at the end of every contract in our inheritance tree. So let's take a look. Let's imagine we have an alternative version of our bug uh counter and that from the very beginning uh we've added a storage gap. We've been smart about it and we we knew we were going to upgrade this in the future. Uh so we added a storage gap 50 and so now well sorry arthropod counter right but let's say we add it as well for the bug counter and now let's say we add aragnets and of course we're going

to reduce the size of the gap by one because out of those 50 empty slots now we want to use one of them. Um everything else is the same but as we see now uh this actually works. Um we've replaced the first gap uh with aragnetss with the proper value and you can see our storage slots look a bit different now. It's basically because we have these 50 slots occupied by the gap. Uh now it's 49 because the first one was replaced uh by the aragnets variable. Uh but after this 50 gaps uh we have warm we have software bugs and the values are maintained properly.

Okay. But storage gaps uh have some issues. Uh you have to mind the gap. So first of all uh you have to use a fixed size of gap. Uh usually it's 50. Uh some cases it's even like a thousand but you have to know ahead of time how big you want to set the gap and you can't really modify it or make it larger in the future without again impacting the storage of of other contracts. Um a big issue is the manual updates. uh you basically have to make sure every time you modify storage in an upgrade uh you align the storage of the of the gap to make sure it it it compensates any new variables you added and then of course uh the inheritance tree is does not really play well with the storage gaps

but we're going we're going to see that. So let's take a look at another example. This is an example where we went kind of quickly through our code. We did the upgrade and we forgot to actually decrease the size of the storage gap from 50 to 49. So let's see what happens. As you can see here, we still added arachnids correctly and we still have the gap. But now the gap spans 50 slots instead of 49 because we didn't decrease the size. And so again, we can see worms and software bugs have been shifted down and the values are wrong yet again. So this is something we have to be careful with.

Um, let's take a look at something else. Uh, we have bug counter. Let's say we decide to inherit a different kind of counter apart from the arthropod counter. In this case, it's a gastropod counter because these are bugs as well. We suddenly figured out, hey, there's many more bugs here than than we thought. Uh, and of course, we add the gastropods to the total counter function. And this is the gastropod counter contract. It's very similar to arthropod counter. Uh but here we're counting snails and slugs which are kind of uh earth uh gastropods and we've added a storage gap as well because we realized okay we have to do this but let's take a look at the result.

So you know there's lots of them. Um but here the problem is we have two gaps one that goes from two slot two to slot 50 and one that goes from slot 53 to 102. And as you can see, the snails and the slugs have pushed everything out of the way and then the gap after it as well. And we only get back to our worms and our software bugs, uh, which are not really showing there exactly, but trust me, they're there. Um, they've been pushed to the very bottom and it shows zero, which is obviously not the case. Um, so whenever we do an upgrade and we add a new contract to the inheritance tree, even if we're using storage gaps, it's going to mess up everything.

So, uh, let's take a step back and talk about something else now. Um, you'll see why. It's, uh, diamond multifacet. Uh, I'm not going to get too into it. What you need to know about this is it's a proxy that has multiple, uh, instances uh, and interfaces that can interact with it. And they all share the same storage, which is pretty cool, and it has some interesting use cases. Uh but you might wonder how do we make sure uh the storage for all those different instances don't collide right so we use something that's called diamond storage and this has the following formula where instead of each contract starting its storage at slot zero uh we actually hash

uh some string in this case we can say my project 20 if we have an ERC20 and that hash is basically the slot at which we're going to start our board and this allows us to have uh different uh interface contracts that connect to the proxy without overlapping their storage. So basically diamond storage uh uses identifiers based on uh string literal. Uh it's not limited to the diamond proxy. So you can use diamond storage in any contract you want, which is something quite powerful like we're going to see. It's currently being standardized with ERC 80042 basically to kind of retroactively uh validate this as its own standard

separate from diamond proxy and it can be used as an alternative to storage gaps uh like we will see. But then we also have ERC 7201 which is namespace storage. Um, this one has been used more consistently, I would say, and adds some additional features and is generally more used uh and but it's quite comparable to diamond storage and they're both doing very similar things. We're going to focus on this one as an example, but everything we do could also be done uh with diamond storage. So, uh this one has a slightly different formula. As you can see, it's a bit more complex. Uh basically we're hashing twice and then we're making sure uh it ends in uh 256. Uh this last bit is uh

for a possible future gas optimization. And the double hashing is basically to make sure under no circumstance it's going to collide uh with some other storage that our proxy uh contract might have. Yeah. Um so I mentioned these first two. uh it's standardized with the RC7201 and again like diamond storage it can be applied uh as an alternative uh to storage gaps. So let's take a look and let's imagine from the very start we decide we're going to use it for our upgradable contract uh because it's a more robust alternative to storage gaps.

Uh but as you can see we've had to make quite a few changes to our code. Uh first of all our variables are no longer in storage. they're in a strct and we have to load them in every single function that we want to use them. And how we load them is we've defined a specific uh storage slot using the ERC uh 7201 formula that we've hardcoded but you can see above in the comment uh the formula and then we also have a helper function that uses assembly to basically uh load this stored slot. Uh, one additional thing you can notice is we've had to add uh, getter functions because we can no longer make our variables in the strct public. So, it's a bit more verbose and we've had to add that.

And here we can basically see um, sorry, I'll go back just a sec. This is our bug counter, an alternative version if we had just added this upgradeable pattern. And now we're basically adding gastropod counter to it. And as you can see, the changes are quite minimal. we just have to inherit it and add it to the counter function because we want to count the gastropods. Uh but otherwise everything stays completely the same which is quite good. And as you can see this works. Uh we've been able to add snails and slugs and everything in the storage works perfectly. As you can notice though, the slots now look quite different because instead of starting at zero, uh we're starting at the at the hashes basically.

So you can see it still ends in zero and one for each uh contract. uh but it it it starts in a different place. So this is how the arthropod counter looks. Now we've applied uh the the same pattern. It looks pretty similar. We've had to add like uh all these extra bits of code. Um the same for gastropog counter. Um and okay, this is good, but um namespace storage still has some issues. Even if it's great compared to gaps, it's not perfect. Uh first of all, the manual hard-coded hash can be problematic because you know you can easily copy one contract into another when coding an initial version and then you forget to update this hash and

suddenly uh you have them in the same place which would obviously not be good because they overwrite each other. Um also the usage of assembly even if it's quite minimal uh is not ideal to have to resort to and also as you saw the uh syntax is verbose and complex and if you're familiar with it it's fine. I personally like using it, but for someone that's not very experienced with this pattern, they might look at it and think, "This is a mess. Why do I have to do this?" Uh, we have to figure out something that's a bit simpler. So, let's now talk about custom storage layouts in Solidity. I'm not sure how many of you are aware, but we actually already uh support them in Solidity in version 0.8.29.

um we're basically able to uh relocate the storage of a contract instead of in slot zero in any arbitrary slot uh that we want. And this was mainly done uh for EIP7702 uh which is the EIP that makes it so EOAS can house uh smart contract code and kind of act like smart wallets with account abstraction. Uh there was the concern that different wallet implementations might class their storage locations. If I'm a user and I decide to switch uh from one wallet to another and obviously we don't want the old values from the old wallet to collide with the new ones. So we can use this to make sure that they're at completely different slots and that then there's we're going to avoid that

collision. So this is pretty cool and we can actually use it for our bug counter. Uh we use a syntax called layout at in this case we're even using uh ERC 7201 as well in conjunction. So again, we calculated and hardcoded our uh hash for our storage slot. And now we just need to say layout with our slot and that's where the contract will start. And as you can see, this works. Our storage uh looks identical. Uh but there's a catch. Right now there's only support for uh one single use of this in the entire inheritance tree. So if I check here my bug counter uh this means all the

contracts in the inheritance tree are going to start from this slot not just the bug counter um which is obviously not ideal because then it means we can't use it for upgradeable contracts. uh if we want to use it like I did in this example, arthropod counter and gastropod counter still have to look like this and still have to resort to this verbose syntax um just to make sure that we're using a different slot which again not ideal. Same with gast counter. But the good news is that the solidity team has announced that one of the features they're trying to prioritize is basically extending this uh much more.

So you can apply it to all the different contracts in your inheritance tree and then this will play very nicely um with upgradeable contracts and again trying specifically to support the pattern of ERC 7201. Um so what could this look like? Uh we don't know 100% yet because it's something still in discussion. uh you can actually go in the in the in the forums and discuss and kind of provide feedback. Uh but here I'm going to take some of that discussion and some of the proposed syntax and kind of uh draw on that to see how our contract could look if we already had access uh to this feature.

So uh here you can see the entire contract here. I didn't uh remove any like constructor or anything like in some of the other ones just so you can see the entire thing. And the main things we have are the layout at and then the ERC 721 formula. Um but we're we're going to take a look. Um this is before this is our simple contract, right? Like the simplest version. No upgradability. Sorry. Uh no uh helpers for upgradability like GAPS or ERC 7201. And now with uh custom storage layouts, how it could look like is this, right?

It's it's not a lot, but it's very powerful. Um we basically have layout at for every single contract in our inheritance tree. So we can specify where uh bug counter goes, where arthropod counter goes, and where gastropod counter goes. And also you can notice we're using the ERC 7201 formula. This is not required, but we want to use it because it's uh quite neat. And so we've defined a helper function for it that is going to calculate the specific storage slot it could go into. And then in layout add at the beginning of the contract all we have to specify is just a string of where we want it.

And for arthropod counter and gastropod counter as you can see they remain super simple like the normal implementation and they're not any more complex either. And this is what we have as the result. Same as before. Our storage looks fine. It works properly. Uh but again the syntax is way more simple and nicer to work with. So uh this is a bit of the timeline right we used to have nothing then very quickly we had storage gaps uh they've been used for a long time uh people are still using them today but now we're starting to see a lot more use of ERC uh 7201 and hopefully uh with these new features in Solidity be it classic Solidity hopefully or even if we have to wait

until core Solidity to get this full feature um then we're going to have custom storage layouts that are a lot more powerful and we're going to be able to deal with upgrades and our smart contracts in a much more simple way. Um, so hopefully this kills the gap completely. Uh, it's been great initially. Uh, but we have to move on to something better. Um, so yeah, that's all for my talk. Thank you very much. Uh, let's build and upgrade together. And yeah, feel free to check out the code at my GitHub or you can message me on Twitter if you have any questions.

Uh, thank you very much. Thank you so much for the wonderful talk, Eric. Um, we have a few questions for you. Mhm. >> Do you think we actually need the new diamond storage ERC considering the that name space namespaced storage is already widely used? Is there any benefit? >> I would say uh the benefit is mostly for standardizing a pattern that was already used in the past. Uh you could argue it's a bit simpler. Some projects might want to still use diamond storage. Um, I prefer ERC7201 and I use it personally, but I think there's no harm in standardizing something that maybe should have been standardized like uh from from uh the very beginning. Uh

maybe even like storage gaps that weren't really standardized like they were standardized in the sense like uh the open Zeppelin contracts used it a certain way and many people copied that but there wasn't like an ERC that uh people were following. Why come up with the new formula in ER ERC 8042 and not reuse the ERC 7201? It feels like it is just going to split the ecosystem. >> Yeah. So, uh this might be a bit of a misconception because uh the ERC for 8042 like the number is higher so it seems like it came later but actually um this was introduced all the way back with diamond storage, right? Uh so it's part of ERC. I think it's 2535.

So that's when it was started to be used, right? It's just now they're basically saying, okay, this has been used separately, not just as part of diamond proxy, so it should also be its own ERC. Um, but it's not something new that's been added. It's something that's already been used historically. Um, but any new project obviously like you can use it, but probably you will use the newer ERC 7201. Um, why don't we make the storage slot calculation a primitive the same way as EC recover? This is something that should potentially be used by every upgradeable contact.

>> Um, this is interesting. Uh, I think it's something we could consider. Personally, I don't have a lot of experience with that, so I can't give a lot of comments. Maybe it's something more for the Solidity team uh to comment on. uh I'm more focused more on the solidity builder side, let's say, uh rather than like the primitives that go into the language. Um yeah, that that's what I can comment. >> Yeah, that's that's a good point. Uh feel free to get in touch with the team. Some of the team members and the engineers are still around. We have one more question.

>> What do you think about declaring the function that return the struct that set the slots internal in instead of private? because that's something I insisted on and >> h okay so you mean uh uh like here right >> yes storage right >> yeah um yeah so I I would say like it it makes sense uh internal as well if you do it internal you can then access it from the other contracts right um in this case with a simple contract I just wanted uh to make it clear that like each storage um should only be accessible in this case uh like from the

gastropod counter if it's for gastropods etc. But yeah, in general it can be nice if you have multiple contracts uh then you can access the access them easily from one to another without having to call like other functions let's say. Yeah. >> For Eric. Any more questions? >> Okay. Thank you so much for that amazing talk. Please give it up for Eric.