Layered Defense at Work: The Protocol Upgrade Saved by the Last Security Measure in the Toolbox

Transcript

Hi folks. Um I'm very happy to be here. I'm Kate. Um I am a lighter da contributor. So um and now I want to tell you how we ship releases at lida. So uh uh this is a quick note about what is lid. Have have anybody has anybody use lid here? Um okay. Lida thank [laughter and gasps] you. Lida is a major liquid staking protocol. It is running on Ethereum. You can think of it as a middleware or now almost sen infrastructure that connects users with ETH and node operators and it is governed by DAO which means that onchain

code and offchain policies are controlled by token holders. Uh by the way, if you are curious about how we are moving from the middleware to the infra staking infrastructure, I highly recommend you watching yesterday's talk by Yuri. Uh he uh talked about how we are releasing CIDA v3. So um what is CLA? Lida is a liquid staking protocol. You st either you you will get uh ST liquid staking you liquid staking token you can use it around defy and your stake is delegated to node operators based in class they are uh currently operating within three modules curated model with professional node

operators a model using uh distributed validator technology and a community staking module and um by the way uh why are we doing it why are we upgrading the protocol B because we are firstly constantly experimenting with it and secondly we are working uh in an involving infrastructure involving ecosystem but to make any upgrade for the protocol you need to run argon to to run voting it is placed on argon framework. So we use contracts to vote not just a multisk or something like that and you need LTO holders to vote to get your upgrades on the protocol.

Um who's in charge here? You can ask. It's LDO, our governance token. It's not financial advice though. So uh if even you are a contributor to the DAO, you can't just do stuff with the practical. Uh you need a DAO approval for that. Uh the governance process, oh wow. Uh the governance process deserves its own full talk. Uh or you can just grab me after. I love talking about biomechanics. But uh in uh simple words somebody proposes something uh on some of the frameworks we use for governance LDO token holders would voila a decision is made looks easy but uh it is not so uh so uh to uh

prepare an upgrade whether it's a tiny tweak or a fullblown overhaul uh you need time brain brain power and a fair bit of luck. On average, the whole journey takes around 8 n months from my perspective. But for example, we had one release that lasted for three years. Here's the preliminary timeline of it. And uh we don't have only one uh release. We have several of them in parallel.

Uh not to make a crazy impression, I I think that it might be better if we don't zoom in. So uh this is how it looks. This is a bird's eyee uh view of how lighter releases look like in practice. How do we cope with it? Light tech is not a monolithic team. It's a bunch of technical teams that contribute to protocol all at once. uh they have their own tech leads, their own road maps, they own rituals even and uh some of the teams work on the staking part and staking module or staking world.

Some of them develop modules for node operators. Uh some of them working on the offchain infra or governance tool or interfaces all at once simultaneously. Uh which is cool but it's also hard to coordinate all the teams. But this decentralization of efforts help us uh to develop upgrades in parallel. So to sum up, light upgrades are regular. We have uh multiple releases in parallel and ship almost every two months I'd say. Speaking of major releases, they are built by different contributor teams. uh they are of course overlapping in time while one one team

is wrapping up design other could kick off testing or audits they are very complex uh each has unique way of surprising you believe me they are very different in their scope in their domain and their size and they must be approved by token holders and this is last but not least so uh how it feels uh it means that some strangers on the internet have to decide on your proposed upgrade. Do the voters have time to go line by line through your smart contracts code? Do they test it? No. They I they wish they could, but usually they don't have time

for that and they believe us like the contributors to bring them something solid, something baked, something we believed in. And this is the moment when the true responsibility leaves when you come to them show everything and say this is ready check. So uh to ship upgrades without in a DAO governed protocol you really only need three things. First uh keep the standards bar very very high. Minimize risks and prioritize security. The minimum acceptable code quality at lighter is perfect. Uh the second one flexibility. So uh you need to build and

keep the processes very adaptable for the teams. Contributors should be supported by the processes. They should not they must not slow them down. And third transparency to comm we need to communicate with token holders clearly in a userfriendly way and provide them enough proofs looks easy. Uh so what we did we built a robust release process yeah it's very very huge so I think that it's even not the end and now it is not just deploy and pray.

uh we still pray uh but now we have a structured process for that with checkpoints, timelines and governance steps inside. Um we of course didn't make it in a vacuum. We gathered all experience uh the real stories, the ugly surprises, the last minute blockers, my favorite, and wrote everything down. So every step that might happen in your process and every time at the very uh early stage of the release we sit down with the technical or product owner of the release and adapt the process. So we just select the items that we need and

we reorder the whole process and uh what does it mean? It doesn't remove the risk but it gives you grip and uh the goal is only one not to miss seeing things and give the D confident confidence that the release is ready. Yeah. Uh so uh the checklist has seven stages from we have an idea to everything is done alerts are green vote is launched where is my tequila? So it has seven stages. Concept, drafting, implementation review deployment voting, and take off. Uh I think that uh a lot of people here know how to make technical releases. So I don't want to

stop on every of them. I will just highlight the things that I find interesting. So let's go. Uh the first stage is uh inception concept. It is the earliest phase the where ideas are born. Uh the teams describe business landscape. They define the problem space. Uh any highlight potential solutions. It helps us explain what we are trying to solve at the end of the day. Another artifact is white paper or ADR. It's architectural decision record.

uh it describes high level technical design and highlights uh the tradeoffs and explains why the chosen approach is superior to other alternatives. So and the next stage is drafting. Uh here we uh go deeper and um here's one thing that we learned. If you have time to think before you could, that's already great. But if you have someone who will challenge and review everything that you designed, that's even better.

So, it's a stage of a lot of uh of writing and reviewing. Uh tech technical leads, internal reviewers start to challenge the chosen approach. Uh technical spec, what's the difference? You can ask. technical spec it's a more technical document it's aimed at uh technical contributors and auditors and LAP it's is is aimed at the broadest audience voters delegate delegates u nontechnical contributors so or other for other guys who just want to look look through the solution and understand what is coming uh all the laps they are held at LA

docs. So you can change you can check it out and look through the whole history of upgrades at LA. And um speaking of uh internal reviewers, we have procco. It's uh protocol security committee. These are senior devs who have been with the protocol from the early days. So uh they uh check every release first on the design stage. they challenge the spec and at the final review they uh look through all artifacts they want. So the code, the deploy plan, the voting scripts. So uh it's a very huge stage.

Uh they put their uh judgment, their reputation and behind the every release. Um do you remember this three-year long release? uh here this first two stages concept and drafting took very long time and uh dual it was yeah I didn't say it was uh the dual governance release actually and uh the dual governance has intentionally sophisticated architecture otherwise you could actually replace it with a small security console deciding everything but that's just not uh the way the path we content.

Uh what is dual governance? Uh D you remember that LIDO is a DAO. It means that LDL holders make decisions but actually ST users uh put their funds into the protocol that that creates a potential conflict that govern government governance decisions might uh favor LDO holders over stakers and the old governance provides a foot voting mechanisms that lets STE holders um firstly um delay the in case For example, if some bad proposal occurs, uh they dual governance allows ST holders to delay its execution and

exit the protocol and moreover they can u block the whole governance. It's called rage quit mechanism and actually designing this mechanism uh took so long. It was so sophisticated and long. The next stage is implementation itself. We used to think about it that about that that we are done when the smart contract code is done. Uh but now we don't think so. You see that it is the first only the first item. There are a lot things uh at the list. Uh for example tests um we use different types of them. As for doll governance uh it went way beyond the unit scenario and

integration test. there was formal verification to ensure that the system behaves exactly at as it was designed. Uh voting script and tests are also prepared here. Uh another uh interesting things that we are using now for major releases. We put all voting items into the contract smart contract and deploy it on mainet and we ask our auditors to check it. So it uh makes us makes us sure that the voting itself also doesn't have issues.

Um review after uh it this is the more the most nerve-wracking part. Uh it feels like a pingpong game but you're the ball. Uh so a lot of audits are made here. LA has spent over five million on audits. And this is not because we're we like spending money, but because when you're shipping to a protocol which secures billions, you audit and then you audit again. And honestly, our audit partners help us a lot during the dual governance release. Special shout out to Sertora team and especially to Tom Genner. they helped uh to spot um several issues long before the launch.

So it helped to secure the system a lot and all the uh audit reports also held at lighter dogs. So it's 78 reports for now. Uh you can check all the issues that were found all the reports and like also uh look through the whole history uh deployment when uh you got green lights on everything on your code deployment plans voting scripts uh offchain code uh you the teams deploy uh contracts to mainet and this is uh an interesting thing we came up with we ask our auditors to prepare a deployment

ment verification report. It's a document that actually validates that the code on mainnet is the same that the they reviewed that that the commit they reviewed had and also they review the state of storage for contracts. They review the initialization part and rose assignments. it uh helps to get everything into one point and of course we make it public and uh all the voters can uh check it.

So uh my favorite stage voting uh everything is prepared. Uh you you did an extra effort to prepare everything and your teammates reviewed it and auditors reviewed it and now you have to inspire strangers on the internet token holders to believe you and to say yes to your upgrade. So uh we show them everything. We publish extens extensively. We show every number every number every item every method that the voting calls is backed by a public proof. Uh but the level and the how to say um

uh so the areas of domains of watering holders are very different. Some of them can go deep into your technical details. For others, we've for example prepared this comprehensive guides how to check uh onchain vote. So uh at first it has description for the vote and uh lately it has uh uh proofs for every number in the cold methods. And also to make it more fun, we prepare pictures with infographics for the voters uh to understand what is actually going on.

When the vote gains quorum, it is enacted and uh here we communicate of course with with everybody and uh the enactment part is actually the maximum risk point. It feels like we are launching a nuclear reactor every time. Uh there's a special ceremony for that with a separate checklist actually. But interesting part here for example we in advance we prepare a list of alerts that uh must be triggered during the enactment and then we check uh every one of them. So we check that everything predicted triggered and nothing in addition.

Uh yeah so uh you can think that everything is done and here the the supervision period begins but sometimes the last mile is where surprises leave. Uh after we launch the doll governance release on mainet we received a bug bounty report. Special shout out to the white hat for for submitting it. Um was it painful? Yes, absolutely. Because team teams made uh huge efforts to to ship it and still a mistake slipped in.

Uh but finding through it through our own bug bounty program while also while emergency uh emergency training wheels were active. I mean we have we have a emergency committee that uh is active for some time after the release and it can stop the dual governance for example because it was dual governance release was what much more better. So a piece of advice here add training wheels to your releases. This design uh helped us to downgrade the severity of winding from critical to high.

Yeah. Speaking of bug bounty, um we have li three on testn net. Actually the whole protocol has its bug bounty program but now we added li3 on test net to it. So go and see participate. Please participate. Uh I hope that it helps everybody. Yeah. Uh the presentation is a reminder of one thing. Build security in layers. Our security is layered deep. It's like an onion only we removed the tears.

That's it. Thank you very much. I'm Kate. So if you have any questions or want to talk about processes or releases, uh why release managers don't sleep, you're welcome. Thank you. >> [applause]