Full Web3 Security Stack - A Blue Team Perspective

Transcript

Um today we're going to present like um how we do security at Kilm to kind of give uh blue team perspective of uh the topic. Uh I've heard a lot of very interesting things on the red team side. But yeah, there is another side more defensive for the security engineers like me. So yeah, quick intro briefly. I've been at K for five years doing a bunch of roles but mainly focusing on security engineering since about a year now. Uh I'm here there is the smart contract team around. So if you want to catch up with us after this talk we'll be very happy to discuss with you. Um to give kind of a summary of what we're doing at Kon for the context of this presentation we are doing kind of the full stack of web3 products. Uh we have validators which we've been operating

since 5 years. Uh at the most we run it protocols uh mainly on um and we have the main validators on Israel and Solana where we are quite big. Um we also have smart contract products. So kilonchain are isam staking smart contract products. We have a liquid staking version a dedicated version that are live since three years now with more like 1.5 billion t something like that. Kindy is for the defy integration of lending protocols mainly. um it's live since about a year and we just announced two days ago our future product which will be an extension of the D51 to create structural product uh onchain with the nav on chain everything on chain new volt infrastructure um and then apart from the infra and the smart

contract we also have API for transaction crafting reporting and we have depths for B2B portfolio management and for B2B2C uh to integrate stuff there is a weird typo but whatever um so typically like if you use today like uh safe wallet or ledger live and you go into the earn section that's the kin widget that's actually white labelled so we cover kind of a wide range of um use cases and from a very high level that's what we need to secure at Kon technically we need to secure software infrastructure smart contracts but we also need to secure the actors that are using those so the employees who are managing these and the users who are interacting with our products and um taking a bit more high level approach like we also need to um take into

account the interaction between all these components. So like the user interaction with our um software, we'll talk about clear signing like the development experience of our employees, the production, the oracles. These are like all the things we need to secure from a high level perspective. Um and so the goal of this talk is to present kind of a blue team approach where we present for each of these components uh let's say the base layer of security and then move to um what we can call iterative security which is how do we add layer to our security stack to make sure that even if we get hack we limit the blast reduce of a hack and the potential impact an attacker can have. So this is a diagram we made for like anti-lashing

strategies but we really try to apply it this kind of mentality of adding multiple layers uh to our security stack. So let's start first with uh the infrastructure part of things. Um the three core principle of infra I don't know what why that does that but um the three core principle of infrastructure we implemented k are first like regular security uh reviews uh you have great certification like so to type 2 web three so ISO which you can do to have like annual external audits that's just a good practice to have we just released the bug booty program for infrastructure and continue minifi haven't seen a lot of web three company do that I think we should all push for like bug booty outside of smart contract as

Um continuous thread detection is very important as a basis security layer for infra. So basically making sure that you know in real time what's happening. You have detection tools like whis that can tell you hey why is this service exposed to internet? Is it unexpected configuration we have CMS like punter where you can aggregate all your tools like whis and others into one place and do correlation detection. Um and these companies are great. They provide like team like 24/7 sock which help you uh basically write and challenge your alerting and different monitoring you have uh and also help jump in case of uh issues. So that's also something we we recommend to have when you're doing infrastructure and of course the last one is a strict deployment uh process on

infrastructure side of things. So we do infrastructure as code and we have pipelines to that are auditable and audited to make sure that the infrastructure we define is actually the one deployed and that you cannot really messed up in the middle. So that's the basis of like infracurity and now what can we do iteratively especially when we think about uh web three uh to secure it. So let's focus a bit on the what we could call the production infrastructure which is the infrastructure that operates our software our oracles and other sensitive services like that. So the first basis we implement at this level is um like making sure that the application we pull on the infrastructure is actually the one we

want. So uh the basic stuff are for sensitive services to pull with like the ash of the image not only the version make sure that um the image version ash is uh is pinned. Uh we also like exploring sign signatures of ash versions. So making sure that when we build the image we sign the ash and we verify that when we load the image on the infrastructure side making sure that in the middle uh nothing is modified. And uh other stuff we do are like uh read only the file system uh for the applications uh to make sure that again uh we have different layer of mitigations. If an attacker has access to a pod, he cannot modify the file system of the application. You can do that as well with egress for your network policy to make sure that you

cannot do extra external code that are not planned from the beginning for your application. So this is base layer of things you can do for applications and then you have web three applications. Websu applications are a bit uh different in the sense that they need a signing key to operate. So you can think of validators that need to sign attestation block proposals. You can think of Oracle that needs to sign smart contract authorized smart contract interactions. This need further protections. Um and the basic one is to separate the application part uh with your uh signing part. So basically not have the signing keys within the application again to make sure that if an attacker has access to the application he cannot do much cannot modify file system cannot modify network

policy cannot have access to the private key um you have like great tools like ashik v to store the keys web three signer to load them and sign in one central place and that makes sure that it's even more auditable like you can focus on your web three signer setup monitoring alerting access control to make sure that the signing keys are always in one place and secured Um something we did to incrementally uh adapt and add new layers of security on this setup is contribute to web designigner uh on the past months to make sure that web designigner has this option to only load keys in memory.

Before that keys were always stored on the file system. This is the kind of small improvements we're trying to make to community tools as well to make sure that we can yeah um iterate on the security setup. So now we have uh now we have read only file system also on the whip signer signing part. Um and something we also adding iteratively is security proxy. So we're doing like typically for oracles. Security proxy is a tool like you can develop very simple piece of software that basically enforce a policy on what the application ask uh your key to sign. So typically think of a very simple oracle you want to have like that. uh you could like check in the security proxy that uh the

application of the oracle is only requesting signature for transaction to a certain contract on a certain chain with certain function sigs and you can even like check for example if if you report a rate with your oracle check that the rate is within certain bonds and stuff like that otherwise you just drop the http query and the oracle will won't be able to sign any potential malicious um oracle report um and yeah this is the kind of yeah again the iterative process you can put in place in your infrastructure to make sure to protect uh this sensitive uh private keys that can do authorized action on on the production. Um and again uh moving again further on this you can have the signing uh

infrastructure in a very restricted infrastructure create your new GCP account at WS can't make only very special access control for this so you prevent lateral movement if someone has access to the application if someone has access to the application cluster you cannot move uh to your signing infrastructure signing cluster um so that's kind of how we think as a more like blue team and infrastructure team at Kon to make sure that we prevent lateral movement uh from happening between our different components. Um we have also kind of the same thing as the security proxy I mentioned for the Oracle but for our software stack stack. So typically like we have a transaction crafting API which like transaction you can uh sign it abstract

basically the complexity of crafting transactions. Um and we added the proxy at the reverse proxy level on a different infrastructure as well to make sure that we verify what the transaction API is returning making sure it's not like malicious payload and that guarantees that basically we have uh even if like the transaction crafting API gets compromised uh well we never the customer never received the FTP response with the invalid payloads. So this is again like think about proxies like that or like small iterative step you could do in your tooling to prevent bad stuff from happening if an attacker has access to one part of your infrastructure.

Um then focusing more on the top of this diagram, users interaction. I guess you all know about clear signing now. Uh we've been very proactive in doing like um ledger plugins for example, but now they have this great ERC where you can just define your smart contract interaction with JSON um and it will be well listed out of the box. So we highly recommend everyone doing that. That should be a basis for every protocol. Um but doing kind of the extra mile also on that we provide a tool that's open source um uh audited running on another infrastructure uh that's called Minel where people can just go uh put their row uh transactions taking transaction and Mel will just decode the transaction

and check for anything weird anything off anything that shouldn't be done within this transaction and highlight the risk. Um again like we want in this part of the stack users interaction to make sure that the user can always um clear sign double check whatever they are doing. Um so first with class signing and then with this kind of open source tools we're we're developing. So that's part also of our blue team strategy at K. uh then focusing on on what might interest you more the smart contract part I guess the first basis of security as a blue team for smart contract you all know it right it's to have a a very secure uh security first development cycle with internal reviews a lot of testing fuzzing and so on um we also

have like uh audits uh with multiple providers we use spare bit trail of bit a lot uh and bug bootying programs but doing kind of the extra mile what we want to do is also prepare um when uh things break, right? We want to be prepared in case of outage, in case of an attacker doing something on the production. And the first thing is real time alerting uh and to be able to move fast. So we use Exagate. We also have internal alerting engines. Uh it's yeah pretty easy to set up nowadays with tools like graphs and so on. Um and we have like incidental on top of that to be paged. So smart contract team uh security team are always on calls. I discussed recently with an auditor

telling me that most of the his clients do don't have on call for smart contract products. I cannot really believe that but I hope everyone has that. Um and then the goal with this on call and alerting is not only to be waken up at night but also to be able to contain fast in case of a problem. So the main thing we do uh and we try to really empress this to our uh team on call team is to make sure that uh we do not spend time finding the root cause but first we spend time when things ring finding uh the impact and if we have any doubt we p the contract we post everything. Um we have post pattern same it could be nice to discuss more post patterns if you have more ideas. Uh we for example implement a post for pattern in our

smart contract product where you can unilaterally as a non-al person pose for a certain duration. Um so like you have time to investigate in case you have a doubt and then you can do a more long-term pose which require a governance and pose. It's a bit more heavy. Um but this way like you can do fast pose in case of doubts like just pose for 30 minutes investigate and then resume activities. Um of course as a blue team as well we're trying to focus on communication if things break. So making sure that every team has templates uh things ready to communicate to customers uh on Twitter and whatever on support to make sure that uh if we pose everything or if there is an issue with the contract uh it's like everyone knows and knows that we are

investigating and on it. Uh nice. Uh the last part of uh here um was mainly about like uh finding the right resources in case things break. So typically like having foreign sync resources ready uh having a contract with your favorite auditors uh to make sure that they can audit fast in case you need that during an incident or help you like track phones for example or stuff like that. uh there is C911 which is very great uh and a great resources uh a lot of talks talk about them but I think that as blue team and as web security company we shouldn't rely on them and we should proactively have SLAs and contracts with different security

auditors to make sure we have war room ready um and also the last thing that's written be behind this cryptographic weird stuff is um that uh we plan war room contacts so basically we have this document defining lab room process and the different key people to contact both internally and externally and that make sure that um we also have multiple uh communication channels because uh it can happens during an incident that like Telegram is down or Slack is down so make sure to always have a redundancy there. Um and then focusing on the last part of the stack which are the uh employees. Employees are like kind of uh interesting because you know they uh they are these privileged users that are

interacting with all your uh components, infrastructure, software, smart contract and governance and you don't want them to become thread vectors. Um so typically like the how they become thread vectors is like well they are malicious or they take malware, social engineering, fishing. Um so for that what you can do is uh enforce uh great device management for all your employees. uh typically like using uh crunchy and construct to make sure like the laptop are all up to date uh and that you have continuous malware analysis and then making sure you have an identity uh workflows between your service and the employee laptop. So typically SSO with octa and uh private networking with tail scale. Um we also

like uh like do communications implement tools like Riot to make sure that our employees are all formed and trained on the uh on all these potential issues they could face. Um and it's very important because uh this year we've seen a big rise in like scamming attempts toward our employees whether it's like Zoom links uh weird calendar invite fake customers trying to to join in. Um so yeah this is something you should do proactively as well the training part. Um nice it's even more cryptic now. Uh so focusing more on the employees and what you can do there is the infrastructure part the access part which we need to protect. So here you we have like multiple things in place. Uh

but the first principle is really to make sure that you cannot do lateral movement as we mentioned. So defining your infrastructure role very clearly so that you attribute them clearly to each employee and each team and make sure that well there is no possible escalation. Um and then we're also trying u um like we're also enforcing identitydriven access. So typically we mentioned SSO octa these are the kind of tool you can put in place and um finally we're also exploring uh some tools to help doing multi-approval flow on infrastructure critical actions so that if one of your engineer need to do manual action on restricted infra it needs multiple approval of other employees.

Um then there is the development experience which you need to protect. Uh the first part obvious part is commit signing making sure that it's actually the your employee signing the um the commits. So for that we use like UB keys. Uh but you can also enforce more stuff on the codebase. So like IP whitelisting with tail scale uh and SSO access that's the basic of like identity access to uh the code. Um and then you can enforce other policies. uh typically we removed all personal access tokens and both accounts from our organizations and that the only like automated accounts we would need on GitHub are all provisions through SSO and managed by the security team. So make sure as well that for this you define clear processes

for your team and your developer team um to uh to have these kind of tools and know what they should do and don't try to generate weird uh access token on the side. Uh and finally smart contract operation. So typically like your employee would participate into smart contract governance. Um here what we want to do of course is to have no uni unilateral action. The only unal action you want is for pose but for the rest you want to have a signature process. So typically we use like safe for all our um uh smart contract important roles um and corum with security auditors and external partners. And we also uh started developing this year a governance as code approach. Um so we

developed this tool uh as a security team and we might open source it in the coming months. It's called Agora. And basically the idea is that um our smart contract team can create proposals like governance proposals with a YAML specify for script and for tests that must be valid after the proposal is executed and Agora and then they create a PR on GitHub with that. Uh and Agura will just post on the PR and on Slack and on whatever you want. um the message with the simulation of the proposal uh the simulation of the execution test after the proposal uh and different like metric like we use exagate for example to double check all the transaction for risk uh and then our team can just come in on this PR analyze uh the simulation

approve the PR if the PR is approved with safe delegate it creates the uh proposal and safe and then it comes kind of in a live mode where you can see all the person who signed the proposal who are signing the proposal and you can see like all the hashes of the proposal like of the different nested safe so that uh our employees our partners who are in the crooms can all double check what they need to sign and the different ashes whether it's on their hardware wallet on agora on safe UI or whatever UI they're using. Um so yeah again this is typically the kind of incremental step we're doing on blue team security to make sure that all our processes especially dealing with smart contract um are uh properly secured and uh

uniformed and monitored. So thanks for listening uh to this uh talk with cryptic slides. Uh on the left you have um my Twitter if you want to come reach out and go into DM discuss these tools I mentioned and on the right you have our four bug bunt. We have three uh smart contract by multi live and one on infrastructure and the apps on continue unifi. So if you want to help us secure K um please yeah reach out and go there. Thank you. >> All right. Thank you Lo. That was fascinating.