Audits: Conventional vs Community Panel

Transcript

thank you muli and thanks to mudit for uh for the Segway and the spicy appetizer so um so yeah so I'm Rajiv the founder of securium um just to set the context we'll assume that most of you are familiar with what audits are but the topic of this panel is the flavors of audits right so audits have been around for a while and recently we have seen different flavors in terms of who performs the audits many other attributes as well so we're going to focus on sort of a high level question of

conventional Audits and the community oriented ones or the competitive ones right so but before we start off we have a fantastic set of panelists here would like to request all of them to introduce themselves and their projects before we kick this off um my name is Hari I'm one of the co-founders of spirit previously I was working at the durian Foundation Building the sorority language and the compiler um yeah that's it Gonzalo co-founder of consensus diligence been operating for almost seven years now mostly services off chain on chain a lot of tools as well buzzers symbolic execution engines so on

and so forth and also we have a stealth thing well kind of stuff that it's called Creed that is coming out so that's cool hey everyone my name is mehdi co-founder and director of Sigma Prime we are a blockchain security firm been around for about eight years doing a lot of small contract Security reviews but a bunch of L1 l2s and bridges and whatnot we're also the founders and maintainers of Lighthouse the rust implementation of the ethereum consensus layer currently responsible for about 35 of the user market share hey guys I'm Jack Sanford uh one of the co-founders of Sherlock and Sherlock has been around for about two and a half years we are a smart contract coverage protocol as well as an auditor

hey everyone I'm Joseph I'm the engineering director of the blockchain team at Trails Beats thank you we got a Poverty of mics over here no no it's okay we're all good we're we'll share uh my name is Mitchell Amador I'm the founder and CEO of immunify and unifies the major the biggest bug Bounty platform in crypto so you know these specifically these Define Unchained exploits are bread and butter um everything we do is about making Project SAFE post deployment and that ends up being a lot of things but uh bug bounties is always the name of the game hey there uh sock from code Arena one of the um Founders and first uh first handful of people in the Discord on day

one um the uh code arena is a uh creator of the competitive audit and we've just really been focused on creating as many opportunities for people in the field to level up and to create as many opportunities for uh protocols to achieve a greater level of security at a more efficient rate than um than a traditional audit awesome so let's maybe level set a bit more so audits we just use the term we may want to use reviews or assessments but I think we're just stuck with it at least for for this panel and maybe a few

more years but what do audits mean to each of you right for each of your projects your missions because the way we see it we may think of audit as something the protocols may think of may have different expectations the users may have a totally different you know zero bug expectations so what do audits mean to each one of you and uh also what's conventional or unconventional about each of your approaches I think that'd be a good start and we can we can go around for this question because each of you is very unique so maybe start with you want to start on that side the Poor Side of mics

well I mean the the reality is that you know I think actually there was a talk that uh Mitchell gave at uh trustex last year and was talking about the fact that bug bounties are really the um absolute last line of defense when it comes to security and the idea that um uh that in fact it it is an indicator of the immaturity of the space in general that like we have so much that's secured by bug bounties and I think in in reality I would actually extend that a little farther and say that an audit actually should be the last line of defense that uh a bug Bounty is a break glass in case of emergency you know at

the point that you're that you know kind of the going rate is maybe 10 of user funds uh that you're you're maybe overspending at that point in terms of what you might want to to spend and so an audit is intending to prevent at the you know last bit the as many bugs as you can from uh reaching production so that's the main thing that I see but it's actually an assessment of what you've been doing from day one when it comes to security okay so I mean you're you're totally right just to call out that one particular point about the whole movie this is move Left Right In traditional security right always move farther up the stack earlier in the process more

and more preventative but if you feel that way about bug bounties you should see what people say about their auditor spend of course I think we've all spent a lot certainly I have and we we don't really have a a choice about that matter the reality of the fact is it's impossible to create secure software and and we've decided to put the most valuable things in the world in a extremely kind of finicky young immature technology stack we're like yeah this is a good plan and by the way let's move the entirety of the world economy on here while we're at it so under these conditions where we have the highest possible security needs and we just don't have enough time we don't have enough people we don't have enough anything like you know what is what is an audit is just another type of code

review right we have our internal tooling inside which is ideally in an Ideal World we would we would have tools or we would hire people and they would just find everything and life would be good and then we would never need any external code review but that doesn't work right for reasons that we all understand and so then we go to the layer of external code review and that begins the process and usually typically best practice teams will start with a traditional audit right which you're going to have highly expert people from highly vetted firms who've been doing this for you so just phenomenal phenomenal Engineers who will help you not just look for vulnerabilities in your code but say Hey you know is the architecture here wise you know are you you're getting boxed into such and such a choices have you you know foreseen the

second third fourth order consequences of that and from there right then we have the the remainder of the flow right the new types of audits which I think are very very exciting and then bug bounties which is really just another type of code review right it's you could think of it as another type of audit but really it's the action of like of let us review this code and this the entirety of this infrastructure with the scope widening and widening widening as you get more and more closer to production let us review this and see if we can find anything that allows us to break the assumptions in such a way that causes you know irreparable harm to the mission of that protocol which in our case of course is theft of money or damage of valuable assets yeah so if we if we think about

what are our goals what are the things we are trying to achieve you know with like this Security review or audits um or you know this type of things from a traditional review for better lack of term what we are really trying to achieve that we are trying to help developers to understand where they are at what are like the risk they have what Alex they are aware they should invest in terms of security and to achieve this goal we have a couple of things to do first one is going to be you know to look for vulnerabilities and that's the most obvious one but a lot of the work that you actually do during Security review is to advise a developer to tell them look here you have a systematic risk in the where you are designing your component maybe you know try to refactor

it in this way maybe you know create like a single component for this like you know multi-component step maybe improve your documentation on this specific aspect so a lot of the work we do is really about helping and kind of filling the Gap with respect to the current maturity of the client and with respect to the risk they might have um that's interesting I would actually disagree with that I think the core there's only one thing you really care about in an audit and that is finding problems that will cause losses for users like making sure whatever this protocol is supposed to do it does it well and it doesn't cause massive

amounts or even a material amount of fun loss for users um and that should be the the main goal and gas optimizations informational you know findings that kind of stuff is great but it's not core to what an audit is um I even think of fixed review Sherlock includes a fixed review after every audit we do but I don't think that's core either um the core thing is did you find as many of the problems that could cause losses for users in that code base at the commit you were given as possible um and to answer the second question what is the conventional or unconventional approach of Sherlock

Sherlock is kind of The Best of Both Worlds so we have a foot in Both Worlds we have the conventional auditing approach where we reserve Auditors pay them a fixed amount of money to show up and do an audit and we also include an audit contest component um and yeah I mean in terms of the title of this panel conventional versus Community which one is better Sherlock thinks they're both absolutely essential to conducting a good audit and you need both of them and Sherlock is the only one that actually has both of them um and part of my talk at 5 20. later today if anyone is interested

is going through the last year of seeing which of those two parties found more vulnerabilities over the last year in Sherlock audits like who has been carrying their weight I guess between the conventional and Community side so stay tuned for that I'd be very interested to come to your talk because um I wonder how you're able to qualify the number of critical findings High severity findings that a firm like my my firm or trail of bits firm or Gonzalez firm has uh has been has been doing over the past year just just because it's very difficult for us to publish some of our work right so it's oh yeah I'm just curious about your data collection

method but I'll rock up in a few hours good looks like we are warming up to you in the audience a bit cold as well so um yeah sorry just circling back to everything that was said I I don't disagree with anything that you guys have said I think everything makes sense um the way we tackle security audits at seek p is the traditional way their time box security assessments you got to think as well that not everyone's willing to spend you know hundreds of thousands of dollars on a Security review right and in some cases people come to us and they're like hey we're cast strapped but could you work with us it's difficult to say no right especially if the project is interesting

uh the team is you know Keen to work with you so in those cases you know we may allocate a couple of weeks um but everyone knows that's nowhere near enough to dive into everything to dive into every single component that makes up the platform so um I'd be hesitant to say you know we we shouldn't be working with these people we should only be conducting assessments when they're 100 thorough because circling back to what was said um earlier I believe that it's just important to give feedback whenever you're ready for feedback right you may be conducting a bunch of experiments you may be trying out a whole bunch of features you don't necessarily know what's going to land in production but

at some point you decide to take a snapshot of your development and reach out to one of us and that's absolutely fine right but this is again the disconnection in expectations right what does what do users expect from audit reports what do protocol teams expect from audit reports and this is a fight I've been fighting for way too long now where you know d-gen's out there would take this guy's reports and say yep cool sounds good to me I've only seen you know a couple of lows couple of informationals there you go I can dump a whole bunch of money into this stuff right well have you even looked at the executive summary have you even looked

at what you know trail of bits or Sigma prime or any other firm has actually looked at so it's a bit frustrating sometimes and you know some of that narrative has been relayed by you know some some people here and it's every time I see that on Twitter I'm just it's not helping it's not helping but all right I'll stop bitching for a bit um I'll answer the question uh what is an audit for me it's a checklist so I think this part like we've talked about it so many times we don't do audit that's the first thing maybe D5 safety does audits we we don't we never did second I don't think bug bounties are our audits at all and and I'm glad that we go that we're calling it in the panel

conventional versus Community Audits and nomenclature kind of doesn't matter because I can call a table whatever I want right like if I have a name for it and we all agree that that's what we call it that's fine but they are not audits in the or or Security reviews they they are a completely different thing I'm not saying that they don't have a place in the in the stack of security procedures that you should do to your project but they're not audits the same way that uh Community audits are not audits so code arena is a bunch of people doing solo reviews that are then aggregated which is not the same thing as a team doing a Security review so while we call them from now on kind of matters I think we should all have a

like a group discussion the same way that we had for like um not calling these things Audits and their security reviews we should probably have the same conversation going forward for all of these things and uh but I guess like Sherlock's still winning because you know in different places we're winning yeah thank you yeah for spear but our approach has been to combine the best elements from all of these people we bring like some of the best uh independent talent to then collaborate um on Security reviews um we try to incentivize them and we have a more of a Marketplace model where

you know we just take a 20 fee um on top and some of the best researchers enjoy working with us because you know we are more fairer we try to like our collaborative approach like ends up like finding uh deep issues even and out of outer scope dependencies um yeah and and I do disagree with one thing by um the Sherlock which is I think while I agree that the end goal is to minimize the number of user losses you can't really get there without focusing early on on like core architectural issues we do also focus on that I think the safer the safest Protocols are the ones that are like very well designed

um so yeah we need to cover that as well yeah it looks like others want to respond um I I want to go back to what my colleague from from Sherlock was saying because I kind of disagree um with a statement that what only matters are the bugs finding the bug is obviously really really important it's you know like you want to prevent loss of funds but that's not sufficient what's going to happen like in practice here developers are going to continue to develop the software they're going to make like a new version they're going to iterate over it and if you want to kind of you know increase your security if you want to be in a better posture you need to understand how to build better

software and to understand how to build better software is not just by getting a list of finding because a list of finding is not going to be that informative in the long term that's why having all this information I will all this feedback about design about component about like structure test is really important in the long term finding the bugs are only going to help you in the short term but not in the next iterative in the iteration of your protocol not like in the in the long term thinking and if we want to kind of win as a security Community it's really important to focus on on you know all the suspect can I have a quick uh yeah so I actually yeah it's still disagree I think that

um like sock was saying we want to make audits the last line of defense the audit is really expensive and it's something that you're trying to do at a point in time so that you can get your code safely to mainnet and helping a protocol team during its development you know helping with the architectural design that kind of thing that's really great that's super important but that's called Consulting in my opinion and the audit is a specific thing to find the problems in the code base with the intention of having that code base go to mainnet probably soon you can ask just like just one question because I really don't know Jack what what how does Sherlock work because I really don't know this is a genuine

question like do you like if I'm a customer and I come to you what's the order of things that happens like what's the user flow do you um do you do the security review first and then you put up my code for a code contest all inside Sherlock are they different services do you do them always both for a certain client yeah so the right time to come to Sherlock and that we kind of recommend is is having it be your your last audit um and that's because we find you know some other audits are more on the Consulting side or whatever and maybe the code base isn't quite ready and Sherlock is really like the most intense audit I think you can get and the findings the amount of findings that come out of it are sometimes unmanageable for protocol teams

um so we think of the audit as that last piece right before you're going to mainnet and a lot of protocol teams don't actually have a good experience because they think that it's maybe something else and they aren't ready for all of the you know findings that come out of they weren't ready for the final exam basically um and so Sherlock kind of thinks itself as the final exam before you go to mainnet before we add the smart contract coverage on mainnet and to answer your question um the contest piece happens alongside the kind of reserved auditor piece so it was happening at the exact same time and the reserve you know we could call them more conventional auditing model with

those reserved Auditors are actually competing against the contest participants so both sides are kind of extra incentivized in that in that sense and it always happens you don't sell the services separately um we do not we have not sold the services separately as far as I know yeah so I think this is great I think we may be too deep into the space so we're probably making assumptions that people know how Sherlock works or how C4 works so that that was sort of the point of the first question maybe you know maybe starting with stock can you take maybe 30 seconds and explain how C4 works it used to work in a you know in a certain way maybe two

years ago but it has evolved right there are a whole bunch of services that you guys offer and same with Sherlock and we do by the way if you didn't realize so we do have the convention quote unquote the conventional bucket would be trailer beds Sigma Prime and diligence because they have been around for a while doing different things um and the the community one is again sort of a loaded term there is a contest base which is Sherlock and code Arena but then we also we have spear bit which is not entirely well it's a hybrid it's a hybrid model so we you guys understand this but maybe not all of us can you take 30 seconds to explain that please sure so code arena is uh a competitive audit which means that people show up in

order to review as you know as much uh you know to review the code in the time that they're provided and find as many bugs as they can the compensation for those bugs works on a Model that was designed by Scott Lewis and Zach Cole um and the model is essentially that the more rare the bug is the more severe the bug is the higher the payout and the bugs are judged by independent uh experts many of these originally were people who we sourced throughout the community Nick Johnson was our second judge after Zach Cole and had a number of folks just kind of come through on recommendation but

today most of those are people who have competed in code Arena and have moved their way into the ranks of of being a judge the thing that we have added over time is uh We've in the course of in the course code Arena one of the things that I think is pretty essential as a piece that Karina contributes is I think until uh really until kotorina came around one there wasn't really a good way for people to identify who were good solo Auditors and what the talent pool looked like in general there were people who out there were out there doing solo audit audits but there weren't necessarily people who could establish uh you know their talent

in that in that same way um you know the the other piece that I think is a key contribution is the fact that you have coding has really driven down the price of uh well-known bugs and we see that all the time in competition where people who compete and have read all of the reports because one of the things that we also did from the very beginning was we put all the reports public we put all the code public so that people who wanted to learn could go back and compare their results with uh the others who they competed against but as a result of that people studied those and things that six months ago would have

been worth uh you know ten thousand dollars or a thousand dollars uh that you know there are times when those bugs are only worth a dollar uh and the first time I saw that I kind of freaked out about it frankly and uh I was having a conversation with Scott Lewis and he's like no that's the algorithm working like we want to pay as little as we can for bugs that are well known when people have not added any additional marginal value in finding those bugs that's a success it means that we've actually educated more of the market on how to identify those bugs and then we're putting more energy into finding the harder more interesting ones and I think that is something that coderina in general has really helped to Pioneer for the space just in that openness in that process of learning

um and that's something that we're really focused on the other piece that that is a part of the process that exists today as a result of that is we started doing bot races where people who are building uh um automated tools in order to identify findings are able to compete prior to the audit begins uh prior to the beginning on it so that way we sort of get those things out of the way and we have some really great Bots that have uh actually started to identify more and more findings in the course of the um in the in the course of of competing but the nice thing about that is then every single auditor who's competing gets access to that report in advance of competing and those are all identified

as known issues right so we're we're now not seeing uh we don't have to have everybody submit the same bug you know a hundred times in order to reduce the value of it to its appropriate amount because many times if they're that findable they're that they're usually more identifiable by uh bot so you know a big part of what we're trying to do and is really identify talent and help talent to show up in all the various areas for that same reason we recently added analyzes which are intended to bring out the expertise of individuals who have more of a consultative view and are able to be when booking as solo are able to be uh taken in by protocols as a security advisor and that's a big part

of where we see and why we are investing in uh really the profiles that we've built recently on the site and and enabling people to book solo engagements through the site because we do want to get people earlier in the process from an expertise perspective so there's a lot of elements to it but there's some anyone else wants to add to that um yeah I guess I'll go next since our approach is uh similar to code Arena so the way to think about a audit contest is a protocol team has some code and they say here's a thousand here's a hundred thousand dollars and you can think of it sort of like anyone in the world can come get this hundred thousand

dollars by finding bugs in a code base um and of course multiple people will show up to try to do that so then you want to split that hundred thousand dollars between different people and the way it gets split is by who finds the most severe bugs who finds the most bugs and how many other people found those bugs how rare they are and that's pretty much you know there's an algorithm that kind of helps determine who gets paid what um out of basically anyone in the world who can come and try to find those bugs and you know we see code arenacies 200 300 400 people compete in contests uh from all around the world and yeah you

just find a lot of crazy things when that happens you find a lot of bugs that might otherwise get missed when that happens um and Sherlock also layers on a more traditional approach as well where we reserve one or two top Auditors who we know are very good and have proved themselves over time they're at the top of Sherlock's leaderboard and we make sure that they're going to show up and also participate in that contest and if they don't do well they'll fall down the leaderboard someone will take their spots they're highly incentivized but we just want to make sure that there's you know a high level of quality in every audit like a default level of quality in every audit so we kind of combine that traditional piece as well they help with the Fix reviews some of these other things that you might expect in an in an audit and

that's how Sherlock works just with those two so we have a foot kind of in both worlds on this panel how did you want to add yeah I'm sure our approach is like three different things um one we are like highly collaborative and the second one is we want to bring really the best talent to look into your code base and the third thing um um of course is we believe in like multiple eyes uh being better we we want like you know four to five people looking into the same chord uh multiple times so that we can be sure that in a lot of the vulnerabilities are caught um do you behave as a Marketplace Harry can

I choose who I who I get yeah you can yeah okay um I guess for the three traditional um players out there um uh correct me if I'm wrong feel free to chime in if I'm not representing our side of the industry well but uh you reach out to trail of bits intense diligence Sigma Prime and you kick off a sales process right um if you reach out to Sigma Prime you get to talk to me usually and I go through a scoping exercise right you share with me your code base and I start going through every single contract in the case of you know solidity evm stuff that you would like to see reviewed and we essentially estimate how much time it

would take for a team to perform a review to provide a decent level of security Assurance so for cgp we always include testing in our approach that's perhaps that might be a little differentiator between our other friends but um so for each component you'll have person days for a manual review and person days for testing and testing can be done in different Frameworks Foundry Forge or brownie or um you know we don't really dabble in JavaScript but yeah um yeah and then we give you a proposal take a look at the proposal give you the

timeline obviously the earliest we can get the job done and I guess an interesting part of the job is for us to be able to spot within our team the right members for that particular Target it's very important right I've got team members who have reviewed perhaps as of today for liquid stacking derivatives right they're across you know all of them most of them if not all of them so when I get a request from a client who's building a liquid staking protocol I'm gonna obviously do my best to assign that protocol to the testers that have that level of experience we can Circle back later on on how we train people how we

make sure that the knowledge is spread across the team but yeah that's essentially it you know we are a services company so if you're if you're happy with that we uh we then get started and kick things off and produce produce a report and a test Suite that you essentially uh yeah is yours you can do with it as you see fit yeah like I agree with everything I just want to take it back to the stack just if we're trying to get people to understand I'd say that like I and this is my personal preference I I heard different things today but like I think that you have the manual review on top then you have the code card that's Below in the bug body like the only Undisputed

100 market share that we have here today is Mitchell like I basically refer everybody to immunify right like bug bodies are completely Undisputed then we might have I don't know like a choice between maybe Sherlock and um in the code Arena or maybe not because Sherlock is integrated so it's monolithic and then you have us and so if any of these entry points I guess like the the five entry points over here because like spear bit is also an entry point it's probably the first people that you would contact right I wouldn't go talk to Mitchell first that's probably the end game so let's let's put Mitchell on the spot Mitchell if you had a protocol on your bug Bounty who which of these vendors would you recommend

well I'm in a rough spot today um it could be all six you know so so actually this is a question that we have thought a lot about we get recommendations and I was you know pondering to myself as I was listening like should I give the audience this this particular model that we've developed for looking at the situation which is you know while these things all seem like oh you know here are Audits and you know there's all these different types and flavors to choose from who's the best the way we look at it is that there are very specific things to prioritize the different parts of the stack so to be quite Frank you know I would not I would avoid I would you know if I was you know CSO or equivalent security lead at a project and they were asking me how do we want to allocate

spend here we need to build out the ideal security setup you know what are you going to do Mitchell well I'm gonna and they're going to say hey do you want to work with with with C4 they got like bot races and so many people I'd be like no absolutely not that's a horrible decision uh not because it's not a very useful service in fact it's an excellent service at a certain stage of the life cycle but that's not my problem right as I finish my internal review process so after I've made as many Investments I could on the internal side so that's hiring my security Engineers that's investing in tooling that's documenting the hell out of our stuff that's building you know a good secure code development process as I can that's you know threat modeling after I've done all those Investments which are not you know

primarily monetary but time then I want to bring my outputs the right the outputs of my work to whatever degree is possible to the finest possible mind I can imagine okay and it needs to be and I need to pair that with my you know my finest possible mind and I need them both to attack what we have and see where it lands and to be quite Frank while I am like truly the most Pro crowdsourcing person here maybe not truly but I'm trying to be uh I would absolutely go to one of the private firms first so I would go to a consensus diligence I would go to a trailer bits I would go to Sigma Prime I might also go to a spear bit but I think there are some questions you know that I'm trying to answer about their service

before I can have that conviction right there's something excellent about them but it's not clear that they bring the level of comprehensive expertise that I would want at that initial engagement because that's where I'm having these architecture discussions right that's where I'm throwing them against my my engineering team and saying here's very expensive people we pay to tell you that you did everything wrong and we're going to have to go back and fix it all thanks man it's a good thing we got this now and I and I'm not really going to I can only get that from an extremely you know impart someone whose brand is both expertise impartiality and you know awareness deep expertise up and down the stack broad it's got to be everything they have to have the whole world view they've got to be smarter than me and the the firms that are most likely

to have that Talent quite frankly are the traditional audit firms because they can best allocate resources to best attract that talent and structure their whole brand Mission and and the life of their their business around that that's something that the the crowdsource businesses will never be able to do right because we we have to select for a different set of conditions and priors in order to build a compelling product so I say that right would definitely pick one of those now I don't want to name names specifically because then the other ones will beat me up uh but I want to make a note about spear bit in that so what's my doubt with Spirit like Spirit has this interesting model where they have effectively been capturing the top end of the the audit contest kind of population and many of

the Auditors and it's great right they've they've taken kind of that that model in a in a Cost Plus fashion and just drove in it as far as they can they can they can pay big wages to people while giving them a lot of freedom and giving them and by freedom I mean you know they can do whatever they want they're not necessarily on contract to go and do the most interesting work I mean it's a it's it's a great deal but is that right in my first initial engagement when I have to say architectural stage My First Security review and hopefully the most important are those people going to have enough sophistication are they going to be process driven enough are they going to have discipline right and I need to I need discipline really bad at this first engagement I don't know if that's true

and that's my doubt about whether Spirit can service that if they do that's would be super exciting but that's an open question I need discipline I need process Excellence you know beyond what I have and I'm you know a guy who invests in best practices on exactly that looking past that now I have new needs for Audits and and for context to unify with any of the stuff that we do on chain which we do we always do multiple audits okay now my priorities are beginning to shift so I may want to go with a younger firm that is you know more of an attacker's mindset from there I can tolerate less discipline if I know that they're really creative and so you know a great example of that is hacker founded firms okay hacker founded firms are crazy and that's why they can make great attackers

but not alone because they can't be relied upon to get broad coverage of the potential issues only towards the end of this stage right you can see I might pick from a few different options only towards the end of the stage am I going to look at a a audit contest like element where I'm really just you know throwing eyeballs hopefully as high skilled as possible but I'm throwing eyeballs at my code base to see if they can find what's left but knowing that it's like I can't know I can't have any expectations that they're going to find everything I can't have any expectations that they have deep expertise you know it's a gig economy kind of stuff it's it's open doors and hopefully it's the best type of it but the reality is there are limited quality controls on it and so it's at that time

that I'm like more eyeballs the better right and and I would do you know I'm very interested in that I think it's hopefully will eventually become a normal practice in our industry to get these multiple rounds of reviews this way so that you get the benefit of Top Class Minds on every major deployment and you get the benefits of you know the best creativity you can muster and you get the benefits of just lots and lots of eyeballs covering best practices that would be phenomenal but it's very expensive all right well I want to just answer the this favorite question there um we do provide like security across the entire stack um all the way from like you know early engineering uh we have a service called vcso

um you can hire like one of our you know leaning secure researcher to like you know work couple of hours a week uh over your entire entire dialogue development cycle to have like a gut check and give you like advice on on how to like architect the protocol in fact many times we have completely like re-architected uh uh work with clients to completely like re-architect the protocol to make it simpler and much more secure by doing this um yeah so if you are building protocols today like talk with us right away we can you know help you build much better contracts so it can be like you know secure by Design almost hey so can we have like content and case studies on there sorry can you repeat can we have

content and case studies on exactly that question for sure for sure and I would love to to see that in public right because there's an open question like you know in a lot of ways you guys are cherry picking some Top Class talent that that could work out well this way but discipline discipline there's no replacement for that can you guarantee that rigor can you guarantee that Excellence you know beyond just someone can sure they can find lots of vulnerabilities I can find you know lots of hackers like that that would not make good Auditors at all and certainly couldn't do what I'm describing right the the skill sets are not co-extensive with one another so I would love to see case studies uh White on the internet for anybody to see about you how you went and re-architected something that would just be marvelous yeah for sure

and very valuable very informative yeah I mean we have like architecture protocols to like you know just a single contracts um they were like pretty new to solidity and they were just you know architect or like contracts as if it was a web to app and this is the wrong mindset to like build uh things in in solidity um aren't you worried about the churn of talent like there's people just dropping out have you ever wished you could hire someone uh I actually think our approach decreases turn we talk with a lot of secure researchers that work in traditional Artic firms they're pretty much burned out working you know um four weeks out of a month uh 52 weeks perhaps

um we we only recommend people to work two to three weeks in a month a lot of our Auditors they enjoy like researching uh as an example like I tell this uh like hiccup is probably sitting here he spends like two weeks uh doing Audits and the other two weeks he likes to learn CK this is something we truly encourage our Auditors to work on and we think this decreases like so and we speak with so many people who are like completely burned out from working in a traditional firm and you know we like to avoid that yeah burnout is real burnout is real and it's to be taken very seriously um yeah we've had obviously occurrences of people burning up myself included and

it's very important to keep a finger on the pulse of every single team member you have on your team and the way we try to do it at cqp is to allocate a bit of time between each review obviously you know every time djans decide that it's a bull market you know our inboxes explode and some of our team members are sometimes a bit stretched but um yeah this is this is a major point and um we're obviously always looking for ways to do better um and you know hopefully you know I'm listening to my team and they're they're the you know the best people to give me feedback on how we run things but yeah I just wanted to say that it's everyone's responsibility as well to you know keep

keep an eye on the work-life balance uh it's it's super important um because if we all burn out then you know all of this is is for nothing so um you know check on yourselves check on your friends make sure that you know you prioritize your well-being above anything else because at the end of the day you know that's what really matters so and just to make sure I understand spare bit model um when they do like research and things like that are they paid uh yes okay our top researchers make 20K a week um on base uh some projects pay even more um so they're very well compensated sorry sorry yeah I don't think that's that's the question the question is when they're not doing engagements when

they're off for those two weeks are they paid uh no they're not but they are paid very well on the one the weeks they are working so they don't have to I think it's an important point because you know our trail of bits or engineer on average are Bill 50 of the time so over a year they are going to generate revenue for six months what they do outside of that they are going to be paid but they are going to work on public goods they are going to work on tours like you know Sleater cerns are going to work on guidelines are going to work on presentation in addition to that a lot of the workers are going to do that they are going to Shadow at it this means that they are going to participate in an audit which is outside of their way of expertise but

they are going to learn and they are not going to be built to the client uh this gives them like you know a huge kind of growth opportunity where for example you might be an experienced heredity you might be really really good at solidity but you have no experience in cairoa you have no experience in Western you want to learn uh by you know shadowing like an audit with expire invest or in car or whatever you are going to learn a lot so it's definitely like some kind of benefit that people might not realize but you are going to get by you know working on this type of company um yeah same thing intelligence but but I have so many questions for you and like the churn one was not exactly what you answered so what do you do if someone doesn't pay you enough to pay

the lead security auditor do you reject the work yes yeah you just say no yeah we say no to a lot of clients we we like I said I know when we see a client we do like we give our genuine feedback on what should be done um we tell them this is already you know you have to like re-architect the whole thing and if they say no we don't really work with them uh okay and well you also don't care about deal flow because like you only pay people when they work so you don't have to guarantee to you don't give any guarantees to people and when I talked about churn I'm I was more interested in hearing how what's the bounce back right like do people keep coming to Superior oh yeah of course once once they get a taste of spirit there is no going back

[Applause] good good what's up yep well I just wanted to jump in on uh something so I'm actually wearing a shirt from him I actually come from a traditional audit background there's a firm that uh I ran with a partner who's here uh years ago and working with traditional Auditors one of the things that I know is the case is you know you talk about the question of discipline you talk about the question of what it is to show up and work and the fact is that traditional Auditors working in a traditional firm in a traditional process have off days and they do honestly because of just the nature of uh our the nature of our uh

our capitalist Society they do push themselves well past where their ability to perform uh May necessarily be um appropriate so in terms of also by which I mean they get burned out yeah yeah right obviously yeah yeah so if you talk about burnout yeah yeah exactly right so the thing that the thing that I see and absolutely love about the competitive model is the fact that there's a great quote from a guy named Peter block and community means always knowing that someone else is going to be there and that's one of the things that I think is absolutely fundamental to the way that conorino works in that it's not all on those individuals the the weight on individual Auditors and I'm talking

about in web 2 when we were dealing with far less dire circumstances the weight on those individual Auditors was so significant I know many people who burned out doing that after a few years just in web 2. in web3 it's even more even more intense and frankly most of the people that I know who work in and as Auditors work in the field and as Auditors they're conscientious people who actually care a lot about what they're doing and why and so when I look at the question part of the reason that I love the competitive model is that it removes the level of intensity and pressure on those individuals for things that really do [ __ ] matter let me quickly take back the mic for a

second so I think we have less than 10 minutes left really yeah we're just getting started but we have only five minutes maybe so I think we've so can we just React to what what this is very important five seconds I was gonna say that like here I think that it's that is a fair that is a fair commentary but I think you're like we're thinking about like consultancy groups that are huge right like the typically that happens in euis mckinsey's we were just talking about this the other day we work more like Boutique shops right like um so there's a way way way bigger amount of care for the people that work in a 20 to 100 person team than there is

in like a huge Behemoth of of a company right and and the the flip side to what you were saying about the community thing and I do agree with you look the community is always there but we are a group of vetted people not everybody goes into consensus villages we take huge pride in the people that are in there right like where is Carolina like Mitchell was saying like it's they're always there it's just different it's just different and I agree with you some time people's burn up this room is full of people who are absolutely amazing who got their start at code Arena and they're disciplined and they're amazing incredible Auditors I just want to react to one thing you said I I'm a big fan of code Areno um um you know the traditional side often recommend my clients to go and run contests one thing

that people should keep in mind is that you're not hiring when you go to trail of bits diligence or Sigma Prime you're not hiring a specific auditor you're leveraging the entire team and that's extremely important the quality assurance that goes through those engagements is Second To None like I've I've come from the traditional web security background as well and the level of QA that was done then was nothing compared to the stuff that we do and I know for a fact a couple of other people here do the same so I I have to you have to go to the next question I know we can keep reacting and overreacting maybe I was gonna fire a torpedo but let's let's continue that so I I think by now we have level set context set people

understand that the different models all of us including so if let's sort of switch right if there were one thing for the conventional guys that you like about the the community approaches and vice versa what would that be I mean you said you love C4 why is that last year Mitch said that last time we were the bottleneck the Auditors were the bottleneck to go to market like it was ridiculous right you couldn't get anything done within like six months nine months and I'm really thankful for everyone here for all the Alternatives that we can provide to the community and I think it's a great thing you don't have to wait six months three months to

get feedback you can get feedback within days weeks sometimes right and and this to me is a really really good good thing this yeah timeliness pricing there's so many things yeah Jocelyn yeah I think it's a greater so like learning opportunities for a lot of you know people that are newcomer and and wants to kind of try their skills um but there is a bet um I do think that people that are starting in the security space and go through kodawana Sherlock or other might focus on solidity security but might lack some Awareness on the spectrum of security and you know I might make a couple of friends here but we are a security event

if you work around like with whom you are going to see a lot of independent researcher that look at their laptop they don't have you know screen protector they look at sensitive information they are like 600 people at this event like most likely you are going to have uh you know government agency malicious sector working around and the fact that some of the basic operational security are not followed by a lot of people are disturbance it's kind of a signal to me that there is some lack of you know training in terms of security that's when you are in a competitive environment you might not get easily yeah I love them for all the reasons

that I that I wouldn't pick diligence right like I said like the same thing that is a pro is also a con in in my view like you'll sometimes will have scheduling issues like sometimes we we won't be able to attribute the individual person that you want because like that's just not the way but the way we work and if that's your need if that's your specific need maybe just need that one person that you really like maybe then you should go to Spirit but and then you should still do coloring and then you should still do bug bodies they're completely different so all of the things that you wouldn't use us for that's why you should what about the other side I just want to like uh follow up to Jocelyn I think training is like very important in the space are not just for like auditing

films but across the ecosystem um in fact to be a bit hosted the very first Foundry seminar um shout out to Nathan and Brock for helping out with that uh with that I think this seminar alone like to our seminar on Border like you know thousands of security searches and and projects like um building uh stuff and I firmly believe like today it's uh easier than I've worked to like build secure contracts um in fact it's easier to write a fast test in solidity today than in any other like programming languages ever um it just it's just a matter of like adding an extra argument to your functions it's easier to do for modification um in in solidity than in any other language today I think the tooling is

you know tooling keeps improving and you know we believe like we invest a lot in um education there's like weekly talks you can go to youtube.com stay a bit to see like what's what's happening um so yeah that's it so Sherlock and C4 is a thing from the conventional space that you would like yeah I mean I think on the tooling side a lot of these firms up here have done incredible incredible things for the entire space in an open source way in a way that just benefits everybody and that's actually like if you look at what they've all built it's really really impressive um and I think another thing that they do well is um kind of going back to Mitchell's Point um I think they're really good at the

Consulting side you know when you are maybe not ready yet for that final exam they can help you get prepared and they can help you you know design some things and have conversations about different ways of structuring something um that I think are really really beneficial and you know that's not a knock like you want to have a you know a professor or like a midterm or some quizzes before you get to the final exam and then I think you know the Sherlock maybe the code Arena are sort of better structured to be that final exam where you have hundreds of people looking for bugs in the code um you know Sherlock reserves two top hackers uh in the world basically to look at the code alongside those other hundred people

um and so I think you know both have a really important place in the space if you can afford multiple audits hold on you think Auditors are hackers I don't know about that man oh the auditor hacker yeah debate I I don't know about that I don't know about just saying but do I have to answer because I like everybody you know you know so the long and short of it is I am I mean I always loved working with the Auditors first they drove me nuts in the old days because I was like why are we doing this this way and then I then I you know experienced some of those problems like I'm so happy you guys are around and then with these these new crowdsource security products and by the

way we're really only scratching the surface there's new techniques being developed there's new companies in the making there's there's new like what we're starting to see is the formation of a sub stack right within that territory just as there was a stack within audits depending on the type of firm and the type of value proposition they focused on we're starting to see a sub stack emerge on the competitive audit side and that's I mean I'm I'm over the moon for that in terms of the value it's going to provide uh to customers in terms of the value it's going to provide to users in terms of the opportunities it's going to create for everybody here it it's just right there but we should remember right we are how do I say this in the best way it's not enough

it doesn't matter we are dealing with the most hostile you know possible environment I have criticisms for the approach of every single model here some of which would be pretty biting I think and yet I am so excited for them to grow and expand and in fact we need more of them because you know the dream is to put trillions of dollars through this infrastructure and to hit those bars you need to max out right the quality of security product that you deliver on every single measure I think some people get that the layer zero is describing their spend amounts and I was like yeah that sounds about right you know they understand they understand the seriousness of this game so I am incredibly excited you know despite all the limitations all the arguments all the you know friendly or unfriendly competition I hope all these

things proliferate and we need more right calling out to the audience here if you have an idea if you want to create something if you want to contribute something we do not have enough as it is we need more Innovation we need more scale we need more participation and believe me certainly at immunify but I also believe all my compatriots here would love to work with you and help you push these things Farther Along fantastic it sounds like we're just getting started but we're out of time so uh we should we should give at least a minute we have a break coming up so um but maybe we can give some time for the audience I'm sure they have a lot of burning questions foreign might be a little bit too controversial uh to Spirit uh so first is uh do you

have non-competes with your Auditors uh how you protect that people cannot go directly to Auditors this is first and second how much fees do you keep to your company so that you run your operations from the uh is if no if it's a secret please don't tell it but it's very interesting to know we are extremely open very few Secrets here um we don't have any uncompetes uh we we want to incentivize the best people to work with us and we don't want to like tie them down um that's fundamental to our business model and our fees is like one of the lowest here we just charge 20 um the rest of it goes to the Auditors

and that's how we incentivize some of the best people to come work for us do you have time for one more no okay yes just a quick question and maybe a quicker response okay then the quick one is going to be if you don't have non-competes do you have ndas sorry could you could yeah with the clients and the auditors but not with the Auditors yeah both of them we do have NDS yeah all right thank you so you'll find all these people uh for the rest of the evening hopefully [Applause]