Protocols Discussion, Session 2

Transcript

when we're doing an audit and we find a bug there's usually significant time put into researching validating peer reviewing and then putting it into a report that could be publicly available and so we dot our eyes and cross our T's is there a way you try to like kind of get ahead of that by saying oh if you found this bug let us know we'll validate if it's a real one or when we know about and then you know Skip it don't do as much due diligence and then move to basically have more opportunities to find the real bugs you would want to have them focusing on uh oh that's quite a long one uh so far we can you hear me Okay cool so far uh we don't have processes in place for this uh I think this is something where

um a healthy dialogue with our Auditors is beneficial in many respects um so maintaining either like a slack or a telegram Channel where you can ask quick questions and and get these kinds of responses uh I would say typically where we would reveal our are withheld vulnerabilities is during the preliminary audit report before it goes to to final and then at that point we would recommend that in the final report they would say um you know this bug that we found was also injected okay there is another question there and then there okay there are a few I will try to yeah okay you can go forward uh yeah can you

hear me oh yeah so my question also for yushin of course about the assumptions behind like the estimation model that you use so I noticed that you yourself use the estimation model like for estimating bugs and so how actually do you apply it do you have like a more complex model which takes into account the assumptions or like do you just naively apply and then softly weigh things based on your own knowledge of how things were implemented um yeah definitely much more of the latter I would say we consider it more of um like we're doing a Fermi estimate so you take a bunch of different numbers and try to get a rough intuition for for what's going on the magnitude of of what you know the number of security issues

you have or or whatever foreign over here right um yeah so I think you're getting a lot of questions because you had really interesting work um I really like your guys's kind of data driven approach to this you know how effective are these security practices but I guess for the injected bugs you guys are looking at how do you know that these are reasonable bugs like like for example if you're you know injecting things that are you know small little typos or something like that like a a normal developer probably has some sort of linter turned on that's gonna you know they're not gonna be injecting these types of bugs right

so how do you know that the bugs you're injecting are ones that actually would be happening in practice or what's the ratio of bugs that you inject versus bugs that you find and and just don't report basically so I'm kind of curious about your guys's decision making there yeah thanks for the question um I would say that um the the bugs that we're injecting are pretty much entirely a function of what's water Engineers are able to notice in normal code review um so there are these these massive massive gaps that are very very likely to happen um where if uh you know if the particular viewer of the day isn't isn't well versed in things like um

re-entrancy then they may miss that's uh there's a re-entrancy button uh being committed to the to the code base uh and this is where I think a lot of the other Frameworks a lot of the other um you know what uh yarn was describing uh in the previous talk where you have scoring and risk modeling um and an idea of how much scrutiny you need for for what kinds of changes really comes into play yeah I'm from auditing Forum first I want to say a question for you James that I love the approach I really love the approach to injecting the bugs my question is about the uh how to estimate the total one because uh we we know that

not all the bugs are equal and in fact most bugs are different so how do you include in this factor that into your evaluation to say some of the bugs are really deep or some of the bug might have a higher impact I just curious about you know whether you have any plan to take into those into account right and so this is um this is actually one of the things where we'll maybe we should admit that uh you know this uh this was a clever idea at first and we are slowly trying to figure out how to answer these questions more and more so to expand on it a bit more um one thing that that becomes an issue is you might say

uh civil attacks on bugs where you take one bug that affects a bunch of contracts [Music] one bug that affects a lot of contracts and is essentially the same and then there's now an incentive to say oh this is like seven different bugs so that the count goes up for for this particular approach um and another issue is as I think as alluded to before you can miss entire categories entire classes um and taking like the ecology kind of analogy it's if uh you've got like a lake full of fish and then you didn't realize that it's connected by a stream to an entire like gigantic Lake

um we we don't have the answers to this I think this is something that needs to be developed with the community there is yeah um hi this is this is Stefan from solidified Oak security so if we we actually your auditor and um I'm only admitting this so I'm only admitting this publicly because you said we did okay which is which is good I suppose um but my concern is also about the assumptions you draw from these injective injected bugs and my question is how do you decide how to you know which bug to live in you said before you these are bugs that you catch internally

and you leave them in which is good because it reduces sort of you know sensible bugs not not artificial things that are that just waste all of this time so we are in favor of this but you know how do you decide what to live in because the concern is you are only finding if you if you do an internal audit you're only living in parks that are easily found in an audit if you're living in things that are found in unit tests you're sort of living in tests but then you have to remove the test and the auditor says oh you have a low test coverage so so how do you deal with deciding what to live in and and what is a good Park to inject uh yeah thanks for the question and thanks again for for being our auditor uh let's get to it's good to meet you in

person um I would say um [Music] there definitely is this bias there is a bias toward finding things that can be easily fixed with a single line as opposed to things that would um you know involve a massive re-architecture because you can't just fix it with a single line uh and this has actually come into play in one of our our more recent releases which is that um you know one one of our team members uh injected a bug and then later on when we

tried to fix it we realized that this would involve entirely refactoring everything it's not just a one-line change um and so there would be a bias against injecting bugs that are like this because if it's something that would require a full you know architecture refactor um you like tried to catch it earlier you you try not to make this like a late stage gigantic change um and that that's a problem um uh but I guess to flip it around I would say um our best use for this has been as a um has a as we're developing this I think um early on we thought it would be a good kind of like

a broad scale Benchmark for all like severity levels and difficulty levels and finding it but now we're moving towards using this more as um a medium hurdle to clear to say um you know just a binary answer of um are they doing a decent amount of basic diligence rather than finding like the the most complex and most esoteric kind of bugs [Music] yeah um hi uh Jonathan from chain analysis a question for for Nicholas I uh sort of really interested in sort of the empirical side of governance and like how much how big are these problems

of um you know DNS and like how do we tracked back to any actual um you know loss of funds or anything on the empirical side has anyone done that type of analysis um can you hear me we haven't been subject to this sort of incidents uh thankfully because we would have been ill prepared and they would have been catastrophic uh I don't know after of my head of any any other protocol where like this sort of uh you know malicious where the attack was for making them verify incorrect information uh that I was talking about uh

but after you know looking at what our interior system looks like and you know given the different safeguards that exist in place uh and how signatures happen and who designers are and how voting happens and so on I think at least in our case and cases they're not like us it is definitely the the Avenue with the most risk in terms of new things being introduced and those things being wrong yeah but I don't know of any actual instance that has occurred this would change uh [Music] um so one of the things that I had a question on was understanding that was

right so if you're going to have someone who proves that it's going to be internal to you or is it going to be somebody that's attacked a test that well that's trying to transact sorry I couldn't hear that that part [Music] so my question is is whenever you're doing the transaction delay right so is that approval within the chain itself or is that approval coming from the actual owner of the currency oh no no so if I get your question right like the actual delay happens on chain correct uh the actions exist the action IDs exist on chain okay under on-chain

requirements about what the minimum delay is for scheduling different different actions okay so it should be impossible uh varying bucks that an action is executed with a shorter delay and when the action is scheduled it is committed with all of the information uh what contract is going to be called what is the data if there's if going to be sent how much it is there and all of that is locked in and cannot be changed in any way except in canceling uh but that all of that happened so I didn't talk about this but like that that being on chain uh we hope it'll be that much harder to make it so say you're fake you know no providers and you provide fake if blockchain data right as opposed to you know a DNS of

some random server on on the on a cloud provider and how long is that delay uh it depends uh we have a minimum delay for anything that is non-trivial of five days which we think is a sensible duration for things to happen someone to have time to look at them and if there's an issue discuss that escalate and then gather signers uh we expect for each of those steps to take at least one day in an optimistic scenario so we make it so no actions are that are not trivial can happen within less than five days um it's quite tricky sometimes designing those systems because some actions that might not seem very important are

actually important for example increasing the delay of an action is also a critical action because I increased it to two years then the action is now unusable right uh but yeah we have that minimum okay thank you okay we have the last question and then we are done with session two hello so my question is to storm Xerox about risk assessment and scoring what was the process like for coming up with the scoring guidelines for each component and how have you refined the process over time and ensure that it is accurate and useful yeah um I I guess uh like I mentioned we had several like uh like a season of

incidents where we were just putting out fires and and trying to trying to figure out what went wrong and trying to start with postmortems and trying to analyze and we particularly uh we quickly like converge into this Dimensions okay things that happen before the code goes to production things so we sort of those checklists of things that are sort of Readiness for deployment for a production contract we made those into dimensions and then you can score them and like be flexible about them so it's not not like uh to because you need to like like I mentioned that we cannot afford enough a

heavy upfront design uh and usually all this thing formal verification or if things happen after the fact when when the code is in production so how do we score it so the things that we control are internal procedures uh and you see those as Dimensions whether a school review test coverage we put a score in the test coverage or it's it's that 80s that 90 that lowers your sport uh even if a strategy is uh wants to hey I want to allocate more fonts into this okay okay can you improve your test for or can you get somebody else to peer review it at this point and uh yeah those are we're still on ongoing adjustment I don't think it's perfect by any means but it so far it has been proven like we

saw the drawdown during was in several protocols during the whole Terra collapse and everything and uh we didn't lose the scent uh some of this this uh came out uh well enough for us uh but uh but for for sure we we can find better ways to improve it and like I mentioned hopefully this uh ideas can be generalized to other protocols I'm sorry side uh there's a sort of you know black humor phrase in the avionics industry that goes that the guidelines have been written in blood

you know it's not that someone came up with those ideas yes they're the result of seeing issues and then going how did this happen how to make sure it does never happen again uh you're talking about biases and the engineers vices we've found that in the box that we that other Auditors found or doesn't show up in the back Bounty or whatever there's often a trend it's like oh yeah of course there's an issue in that part of the system because we never really think much about that part of the system so let's make sure that no longer happens right like let's make sure these types of issues are now more thoroughly thought of from the get-go uh it's a lot of trial and error uh and I guess hopefully errors don't go very very bad but uh yeah you can ask a question but we

are gonna you're gonna get the answer of chain I think offline it's real quick uh the question is how often do you cycle through Auditors or do you use the same three otters like every time and the second question is do you use the same kind of bug program internally to try to incentivize Engineers to actually trick other Engineers to approve these bugs and then realize that they're just rubber stamping yeah uh yeah the answer is gonna like we're I'm sorry man that's not my fault okay uh that's it for session number two thank you everyone and see you after lunch [Music] [Applause]