From a Hackathon to your first Code Audit

Transcript

so hey hello i'm rob from pods i'm co-founder at pods finance we are an options protocol like we try to like be like easy to understand all those things like related to derivatives like we have complexity enough in this space like you need to know metamask blockchain gas and then derivatives like options futures forwards so we want to bring like the easiest experience for the users like to grasp all this complexity and the reason that i wanted to do this talk is because like we are a team that came out from hackathons we last year we had like this project oh my d5

on if denver last year by the way this is vitalik he's not much excited but we were like a lot it's okay and yeah so after that we decided to start pause like as a real company and we also participated in uh hack money it is another big hackathon i think they just had like the last version last month so we had like i think like i wish i had no a lot of things that i'm going to say today like because there's not you maybe you already know let's that's not a lot of content out there that like show you the stones the path of stones

that makes it easier like to understand what you need to do in order to have a cold out cold audit ready so when i started i thought that the process was like that okay i do the hackathon i have some code i do an audit maybe not and then okay i'm ready to production that all things in web 2 like seems to happen right while here we we are like different but the reality was something like this i code i test the code i do do some documentation maybe it's not enough i then i run the coverage my cover is like 30 i do a lot of stats again smart testing

and then i have to create like a test network oh but ave is a different network like avi's in colvin and my system is like i don't know he's hobston so i need to create like mock contracts oh chain links not already in my in my network oh oh maybe optimism they don't even have like the spot price of the asset with the oracle that i'm going to use so yeah the reality is a little bit complex so this is kind of the first version of the contracts that we had when we just after the hackathon like almost like four contracts and one like abstraction and after we ended up like

before the audit with kind of these architecture here so is like much more complex than we thought you started to realize that is missing a lot of stuff in order to have like a real system running so coming back to the audit like you already know they are like a lot like so expensive especially for a startup that you don't have like a lot of runway they are slow like i've been talking to developers here in the last days and they are scheduling like audits for december they don't even know like if they are going to have runway or if they are going to have like codes about it

but they are saving spots and after starting the process it's going to take you like at least one month to finish like the whole process like you send the code you like receive the first feedback you do like the fixes and they send again second feedback so it's going to take at least more one month there so you like requires a pre-work and they are not going to tell you that they are not going to tell you oh you are going to start an audit next month oh how is your documentation like do you have the ads already they are not going to tell you that but then after in the report there yes so that's the reason i'm here to

tell you some truths and to like tips in order to get there and like uh at least not feel that you paid like a lot of money and then you feel oh my god documentation we're doing like oh your documentation is wrong or it's a typo here like you're not paying this a lot of money to have like those kind of errors we are searching like for another type of errors here okay so if you like game analogies like real-time strategy for me audits they are similar to like channels the heavy army they are slow expensive and they are effective for like a very specific kind of errors that he wants to to search

so and usually they are for the late game so not in the beginning like i i just finished my hackathon and tomorrow i went an audit no they are late game so why do we need it like do we really need it for me the reason that we need audit is because we have like a different paradigm here like we are talking about web3 like you cannot like pull the plug into as at the last resort like okay i have a bug let's shut down fix deploy again like i don't have this option so and you cannot like probably if you

already work at like in the web 2 oh let's write directly to the database i have admin powers here so if you have an error you could go there in reverse here we don't have this option so for me that's the the reason that we need audits and for me that's one of the best phrases that i i've heard i in my first like this phrase came out appeared when i in my first conversation with one auditor that we had and he said rob buildings put that in your head like it's another mentality here is more similar to the hardware like it's more similar to like building a phone like you don't want to have bugs in your camera after like

you shipped directly to your clients so for me that was the hardest thing because i i was not use it to it like beautiful i i came from this mentality here build fast and break things uh but here is a you have to that it's since it's harder it's not it's similar to hardware you should try to build fast and don't be wrecked it's not easy we are still figuring out how do we do it and the most of the effort comes before the application not after the launch okay so how to make sure

that your money is going to be like worth it rework let's talk about like what you should have in order to have a good audit so again to talk about that let's understand security layers is again now our game analogy like you have the candles the red one the out is right there but what are the other soldiers here and what type of errors what type of like opponents in our case opponents they are the vulnerabilities the hacks so this is a list that i think

you should have before the audit at like the minimum i think static analysis is fuzzy testing like this is optional at least for the first audit if you have time if you have like more developers in your team that you have like time to learn and and implement implement it's okay but at least those four elements here i think they are necessary so the first one unit tests no secret here like the soldiers uh very easy to do i don't have much to say about it like the most important thing is you need to

test every execution tree that you have for those who don't know what is an execution tree like if you have one function two ifs then you have like two paths far past like one if like positive false so in order to test all those things i recommend to use waffle and they help you like mocking your contracts and like creating fixtures because like testing here is going to take you a lot of time each time that you change something and you need to run for example in our application we have like 25 contracts takes around 20 minutes every time that

we yeah yeah we have like 500 tests and it's taking like 20 minutes in our github actions to run the test every time so it's better to have like this in is in your head like do fixtures since the day one is going to save you some time uh yeah so unit tests is like specific to functions then you have like the integration tests is where you test the scenarios like in our case we do options so for example in order to exercise an option you need to mint first so here i'm not test in order to test only the exercise

function i need like to prepare the state of the the system to be ready for that uh some hints here i recommend using like main network or tenantly farc so then you can like already test like for example if i'm doing something related to wheeled farming rewards on navi then you fork and then you test directly this state of the blockchain and if your system is not self-contained keep that in mind that's not going to be easy to have like a test net fully testing so that's why i recommend the main network okay next one coverage coverage is not

like uh for me in my head is not like a soldier is more like you know like i don't know no but it's like like a thing that helps your soldier that make them stronger it's very easy uh to implement that's why i put the effort there and help measure and visualize and track like uh constantly what is the current state of your tests if you in in if you are especially if you are missing some of the execution trees that you didn't create a unit test for that uh going really fast here for me the easiest way to implement coverage

use hard hat like for four line of four codes here and you get ready the coverage solidity coverage is the one that i'm using there coverage coveralls they are free for public repositories and they have like those there you can see like they show for each uh for each file what is the current state and down here they show like in yellow what is the execution tree execution path that is missing so i think about like a good way to do it is like you do your tests like from your mind upon your feeling in the first place and then like one week or two weeks

after you install coveralls run the first time and then you are going to realize that you forget forget to test like some execution paths it's also good to keep the coverage because there's another sentence like there's another phrase that people say that reaching a high state of coverage is easy maintaining is hard because every time that you change something you need to add more tests so you can add to the github actions to test every time you can check inside our repo like in your workflow close you can copy that i think the hardest part is going to set your secrets on github token there

but then you're going to have like the process automatically like running the coverage from hard hat uploading the results to coveralls every time that someone do like a pull request so i recommend that and this number minimum of 90 percent came out from the auditors i i've heard many times like oh the minimum that you should have is 90 percent i'm we are trying to do the 100 one but some of the tests requires like another layer in order to find the execution path that fails you are going to need like fuzzy testing to find the boundaries and then

so it's not easily let's like like focus first on having at least 90 percent okay next month documentation for me documentation is also not a soldier is more like a sparta too it's like the priest because you are going to come you need to convince other developers to build on top of her system for me that's why he's like the priest uh and i put like three stars for the effort because i personally don't like i think developers like doesn't like to write documentation but you going to need it uh we are using ebook also i know it's not free it's kind of expensive

if you have an alternative there we spent a lot of money uh but like it's easy because it's not like is they have like a good user interface that other non-developer people could use so like our business thing also help us like on writing the documentation and then don't need to like use github and the cli okay so the most important thing here is documentation should be like an anonym like don't documentation writing documentation only in the code is not enough having only the white paper is not enough because you have different types of like public

that are going to use your documentation so this came out also from the auditors like in order to like rob your system so complex i can't understand looking directly to the code even if your documentation is correct so can you give me like an overview like why are you doing this why this is different from the other like elements in the market so yeah start from like the big contact the big picture and then like the onion deep dive later in the specific parts that you need like more explanations you know for example in our case we use uh like a numeric method in order to guess like the next value so

we have like a specific part of the documentation just explaining this this part of the system okay next one fuzzy testing for me they are the walls and towers why because like you are protecting all the possible roles in our system so if i have like a small hole that i didn't find yet okay let's build on a wall here and try to protect from the other things that i'm not expecting but they require like a big effort it's not easy like the tools that we have is like kind of hacky uh ekijna is what we are using

uh why do you have to know like the two most like the two basic types of first testing that you have out there is like the asserts and invariants what is invariance is like the general rules on your system that you don't want to like crash or like to do not follow those situations so for example every the total supply of tokens should be equal to the total amount of exercise options for example and then you run for like for a day usually like akijna takes one day to run like 50 000 simulations there and then you can start finding your

boundaries so if i have an overflow oh okay what is the minimum threshold to reach that overflow you can find with fuzzy testing then you come back to the coverage and then you create like a unit testing testing only that boundary yeah and so that's to talk about the pre-work but that's only technical stuff this is also important that negotiation with the auditors like i think we had some problems so it's good to understand well what is the scope like okay what are we going to have like two three four phases of feedbacks

like if i add a feature in the middle of the audit because like you said oh your system requires uh i don't know like admin function here okay i added this function and they need like to re-audit the function that i just added is that going to be included in the audit make sure that you negotiate all those elements the deadlines like oh that's going to take me one week to fix uh the thing that you you you found is that okay one week two weeks and the type of tests like uh we we have talked to some uh companies that they for example they don't search for economic

problems so it's kind of strange like we are inside like a defy so they say oh this is economic testing like it's testing our model we don't do that we test only security like re-entrance those kind of errors so it's good to have in mind like what is the type of that's that they are going to support yeah and an extra thing like after the audit the audit is not like the end goal before the the the launch we need to remember what is risk so risk is a function of the probability of something almost

wrong and in the size of being packed so all all the things that we have just talked is for like probability of events like i'm trying to reduce the probability of the events but that's another way to tackle this i can reduce the impact how do i do that guarded launch i can add like a cap so in the case of something goes wrong i have a limit for that so i'm reducing the impact not not the probability and like kind of an emergency stop button that it's arguable if that's going to protect you but we are trying to attack like the impact not the probability

so i think that's it uh if you wanted to check the blog post version of this talk we you you can check on blog dot pods finance and if you have like other questions about our experience with audits and from turning our company like from a project to a hackathon because i'm just talking here about the tech but is also like all the other things like fundraising corporation law you can also talk with us like we really want to help you and that's it guys i think you have like support enough to go through your journey here

so question yeah speaking um do you have any insight uh comparing hard hat and javascript based environments with brownie which is like python based 2 chain like do you prefer one over the other or like do you have some insight on that uh yeah that's a good question yeah like you you you said about browning but it is yes it's also important i've seen other projects doing tests directly with solidity if you check reflexer like the right thing they don't use like node python they use directly solidity to

write units tests it's kind of it's more complex but the reason that we pick it like javascript is is easier so if i want someone like like do you wax for example oh to help us with something or to understand what is happening that's the reason that we pick it by the way when we started brownie was not not live yet so yeah that was the reason so a hidden is kind of difficult to set up i'm just wondering was it worth it did it actually find any bugs sorry sorry so hidden is difficult to set up right initially i'm just wondering was it worth the effort did you actually find any bugs

in your contracts once you managed to set it up yeah that's why i i i told that is mark to the end because like we to be honest we launched it on polygon before having fuzzy testing and it's hard to set up by the way using docker i still can't use echidna i'm running directly using the binary yeah just so this but but in your case did it actually find any bugs yeah they found boundaries not bugs okay but boundary in it's important to find boundaries because then you can check like oh should i like should i take care of this overflow oh it's going to happen like only if 1

trillion if okay so that's not a market situation yeah so important for boundaries any other question no yeah just a small question about maybe after the arkansas uh i don't know you know you have like a small team and maybe not uh everyone is a solid dc developer so what's your best advice do you hire some freelance to like some to do some peer reviews just before going to an edit i mean just to um you mean like uh

uh finding someone to review your yeah yeah because i know that if you're the only one or just two writing the code uh i'm pretty sure there will be some issues and vulnerabilities so i was wondering what's the best way to avoid that uh i don't know maybe to ease the work for the auditors yeah yeah no for sure like uh in our case we found like the community is pretty helpful like for sure you're going if you find like any oh you're working a similar project do you want like to trade and we can review your code you review hours something like that would work yeah also

if you want to help i can like point you some like guys in the field that would be happy to help you okay okay okay thank you so that's it [Applause] okay oh