Beyond Coverage: Mutation Testing

Transcript

[Music] all right thank you everyone today I'm going to talk about mutation testing um so just a brief introduction uh my name is Eric Marty Haynes I'm a blockchair blockchain engineer at nethermind uh my background is at in smart contracts sorry my background is in computer science and software engineering and since then I've been focusing on Smart contracts and defi basically exploring the the different areas in the space for the past year I focused on tokenization uh I've

basically developed the smart contracts for Libra Capital which which is a hedge fund tokenization platform and I've also written a few articles about tokenization about nethermind a bit about nethermind we solved the simplest solutions to the hardest problems in blockchain we're a blockchain research and engineering company and we focus a lot on the ethereum ecosystem uh with it the startet ecosystem and we're doing all sort of stuff there from just like uh the core clients uh cryptography research application development auditing all sorts of stuff so let's go to the beginning in this case the beginning of uh software testing uh software testing has one

simple goal which is identifying and preventing defects in a software product right and software testing has been necessary since software has existed so basically since the 50s when software first appeared there's been a need to test it and to basically make sure it was working uh as expected obviously since then even though the goal of testing has remained the same there's been advancements on the techniques used and kind of shifts like in mentality so for example um when short after testing was first introduced uh we came up with unit testing which is basically uh trying to test the smallest possible complete unit of code that can run by itself and this way we can isolate

different functionalities and we can test them separately and make sure they're working this is very popular still used uh nowadays um a bit later on like in the late '70s there's a separation between testing and debugging where debugging is defined as trying to find the cause of a bug or a defect that is already existing and fixing it and testing is defined is trying to prevent these bugs and defects from a software product even though they haven't been uh found yet right uh later like in the 90s there's another shift towards automation um you know like manual testing can be very useful but it's very timec

consuming and inefficient for programmers to do um and so with automated testing it can be much simpler to just run tests all the time and make sure we detect uh defects as soon as they're introduced nowaday there's also a a bigger focus on prevention so we're just trying to make sure that our code has as little defects as possible or what's acceptable given our business uh use case and yeah we're trying to do that basically uh prevent these these defects so let's talk about smart contracts in this case because testing in smart contracts is very important and as you can see here just in quarter one of 2024 uh we've had losses of over $300 million and like defi exploits

hacks Etc and this is just quarter one right like we're later in the year there's been more and this isn't even the year where we've had the most of uh so far right and the main reason this happens is we need to prevent every flaw right uh but attackers only need to find one and exploit it right uh so we have security as an asymmetric game where like the rules aren't the same uh for both players basically um and why this is possible is because uh most of the time smart contracts are open source uh we need this to be able to trust and verify that they're doing what we think they do and feel comfortable depositing our funds in them but at the same time this makes it possible for an attacker to dig through the code and find any

possible exploits um at the same time they permissionless so this means if an attacker does does find an exploit it's possible for them to just go on chain and exploit exploit it without having to like create an account linked to their Identity or somehow reveal who they are or ask for specific access right we also have immutable code in the sense that sometimes if a smart contract gets exploited it's game over it's immutable there's nothing you can do you can't fix it and even if you can with a lot of smart contracts nowadays being upgradeable this upgradability is not perfect sometimes there's a delay component to it and often even if it can be upgraded and fixed it's too late and the funds are gone right and the reason

why all this is happening and smart contracts are targeted is because of the locked value they have in them right we have this huge amount of money in cryptocurrency basically just sitting in smart contracts and this is just very tempting for attackers to go after so we have a few different techniques uh to test smart contracts a lot of these can also be applied to other like uh software testing um we have unit testing like I explained basically verifying the functionality of individual components of code then we have integration testing which is basically uh putting these tests together and and and testing that all these features that we've tested independently or are also working as expected uh when combined uh we have invariant testing which is very popular

uh with smart contract test which is basically verifying that some invariance that we set in our code that should never be violated are basically holding up uh when we call functions in a random way uh we also Al have fuzzing which is similar it's basically we have our our functions and we test them with randomized inputs and this helps us detect some possible edge cases uh that we might find in our code here we have some different tools that are available for each of these tests a lot of them you might see repeated I'm not going to like read each specific one uh but yeah these are just some popular ones we also have some other tools we have static analysis which is basically a technique for analyzing the contract

code without executing it we have symbolic execution uh we're analyzing the execution of the code uh the different paths that the code has depending on the inputs we use uh we have formal verification which is a very interesting technique where we prove the correctness of a given program or contract given a formal specification that we have predefined and then we also have mutation testing but this is great out because uh we're going to get back to it later and again here you have some tools that are pretty popular uh to do some of these kinds of testing all right we're going to take a look now at bank. Soul this is a simple uh Bank contract uh bear with me I tried to make the code like pretty simple um

we basically have uh a mapping with the balances a deposit function that adds eth basically to our balance a withdraw fun well before the withdraw we have a get balance function it's just returning the balance of a particular address and then we have the withdraw function which is is basically checking you have enough balance to be able to withdraw it's uh subtracting that amount from the the balance mapping and then finally it's sending that eth uh to the address withdrawing and checking that the transfer was successful and of course we're going to write our tests uh for our code and these are solidity tests basically the kind of tests you would see like using Foundry or something like that again these tests are pretty simple uh we have

the setup here uh then we have test deposit which is just basically checking that our deposit is working we're testing that first we have a zero balance and then after depositing our balance 50 ether our balance is of 50 ether over here we're testing withdraw we're basically the same we start with a balance of 75 ether uh we deposit it uh we check we have that amount then we withdraw 50 ether and we're checking that the balance is different um and finally we have test withdraw fail which is similar but we're trying to withdraw 15 ether when we only deposited 10 and so obviously we would expect that to revert and the test is making sure of that so we can run these tests uh basically command fors tests if you're

using fors test if you're using Foundry and we see all of them pass we can see the gas they used the time it took that kind of thing okay so our tests passed but how do we know if these are actually good test tests well we could use code coverage and code coverage is a measure of the percentage of source code that was executed during testing it's often used to evaluate the quality and thoroughness of tests um and usually High coverage suggests that there's a smaller chance of undetected bugs right if we covered most of the code in our tests and we ran through it uh we shouldn't expect to find like a huge amount of bugs there at least in theory so there's different types of coverage uh there's line coverage which

is the simplest one it's just a percentage of lines of code that our tests have covered we have statement coverage which is similar but basically with the number of statements executed we have Branch Branch coverage where we're basically checking the different uh branches of code that our program has and which of them have been executed during the test and function coverage which is just a percentage of functions that we've called in our test so uh we can get the coverage for our tests with the command Forge coverage in this case and we'll get this it looks pretty good we have 100% line coverage 100% statement coverage we have 75% brand coverage we have four branches there's there's one we missed and we

have 100% function coverage uh so this would suggest okay our tests are really good right like our our code is perfect it's well tested no issues well code coverage has some shortcomings and it's it's basically because a piece of code being covered be a line a brand Etc doesn't necessarily mean that it that it is correct or that it is checking the expected State change that kind of thing um so so it's not necessarily checking how correct it is just that we executed it um and this this is problematic for two main reasons the first one is it can lead to prioritizing coverage over actual uh quality of tests uh sometimes we try to get a good coverage because

it's a you know it's a metric people use uh audits sometimes emphasize the coverage of the tests that are in a short test street so we try to go for 95 100% coverage but we're just thinking about improving coverage about actually making our code secure and properly tested and it also gives us a false sense of security because we think oh our coverage is so high our tests are perfect but it might not be the case and here's a small spoiler in this withdraw function um in the test withdraw function that we have for testing it we actually have an issue where we're not properly testing it uh it's fine if you don't see it now or we going to get back to it later so there's another way uh which is mutation testing uh the easiest way to explain it is basically

testing tests and mutation testing was first uh proposed in 1971 by Richard Lipton and it is based on two main hypothesis the first one is the competent programmer hypothesis uh which is basically that competent program programmers write programs that are close to being correct they might not be 100% correct but they're pretty close and if they have issues they're usually small deviations from the correct code and we also have the coupling effect hypothesis which is basically that simple faults in the code code can couple together to form other faults that can be maybe bigger faults or different kinds of faults so here's where we're talk about the the mutation testing technique to explain it very simply it's based on

mutating components in our contract or in our code to evaluate the existing tests and how good they are so it works by modifying The Source program using mutation operators we'll get into those uh to create a mutant piece of code or contract and then running our existing tests against it if a test fails it means this mutation is killed and for this to be possible though we need three conditions to be met uh the first one is that our test must reach the mutated statement right if our test doesn't reach it we're not able to to execute it um the Test's input data should create a different state in the mutant when compared to the original code right it doesn't matter if we reach it if like our our test is just

doing like the default data null zero for example and it maybe the mutation doesn't cause any change in that case right and third the state must be propagated by the mutant and then checked by the test if our test is not checking uh the result uh then then you know it it it might not make sense so um these three conditions also mean that we have a set of qualities that a test suit Suite must have in order for mutation testing to be the most effective just to be clear it can work if some of these aren't met but the better you meet them the more effective mutation testing is going to be when when applied um so we should have a high coverage basically so the mutations can be detected because we want to reach

those all those lines of code and branches Etc functions in our tests um the input data should be varied and realistic to trigger different state changes in our tests and also our tests should check the results of these State changes to ensure the correctness of the program so uh we talked about mutations that were killed basically the mutations uh that are tests detected what about the ones that are tests don't detect so we have two kinds we have equivalent mutants these are basically false positives that contain mutations that change our code but don't really change its behavior in the case of smart contracts this even means sometimes we might have a mutation with the exact same bite code because the solidity

compiler compiled it into the same thing even though the solidity code might be like slightly different uh these we don't really care about because they're doing the same thing as our code they just look slightly different uh the surviving mutants are then important ones because they basically contain mutations that alter the logic of our code but our tests did not detect it and this is concerning right um so it basically means that our tests are not checking all the conditions that we thought and we should probably look back to our tests and improve them so let's take a look at possible mutation operators uh this is focused on like solidity smart contracts there could be more mutations or different kinds as well uh this is just an example we have arithmetic operators so just you know changing a plus sign to a minus

sign a modulus to exponent that kind of thing we have logical operators right we can change logical and to logical or or the other way around we can change bitwise operators so we can uh change bitwise or to bitwise and sure as well uh and we can also change uh bit shift right to bit shift left this is a very relevant one we can change uh uh equality and inequality comparisons like uh equals not equals uh greater than less than that kind of thing uh we can change hard-coded values so if we have constants in our code um we can change a negative number to a

positive one a number to zero true to false a string to an empty string an address to the zero address we also have deleting lines or function modifiers in the first case here we just deleted a token transfer in our mutation and the second example we deleted the only owner modifier from our function so it would make the function accessible to everyone instead of just the owner and the mutation and we can also swap the order of lines so here we're doing a transfer first and then we're registering when the last transfer happened um but in the mutated version we're first storing when the mutation happened and then doing the transfer this could mean nothing or if

our transfer has some custom logic that's using using the last transfer for something it could be problematic right so uh we also have automated mutations because basically doing this by hand again is very timec consuming so like with most forms of testing doing it in an automated manner is is usually better um yeah it can also lead to human bias uh because we're the ones testing our Cod we might think that some sort of mutations are better but we might miss some edge cases um so we have automated mutation testing uh it's basically doing this sort of technique uh using a default set of operators randomly on our code so these are some tools you can use for running mutation testing on Smart

contracts I'm going to show you an example using vertigo RS but here you have a few other ones uh Foundry has a question mark because it it's had an open issue for a year about um including uh mutation testing inside of Foundry it's still an open issue but there's some discussion uh going on there so hopefully we'll see it at some point so let's mutate bank. soul and see the result so if we run vertigo RS on our um Foundry repo we will get this output so it's basically saying we have three mutations and two out of those three mutations were killed we have the first one that was in line 11 where basically it mutated the greater or equal sign to

a less than equal sorry less sign and yeah this was basically checking if the user had enough balance our test detected it so it was killed uh here we have the other two the first the the second one was also killed it was basically changing a A plus equals assignation to a minus or equals one um basically in in the balances when we're depositing obviously this breaks our tests so it was killed but this is the most interesting one we have the one that lived one that survived uh in the withdraw function where we went from subtracting a balance when withdrawing to adding to our balance and somehow our tests did not detect this and this

mutation survived right so this is concerning we should take a look into this um so yeah if we go to our contract we can see there that's the line that was mutated we turned it into a minus equals and we can see if we do it manually and run the tests they still all pass right so something something's up there so here we can see the actual line of code where we're testing and we can see that something doesn't make sense we're checking that the balance is different than the starting balance but we're not checking that it's the original balance minus the amount we withdrew so this means since our mutation changes the amount of of balance the user holds it just increases it instead of decreasing it our test

goes oh it changed that means it works great but that's not how our our test should work uh so we can fix it now we're actually testing that the that the resulting balance is the starting balance minus 50 ether which is the amount we withdrew and if we change that and run the tests again we can see that the assertion is failed this time and our test is failing uh with those amounts over there so now if we run our uh we run vertigo RS again on our fixed tests we can see the same number of mutations but this time the three of them were killed we have the first one again the second and the third one and here what's relevant you can see it was killed um so we we just use mutation testing to improve our

code and as a conclusion to this talk I just want to like if you keep something from this talk I want you to remember these things I think they're relevant uh testing is critical to reduce the risk of exploits and smart contracts it's extremely important uh mutation testing is a very powerful technique and it works really well together with other techniques and it's great for prioritizing the quality of tests over like the quantity or or the coverage of your tests and also mutation testing tools are not as widely used as other security tools they're not as popular uh the tools I I showed some of them are not like the most well-maintained uh so it would be very interesting to see people using these tools more uh supporting them and that kind of thing so I hope at

least you you can take home that this is a technique you might not have known about and that it might be useful to you at some point so that's all for me if you want to find out more about nethermind you can check our website you can follow me on on GitHub or Twitter if you just want to see what I'm up to or ask any questions and here you have the example code if you just want to take a look at bank. soul and yeah that's it thank you very much so if there's any questions you can uh feel free to ask them now um thank you for the talk I actually have uh two two or three questions but

the you mentioned at some point that the the the game was kind of asymmetric mhm but uh if everything is public I mean the attackers and the code writers you know they have same chances it's it's basically like saying that you know life should be fair which is not you know uh so I was maybe you mentioned the fact that they need to find one bug to exploit but I'm more thinking from an information theoretic perspective mhm so um and my my second question about the tools maybe uh because they aren't uh I don't know if they're there is any uh you know uh if they discovered any real

vulner vulnerabilities in the in in the ecosystem so maybe that they can have more uh you know more uh visibility I don't know because it kind of feels like a a weaker version of symbolic execution at some point but maybe more effective uh and less demanding on resources I don't know if what what you're thinking about that so on your first question I'll just basically said say I meant asymmetric like you know if there's 50 critical bugs and as a developer you fix 49 it just takes that left over one right so you know you you fix most of them but you still lost kind of thing but yeah on an information point of view like there's equal information on both sides probably the developer knows more

about the codebase as well right um on the tools what you said is interesting uh the thing to keep in mind this mutation testing isn't really testing your code as much as it it's testing your tests right so it's hard to point as at a specific uh line of code in your smart contract and say oh this like this was prevented because of mutation testing like in the example we we did there wasn't a flaw in the contract there was a flaw in the test um but still flaws and tests are important right because our code goes through Cycles where it's maintained it's changed new people uh come on to develop it so even though our contract at that point was fine who knows if in the future someone accidentally rearranges line of code or does some refactor they

don't realize they've introduced a vulnerability and our tests don't detect it right um so that's where mutation testing Can can shine but it's very hard to point that an example where was successful because you can say oh like thanks to it I fixed my test and this test three months later detected a critical issue right like it's hard to keep track of those things um but I I do think you rais an interesting point if people were more aware of some examples where this had worked to prevent some like exploit or something I think people would be much more willing to give it a try or or explore it any other questions all right well thank you very

much