DappCon 2023: How To Test In Production Without Blowing Things Up?- Bogdan Habic

Transcript

so before I start this is a trojan horse type of presentation this isn't the actual title and I'm not actually going to talk about this and we're going to see what that means in a couple of seconds so before we start obligatory introductions hello everyone my name is Ban hich I'm one of the co-founders in CTO at tenderly and at tenderly we're building a fullstack infrastructure solution for web 3 teams and daps that means literally nothing to anyone ever but it sounds very nice on the landing page so that's why I'm sending it here but before we start raise of hands how many of you have used a local node okay keep your hands up and then lower your hands it's something isn't true how many of you have used a local Fork of a network of a production

Network cool how many of you have used the test net before cool how many of you have deployed to mayet to test something out stand up please no no no stand up come on this is the the point of the presentation you're screwing up you're screwing up The Vibes here so sit down if you didn't receive a death threat for deploying to production now that I got your attention hello everyone and today we will talk about how to test in production without actually losing your life I didn't have an aneurism on stage right now and talk about something that's completely tental to this topic let travel back all the way to October 2020 the defi summer is

in full effect even though this is autumn we were calling everything defi summer because we didn't care about names we only cared about yield and one of the projects which was very famous at that point and it's also still extremely good at this point is urine Finance so everybody was using urine Finance in something called vaults we'll talk about it briefly to get as much yield as they can so you put some money in into a magic thing and suddenly during dinner you paid off for the dinner that you had and then also your web two friends said it was a Ponzi scheme they hated you for earning money for basically doing doing nothing yada yada yada it was an honest way to make a living and let's meet Andre cron I probably butchered his last name so sorry sorry Andre if you ever actually

listened to this but he had a problem what was the problem that year in finance was solving the problem was each morning he would wake up and try to find the best yield on the market he would find a new defi project hopefully that's not going to be exploited in the next couple of hours because the apy was over 800% basically at that point and he wanted to automate that and he created something called The Vault the idea behind the Vault was okay here's a pool people throw money into the said pool and the pool is intelligent enough to find the best yield on the market and to move it around this is now a defi primitive back then this was complete Innovation all new Etc there was a problem while he was

testing this it was extremely but like extremely hard and impossible to test this out if you to deploy to to a testnet none of the D5 projects were on the test net and the apys weren't the production one it was extremely hard to figure out how good the Vault will actually perform um when other people put money in it there isn't actually a way to collaborate it was extremely hard so he came up with this Mantra saying I test in production there's a couple of benefits to this so all the data was readily available it's production the data is there so all of the balances of the apys everything was there other people can use it du it's un me it of course other people can use it and it's the same

environment is production because it's production but there were a couple of cons as well when he would deploy a new version of the project that he says people shouldn't use people would ape in millions and millions of dollars thinking it would get higher yield and those products those test variant of the products would get exploited and he started receiving death threats in October of 2020 now a good example of that was for the EMT token if I remember correctly or em I cannot remember the acronym correctly but the point there was that he put something on chain $15 million were aped in in like three hours someone exploited the test contract basically stole 7 million of those dollars and then gave 8

million to him to try to to make it seem like he did it so how did we end in that state and where are we now so web 3 testing fiveish years ago was extremely rudimentary so you would either use ganache or remix there was more or less no other way for you to test something against the network the idea is you spin up ganache you are using truffle you deplay deploy all of the contracts to ganach and then you either click stuff around or God forbid write unit tests at that point and then hope everything works and then if you wanted to share something with someone you would either literally pass them your laptop because it was pre-o in office was cool and we actually saw each other at other places

other than conferences but basically would pass them your laptop and they would test stuff out if someone was on the other side of the globe you would probably deploy to a test net and give give it to them for testing purposes sure moving on and then we have like the second Renaissance or whatever you want to call it of of web three testing and this is like from 2ish years ago up until now so defi summer was in full effect and I think it's very important for historical context to know why a certain thing was invented so at that point both hardhead and Foundry started popping up ganach added forking maybe a bit earlier than that but why we started to have a need to Fork a network is because of the money Legos so if my

project is using D to create a new defi primitive I need the production die data and then if someone's using my project they need my data and D data as well and we had a situation where basically people would build out a project that would use other projects and those 10 projects would use like five different projects and you would get basically that you need production data for around like 70 something different different products to actually figure out what's what's happening with your functions that are being triggered so CAD Foundry local forming forking ganach as well around that time a bit earlier this is a small shell we also deplo developed tenderly Forks which the idea was they're hosted you get an RPC link it's

a fork of a production Network and the benefit there is like it's not on your laptop it was faster and then also it can be shared with people but this was like the bleeding edge and this is more or less still the bleeding edge and test Nets were still being used for crowd testing so again if I wanted my community to test out something I would need to deploy it to a testnet invite them they would start using it stuff wouldn't work networks will get would get congested and even now people are begging for Gorly eth and we think for some reason PO is going to fix it but the same thing is going to happen there as well eventually and that's how we did testing this is amazing is it it's not this is extremely sad actually what does

Web Two have so confession but I guess a lot of people in the room can make the same statement in my previous life I was a web 2 developer uh Tamar from our team would say I was a phdp developer because I wrote PHP for two three years and and everybody remembers you as a pH PHP developer they completely forget like 8 years ago of ago afterwards but the thing that we are definitely missing from web 2 is a bunch of things so at the at the neighbor's Garden the flowers are much nicer so you have stuff like per PR staging environments the idea idea here is I make a pull request on a GitHub gitl or whichever other repository that I use and I get a full-on staging environment anyone who

who is reviewing the pr can take a look and use it we have red green deployment so the idea is if I have six replicas of a service I can basically change services one by one and then if anything is going wrong wrong it just automatically rolls back if if everything's okay the deployment is green and we deployed there's Canary deployments as well now this is a complete tangent to the to the presentation but those who know me know that I like saying interesting trivia so how Canary deployments came up with the name actually was in coal mines in one of the two previous centuries to be completely honest I don't know which one um people who are digging in Coal Mines would take a canary the bird would take a canary in a cage to the coal mine

start digging and then if they hit a natural gas deposit that humans cannot smell once the canary passes out you would know that there's poisonous gas in the said mine and people will start running out in engineering it's it's much less it's much less of a bad thing for the bird because there are no birds but basically how it works like is you have 100% of your your traffic and then you direct like 10% of the traffic to a particular service that has a new version and then like to 15% 20% 30% blah blah blah and then when the whole traffic is going to the new deployment basically the canary one is promoted to the production one and then finally which is going to be very interesting for this talk is something called

traffic shadowing this isn't something common it's very hard to implement in web2 the idea here is let's say for example Netflix is testing out the change what they can do is list Lally clone all of the traffic all of the interactions and everything that's happening in production with the difference that is hitting staging so you get real user data real user interactions real user traffic everything only it's in a safe and isolated environment and we are missing so much but there's one thing that web 3 has that web 2 doesn't other than amazing memes and other stuff it's that all of the data is there there isn't a thing that any of our

projects do that's not basically in the same database and we can do much better than that so story time this is eil protocol if it's a defi project who I'm not which I'm not trying to pitch because I'm an engineer I'm not a financial person I can read the description for you if you want Ethel is a financial inter interrup ability layer that connects the whole web three space facilitating new value creation via crowd Lending again I'm quite sure amazing job the team is amazing we worked with them I'm not going to say that they understand all of the financial Wizardry that they're doing and they want to deploy to arbitrum but there was a question are we making something useful are we making

something that will work and like will we get hacked like in the first two days so they came up with an idea let's spin up our own testnet what can you do right now to do that you can take gu Aragon or anything off the shelf even Z KVM or whatever and spin up a network that's completely empty and people can use it that's not sexy enough I think another thing that you can do you can probably deploy it to sepolia I guess and then when you deploy it to sepolia you have the production data issues but you're on a testate it's safe nobody will get hurt Etc you can also deploy to production and then hope your anonymity prevents you from getting killed basically by a d gen but also why risk something like

that so they literally spun up an eth test net what was the idea there they forked arbitrum in a certain way we'll see how users can connect their wallets to the test net itself but the interesting thing is because it's the full carbon copy of arbitrum all of the production data balances token balances native currency balances all of the contract data everything is already there they add it to mamas they add it to whichever wallet that they're using and sudden ly they're transacting with the same wallet in a completely Safe Way in an isolated environment they can also have reports in bugs being reported to them sorry uh

uh bugs and xpl being reported to them so the idea here is if you have a bounty program and this is how bounties work in general but if you have a bounty program there's much of a higher chance that someone will report an exploit to you if it's in an environment where they cannot like where the incentive isn't to actually hack the thing but to get the Bounty why because someone else will be quicker and get the Bounty and it's still fixed and they did just that so this was in July 10th uh this was us in the background we're going to see how theil testnet is live here and they said hey everyone come here find the bugs try out the product give us feedback report exploits and everything else that you can do with that another cool thing that they did is that they showed to their Community how

they will do upgrades how they will add new services so how the product will function once deployed to arbitrum so here we say that they added GMX uh yep GMX and a to their test net so again isolated environment but showing to the community giving them a pece of Mind basically that how you're going to function here in this test net is also how your team is going to function on the production Network here we see a feature so not an integration so the feature here is call options the call option service they deployed it there and people could test it out use it and see how it works and I think it was a great success so they had over 3,200 people participate in the test net itself over 10,000 transactions as part

of their contract ecosystems were done through the UI so this isn't this isn't Bots doing traffic or whatever this were people actually using the product they gave out some prices one thing that's not listed here and and one of the founders actually told me that I can I can say this one exploit was also being reported where the product would have been killed literally a week uh like within the week it was deployed so they were kept safe because they had their own test net basically how did they do it so coming back to the Trojan Horse it was more like a Babushka thing it's like a trojan horse inside of a trojan horse and then an announcement I'm glad to announce something called tenderly test Nets now

I can pitch it to you but first of all let's see how gorgeous they look so here on the left you can see all of the transactions that happened on your test net that you have spun up so here we can see that in the last 24 hours like 123,000 transactions were done on a particular test net here you see all of the gas that was being spent people generally use tenderly for gas profiling and optimizing gas but there's always a question if I do this optim optimization in aggregate will I actually save on gas so here by having people use it for real you can actually see how much gas is being spent and then finally RPC request debugging so you can see how your UI is using RPC requests you can see how your

backend ecosystem is using RPC request you can see how your the graph indexer is using RPC requests and what's even cooler you can debug non onchain RPC requests so you can debug e calls so a frontend engineer for example can come and say hey this eth call failed and then instead of bugging someone who knows solidity they can actually go into the tenderly debugger and see it it's a full-on environment so it comes with the faucet itself here you can see either people requesting faucet amounts or you you sending it to a faucet this can also be public in a sense it's an RPC link only a very magic one so it's very easy to integrate literally with anything that you already use it's if it takes you five minutes to

integrate sepolia it literally takes you maybe even less because you're forking a production Network and maybe the best part you can have an infinite amount of them so on every pull request you can have a test net that does something which people use so it's a permissioned RPC URL generator we can pitch it in that way you can share it with your community integrators Etc to use it it used the fork chain State this is similar to tenderly Forks you will see why this is special but it Forks a network in a particular way all the transactions that happen on the testnet are readily available in tenderly with all of the contracts verified everything basically everything

that people use tenderly for production is already there on the test net itself it has an integrated faucet you can set token balance is an override state so when you boot up the test net you can say hey these people have millions of usdc this contracts code is actually different the allow list for for this particular contract is different the signers in this multisig are different so you can override all of that and maybe the coolest thing and then engineers in the room will understand why this is extremely hard it follows the forked chain so what does this actually mean this means that once you Fork main net for example chain link is constantly up to date all of the lp positions are constantly up to date all

of the balances are up to date everything is up to dat so it's not a snapshot of a network it's not a fork you didn't fork in a point in time and then it dies off somewhere it follows the production chain everything that's happening it's shadowing traffic that's the correct term and you can test your application with all of the production mained data with a single click more or less what are the benefits you can integrate your dap or your protocol whatever easier with founders with other partners sorry so let's say for example I'm safe and I want to have people integrate modules more easily I can have have a safe destinate as an example you can showcase

new versions of the protocol to the community self-explanatory you can do bug Bounty programs again you can say anything reported here actually has a higher higher Bounty than anything happening on chain you can have staging environments for your whole team to collaborate and I'm quite sure just like with all of our tools people we'll abuse it and those use cases are actually much better than the ones that we thought of but they think it's it's it's an extremely cool thing that it can follow everything like that and I guess the question is when can I test this out so I'd also use this opportunity to steal your private keys I'm joking to invite you to a beta so if you scan this and and leave your information it can be telegram it

doesn't have to be email so we won't hunt you around with ads over the Internet uh we will be holding a Clos beta so the ethil thing that they showcased is a closed Alpha but here you can sign up for the beta and test it out yourself stop begging for test net e please for the love of God and thank you so much I'm open for questions so you can drink some water before drink some water I can do an ASMR version wait any questions yeah clear the throat well so gentleman over here the

ASMR that the water um thanks for the talk so the question is if everything is up to date what happens if the state isn't working can imagine I have five even main net and I spent these five even the test net and then on mayet I spent them these five again what happens then how do we resolve these issues with transaction that would fail on the test net but we're successful in main net sure I'm coughing actually to buy time here sorry I'm good any who so once you touch a storage lot this was such a bad idea you want more no no I'm good once you such a storage slot that storage slot is followed only from the test net itself I hope that answers your question

I would do it in more words but I'm dying here right now any more questions I think over there if not hey um one of the main uh problems that if we have to set up a test net is the cost right because we uh yeah for any use case let's say we have testing a staking infrastructure and we set up the test net uh this probably costs quite a lot to us and currently um sometimes we need to maintain these test Nets as well for a longer time so in this case can we leverage it like as a paper use model and how would the cost look like in this case it's a p paper use model yes how

much it will cost I have no idea to be completely honest we're still testing this out um the thing to note there is a couple of things so this isn't running any consensus layer per se so it doesn't have validators it doesn't have staking it doesn't have stuff like that it's a virtualized network just like our forks for example don't have Val validators and stuff like that and I think when you're running a test net unless you're the test net itself is your product like you're working at polygon and want to have a zkm test net if you're building a d project nobody cares about the consensus layer part everybody cares about the smart contracts and then if you take all of that out of the equation the test net validators and everything else it's actually much cheaper so I hope

that answers your question I don't have a concrete answer but yeah if you sign up I'm sure we're going to send you an email about it he can ask a do you already have like sort of GitHub actions ready configurable to like start using this um on every I can lie to you and say yes so it's very easy to configure a GitHub action with our CLI and we are working on having a GitHub action that you can just one click install but even with that and this is one of those don't trust verify it shouldn't take you like more than 10ish minutes to set up our CLI for devits which is like a feature similar to this one only for local development similar thing is going to be for T Nets so faster than a Bitcoin block yes

faster than a Bitcoin block any other questions I don't know if you're yawning or raising your hand no okay okay if there are no questions we're going now into a break so thank you everyone for coming and see you later thank you all so much