Parithosh - What?... Merge testing is evolving (Insert pokemon evolving logo here)

Transcript

all right good afternoon welcome back everybody uh we'll go ahead and get us uh started here with this afternoon's um list of uh presentations and panelists and leading us off first after lunch here is going to be Perry Perry is a uh a researcher with the ethereum foundation and he's been testing a lot of the uh ethereum updates hey everyone can you hear me okay perfect um so today I'm going to be talking about testing how we've changed stuff since the merge and how the

testing stack looks today uh to begin with you might have realized there's a lot of things in the pipeline for ethereum so the question does come into play how do you generalize testing for example how do you generalize it between dun and between wle uh the simple answer is you don't um we tend to have a specific testing stack for each uh upgrade along with some inherited generalized testing from the previous steps a good example of this is you're going to have to look at the EIP you're going to have to read what parts of the stack are being changed and you're going to have to talk with client devs to figure out what they are worried about for the upgrade and

try and address those problems for us uh for example Peter has this really in-depth uh PR about how the transaction pool changes for blobs we just went through and everywhere where he had a concern or every single bullet point we tried having something on the testing stack that addresses that bullet point so what does testing look like today what are we kind of covering um where of course covering the current Fork uh that were that current upcoming Fork so Denon uh we work on a potential future Fork so verle um we're working on a couple of features that might show up as a future forx so you might have seen uh mik's

talk earlier about deposits uh so that's EIP 6110 so we have some testing for that uh we have some testing for whisk which is um helping with validator privacy there's a couple of testing Stacks that are helping client teams optimize their clients so we have um we have like a test night running for ethereum GS to try and figure out how snaps sync works for them we work on Big Boy Beacon chain tests to figure out how big the valid data set is and as you can see there's there's there's very few things that are shared between these topics themselves but the problems that were raised between all of the testing Stacks were kind of General so I'm going to go through what those problems are

and how we solve them so the first one was the setup toil so testnets used to have a lot of redundant configs extremely easy to break them extremely difficult to set them up it was also very time timec consuming to add tooling to any sort of test uh testnet and if I had set up stuff for one network and another one a month later there was already a huge config drift we didn't we hadn't built it in such a way that you could share updates between networks so it depended really on who was working on what so what we did was split everything into a lot of parts that make it way more easy to maintain uh so we we use anible for

all of our testing uh for the test nties so all the barebones logic such as this is a teu node and it needs to be run on that machine or I need Genesis and Genesis files need to be moved to machine X so kind of the bare barebones logic itself was moved into answerable collection General and then Concepts or components were moved into their own tools as well so Genesis is something that every network is going to have to go through so we simplified it and we have the ethereum Genesis generator so it's just very easy to configure easy to toss in and you can just use the output of that in the aforementioned barebones logic tool similar to tooling um you would

spend a decent amount of time configuring the tooling so instead what we did was we uh switched to kubernetes for tooling so we have Helm charts so there's a Helm chart that runs the block Explorer Helm chart that runs the faucet Etc and there's this cool concept called gitops which is what you see in git is what gets deployed on on a kubernetes um stack so we use ago CD for that so anable generates the git configs you commit it to git and the tooling just shows up which is perfect you never have to really think about tooling um of course there's some friction first couple of months getting stuff working well was a challenge but but I feel like we've gotten really really good at this now and the next

problem was okay we have all of these parts and all of these abstraction layers how do you how do you bring them together um GitHub has this feature called templates so we're using that to create a template test net so that sets up the entire folder structure you need all the same defaults you need along with a readme on what values you should change and on the top right there's a nice button that says use template and then it'll create clone it into a repository in whatever name space you want and that has everything you need to make um kubernetes work everything you need to make the test net work it's it's kind of really A reduced set of changes to to get everything up um the next one was how do we keep explorers working so explorers are

really specialized pieces of software um main Net's getting bigger main net gets harder to run so explorers had to keep up but the more they had to keep up the harder it was for them to also support testing and Deets are really weird dependencies so a simple example is for veral we need to have a lot of veral libraries no realistic Explorer is just going to import random libraries that they' never use on Main net onto their Explorer so we had to maintain a lot of fox um that got harder as time went by so we had someone in the team right a lightweight self- hostable extremely Fork friendly Explorer we call it Dora um and that that helped us uh we could add a lot of features that we wanted

into the Explorer more easily and for smaller scale Dev Nets as well there was also some toil in getting Docker images working so every single client team has their own CI but they have their own logic as to when they push which architectures they push and so on um a simple fix was to just host our own CI so we build Docker images for every single client out there sometimes we add some debug tools but if nothing else they're at least debug tool friendly um and then we do them on hourly builds for branches that we track and if we don't track the branch you can also just trigger a manual build for whatever Fork you want um the next one were Forks was really hard to debug so every node on

the network has its own version of what the network looks like and when there's an issue you kind of need to understand how the node perceives the network um so we wrote a fork visualizer that uses the standard API and then the uh Fork view was also integrated into the Explorer so now it's actually easy to figure out how Fork the network is and then it gives you a good starting off point to know what you need to debug and where you need to start the next one is a bit of a bigger topic uh the peer-to-peer layer in ethereum was really hard to analyze um we didn't really have the tooling and even if we did it was built into custom to uh custom code bases for the clients

or they had metrics but the Met metrics aren't really standardized it was hard there was no real analysis pipeline so what we did was uh the team buil zatu which is a network monitoring and added an an analytics pipeline to that so you have the entire Suite you need to actually analyze the network and what that looks like is a centralized server so that's a satu server uh collects events from various sentries which are just lightweight instances that run alongside uh El and cl and it connects to the cl's event stream annotates stuff passes it onto the server and then the server handles persistence passing it on to the pipeline and so on uh we also have a

discovery node that just kind of crawls the DHT and figures out what's going on in dis V4 um and we have mimicry which is a tool that collects a lot of information about transactions so an example might be that you want to know when a transaction was first seen on a network versus when it was included in the network depends on what your use case is but at least we're collecting that kind of data to understand how the network behavior is and then Canon is a client that backfills data so that it it helps for historic users as well uh we have a nice overview on the notes page so have a look about that one and we do run this stack so we're running it on mainnet seoa girly uh hsky we run it on all the blob Dev Nets at least the latest ones

so that gives us a really good insight into how the network is actually looking um and we can do overview graphs like this it's a bit of a chaotic screen I apologize for that but you can have heat maps on when attestations are seen heat maps on when the blocks are propagated you can have a graph that tells you oh if there was one blob the block took so long to propagate if there were six blobs it took X plus 100 milliseconds to propagate and that gives you a lot of deep an analytics into okay maybe there's some assumptions we made wrong or worst case it reinforces the assumptions that we did make and the last one I want to talk about is that it was really hard to test

changes quickly um earlier we needed to spin up a network or you had to maintain veros shell scripts it was just extremely difficult to just test an update so we switched to using local testing along with a tool called curosis so curosis allows you to specify a config either in Json or in yaml so here's a Json example for how you spin up a local network with three le nethermind as well as Lighthouse nodes um it handles Genesis for you it handles everything that you really need um and because it's a testing Suite it does some smart things you can change to slot time so and you can change when the fork happens So within a matter of a few

minutes you can have a local network with every single CL and El out there um and know if a fork is successful or will fail and we of often do this to sanity check stuff and we've even added the me workflow so you can have U the entire relay infrastructure set up and it can help with either local integration or to just sanity check things or make sure that your fork works well and the nice part is that it supports both the docker back end as well as the kubernetes back end so if you do want you can point it at kubernetes and get a more scalable setup with X nodes over certain regions so it really really helps us with testing rapid iterating changes before

we put in the effort for a fullscale dnet and here's how the testing workflow kind of looks looks like today it's not a comprehensive workflow but at least it gives you an idea um you have the specs specs get released after a lot of work from researchers um the clients work on this in a fork or a branch um parallely there's this testing tool called Hive Hive is you can think of it like you pass it an input and an output and Hive asserts if it's sets up the network and asserts if it's doing what it should be doing so Hive gets updated and because Hive is extremely powerful and we're already able to think of a lot of scenarios where clients might fail we

push all the client teams to first test on hyve so everything that we can expect and can catch we have caught at this step and if we've done that then we start using local test Nets with kosis and that gives us the sanity check that hey if we spend half an hour an hour two hours setting up a network it's not going to fail immediately um and then we can start our full feature Dev Nets um and if you have noticed from the dev nets for uh Denon versus the dev Nets we used to have for uh for the merge they stopped outright failing anymore and the reason is because we've pushed for a lot of um changes Upstream to catch all the bugs that we can Upstream way faster than we earlier could and whatever we can't

catch there is what what's being caught on dev Nets now that's great from a cost perspective time perspective from a lot of different ways and once we're happy with Dev Nets we typically tend to have a public test net or one that's more announced and supported so external parties explorers tools can start using it start integrating it and once that's working well we tend to have this concept called shadow forx where you Fork a main network but in in on the side you don't affect the main network uh and this helps us get metrix about um how how that that fork would have potentially gone in your in your side chain and once we're happy with that we tend to go for public test Nets as well as main net shortly after so that's

roughly how that's going to look and what kind of tests we run for the different layers of the stack you have um depending on if it's evm changes then we rely heavily on fuzzing or on fuzzzy evm and tools that compare how the evm output looks between different clients uh if it's peer to P layer we have chaos testing tools we have uh Dev Nets that we run if it's engine API based we tend to do more static testing uh if it's sync then we tend to do sync tests and so on and so on we have we have a lot of tool sets for different layers of the stack and here's kind of the the question where is Denon today um we did have a small spec change that has been

added not too long ago and it's being implemented in all the clients I think the spec change is partly caused by some of the outputs from the testing partly caused by researchers thinking about the problems a lot more um and this is going to help us simplify a lot of things and make sure that Denon is a lot more safe once it's uh hitting main net once that spec change is implemented we're going to be doing a public devet so we'll have devet 12 and we have a lot of Shadow Fox so we're looking to have bigger Shadow Fox this time like a couple hundred nodes uh and that'll give us a decent amount of time into how long a network takes to process it'll give us a decent amount of information about uh basically everything that we would need to know

before we can commit to public test Nets and once we're done with that we're going to parallely start foring public test Nets so we most likely suggest gly hsky and saoa and that'll give us uh information on how uh really big network is going to behave with blobs in it and in the last couple of months we have two big tools that are interesting so we have blobber from Mario from the testing team and this is an evil manin the-middle peer-to-peer layer tool that sits kind of between the validator as well as the beacon node and the blobber decides which part of the network gets honest messages and which part of the network gets malicious messages and this

is a really really interesting way you can create a lot of scenarios which we didn't earlier have the capability to create and the second one one is attack net um so this is using a chaos testing framework uh we're using chaos mesh and it's a collaboration between like trail of bits EF devops and curosis and this would allow us to push the network on different layers of the Linux stack so you can mess with um the network latency you can mess with um packet drop you can mess with CPU faults Ram faults you can add latency to the uh read and write from the disk you can do a lot of things here and we just started using this tool

so hopefully we're not finding anything too big but it's going to be interesting over the next couple of weeks at least while we parallely go ahead with uh with tenun and what does Denon mean for the home stakers always a big question uh So based on the last couple of Dev Nets it looks like there's only going to be a marginal increase in CPU and RAM um it doesn't seem like there's a big one so you don't have to worry about going out and getting a new machine for supporting tenun there is an increase in dis use roughly 50 to 100 gigs it really depends on um on the values that we choose for the network and that's what we're

focusing on a lot of testing over the next couple of months to figure out those values there is an increase in network use so you can imagine roughly 100 kiloby kilobytes per second up and down increase um so yeah this there is some amount of spiky behavior in this so it depends if there's suddenly a lot of transactions that get added to the mempool then you have to go fetch them from your network and it might be more than 100 KOB bytes but it's on average at least and there is some increased read write um pressure on the dis because there's a presence of blobs as well there's a new piece of the puzzle that gets added in right um and one of the things we were really glad to have

tested in our extremely surprised about was that test on arm machine show that they were great uh we had some help with the ethereum on arm folks they uh they're here this week um and it looks like they're able to track Dev Nets perfectly fine U I don't see any reason why they wouldn't be able to do public TK Nets as well um it shows that at least when it comes to tracking the chain there's no added latency that and it could have been that it was worse than we had expected purely because we're adding a lot of new libraries into 4844 so it could have been that one of them was not optimized for arm machines but it seems to work beautifully which is awesome and yeah all of these values kind of still depend on um hard to model

transaction pool Behavior so we we will be so as much as we try to do Dev Nets and Shadow Fox Etc it's almost never going to represent how main net looks like main net has th 10,000 nodes 10,000 plus nodes and we're not going to be running a network with 10,000 plus nodes so it's going to be hard to nail down exactly the metrics you need for such a network so the only place we will be able to get them is on networks like gly hsky Etc which is why we're looking forward to um having tools that have mentioned earlier Leu which would gain a lot of deep insight into how the network looks and how that's going to go and what next um an open call to ask people to get involved in testing um so

syn a node we have devet L live now uh most of the guides that you would have used for previous test Nets would still work um we should have devet 12 up once once we're all back home I guess uh submit blobs it I think a lot of the ecosystem hasn't necessarily started using blobs for testing yet and it would be nice to start um so that once the upgrade does go live on mainnet you're actually seeing stuff and break things uh one things we did notice in merge testing was that when regular people start getting involved in syncing stuff we tend to find a lot of behaviors that we didn't find otherwise behaviors that we couldn't have modeled otherwise so please try syncing nodes submitting

blobs Etc and then raising issues in case you're finding them most client teams have started merging uh their their uh Denon code into their Master branches so you could potentially even just use the latest release of the latest Master version to join one of these Dev Nets so that's uh mostly it and thank you um help testing The Surge and you can find all the tools I've list linked and everything under the e panda Ops organization on GitHub thanks thank you Perry we do we do have uh time for a few questions

here hey one question I have is what is the biggest difference like what are the differences between test net workloads and Main net workloads what are the biggest differences you've observed like one testing on GLE or hold SK uh let's let's take GLE and versus on Main net like so what are the biggest differences You observe between testing on M so one of the really big ones and one that took a while for us also to figure out how to test is that 4844 and Denon is largely a peer-to-peer focused upgrade um and that's really hard to model we needed a lot of tools to be able to actually view it there was a lot of things that you still can't model um so that's one of the major things and

people tend to use a lot of eso esoteric tools that we might not even know about you might have staking setups that use Windows machines we don't test on Windows you might have um vouch running you might have DVT setups you might have a lot of these different things that we just don't have the bandwidth to realistically test so we test the default which is you have an El CL on one machine has pretty perfect uh setup and public test Nets show us that that isn't the case and that's that's what we hope to learn from them I have a question yeah it seems like a lot of these tools would be perfect for an attacker to just like

take off the shelf and use or take off the shelf and modify a little bit and use to attack main net how do you think about the relationship between what you build to defend the network versus like it also being an open tool to attack the network I think very often before we fully open source the tools uh we also try to attack or think about how to attack the network and then make sure we're at least mostly resilient to it um yeah as always it's impossible to have every case covered but we at least try to go through a bunch of um bunch of things and if if it's something we notice uh is completely horrible if we release it today we also don't release it uh you can try searching for attack net you won't find it online um that's that's like a prime

example we we try to hold back tools that would be too dangerous until we're quite happy that we're not able to find anything with them and most of the uh ways we present our guides Etc is more meant from a researcher Slash from a Observer point of view rather than from a attack you where do you think the most under tested portions of the stack are at the moment o I think a lot of it is still in the unexpected scenarios on the pure topar um it seems like we're just starting to get good at testing the

peerto pure leer we haven't finished that cycle yet um yeah I would say that's that's the main one um I would actually look forward to a simpler for that I don't know every time someone talks about like workers like okay it's a nice box there's a database change you can do like a million Shadow Fox and if it works a million times it's going to Mar the millionth and first time um and same for eof they're extremely confident that a lot of fuzzing takes care of a lot of issues and we tend to treat the evm as a nice black box that's packaged for us and the client have worry about it um but when it comes to 4844 we couldn't do that it's it's a live Network and we do have to test it yeah all right well thank you please

another round of applause for our speaker thank you