From Spec to Sync: Ensuring Upgrade Readiness

Transcript

[music] So today we're going to be talking about how we go from specification to syncing a node. Um I was told then that it should have been spec to mainet but that doesn't have the catchy vibe I guess. Um [music] before we move on uh look up from your phones. There's about a 60cond TLDDR and you can move back to your phones later if you want. Um so I'm going to talk about what an EIP is. I'm gonna talk about um how an EIP author writes the tests. I'm going to talk about what phases tests have and I'm going to talk about um yeah what uh fun tools we've built and how we're going to move on. Uh feel free to go back to your phones if

you want. Uh the rest of the talk will be a bit more verbose I guess. I really like this format by the way. Props to Kev. Uh he tried doing it but he wasn't allowed to do it on stage because their presentation software didn't support video. So I was very happy that this one does. So [laughter] what is an EIP? Uh an EIP is a design doc. Uh I know that the last couple of uh presentations have already gone in depth about what an EIP is, how the numbering system works, etc. But the idea is that um if you were to take the yellow paper, add up all the EIPs that are approved, then what you get is mainet Ethereum. In practice, it's not that simple. there's a lot of other

features involved and there's that's part of the reason why we have the executable specifications today. Um if you're ever in doubt about where to look to understand how Ethereum works today, especially on the execution side, have a look at the executable specifications. Uh a lot of people from that team are here today. Huge props to them. Um find them, speak to them and so on. So every change needs to contain an abstract, a motivation, specification, security considerations and so on. Um this can take a bit of time to actually get the right format because you're not writing it for the average reader.

You're writing it pretty much for for core dev audience. Um and it must also come with tests. This is something we're trying to push more and more. Um the newer EIPs do come with tests but in the past I don't know even I've submitted an EIP that just says update gas limit to XX number which is yeah for the process reason we had to do it but well it it it doesn't actually help anyone to have that right um so we're trying to push more and more to have tests to have a lot of these um to to have more considerations happening before we move to the later stages of the development process.

>> [snorts] >> Um once you once you have the EIP proposed, it goes through a bunch of uh phases in the life cycle. This is a um screenshot I have from a talk that Leland gave a while back. I'm sure it's on YouTube and you can find it. Uh it shows you all the stages that it goes to but at a very very high level. Once an EIP has been accepted, you can decide to propose it for a fork. That's when it gets included in the allcore devs discussion process. And then it gets one of three statuses. It's either considered for inclusion, scheduled for inclusion, or declined for inclusion. Uh declines for inclusion is great for me because that means I don't have to look at it for this fork and it's something it's a problem for the future fork. Uh considered for inclusion and scheduled

for inclusion implies that not only does does the EIP author need to spend time with the testing teams, but also the testing teams have to spend a lot of our time understanding the EIP, the implications and so on. schedule for inclusion is sort of like P 0 for us. This means unless we find something drastic, there's huge burden of proof to remove it, it's going in the next fork. Um, an example of what's scheduled for inclusion for the next fork is block level access lists and EPBS. So unless there's massive new information found, we know that these two forks are going in. So in terms of uh team priorities both the steel team as well as the panda ops team would be allocating ample resource to to to these two EIPs and we

also help with all the other CFI EIP so that's the considered for inclusion ones but they still need to prove something to um there's probably a couple of unknowns that need to be figured out etc before they can move to the schedule for inclusion stage. Now, let's assume we have a bunch of EIPs. They're all scheduled for inclusion. What does the testing process look like at a high level? It's this flowchart. Um, in reality, there's no one formula for how you test a fork. You It It's pretty much case by case basis.

I don't think there's a single fork we've gone into where we hadn't needed a new tool. um there's probably something or the other that had to be invented or developed just purpose made for that fork. But at a high level, these are the phases that we're looking at and these are most of the tools that we've settled on. Um I'm going to go by go one by one and then give you guys an idea of um what we're trying to catch with each stage. So the first one's the spec tests. So uh earlier I told you guys the yellow paper plus the EIP equals mainet. Um today it's the execution specifications and the com the combined tests that come with it uh that would define what mainet

is at least on the execution layer. Um the steel team is responsible for these and we also have another parallel specification for the consensus layer. So the consensus layer specifications are also what defines the consensus layer of Ethereum. Um yeah, this is probably also the most descriptive way to define an EIP. If you're ever in doubt, you should just reach out to the team how you can even think about write think about implementing an EIP before you actually write it just so you get a better idea of what could be important in the EIP.

Um and yeah, tests are derived from both the repos. Uh one of the benefits is you get these test fixtures as an output and the test fixtures are what's consumed by the client teams by the ETH panda ops team by the testing team by hive and a lot of these other tools. So these are like ground zero. This is usually the starting point for when we need to do changes. The next thing we do is let's assume that you have a bunch of these fixtures. You've tried it. It works locally. Um Ethereum values client diversity a lot. We have about six uh execution layer clients, six to seven um uh consensus layer clients and this means there's a lot of options for interop. So there's about 36

combinations that we tested for Fusaka. Uh that number tend seems to be trending to go upwards. Uh and we try to make sure that you can use any combination of execution layer with any combination of beacon node with any combination of validator client. Um and this still doesn't include uh things like SSV, OBL, middleware proxies, etc. The testing matrix is huge and the only way we can keep this sane is by moving the tests to local first and enabling each client developer to test and catch as many bugs as possible on their machine without waiting for us to catch them. And one of the tools we use to do this is Curtosis.

Um so this brings up all the dependencies that you need. It spins spins up all the tooling that you need. It uh can programmatically run tests for you. Everything on your local machine. Uh we've abstracted it all away with Docker. So all you need is the Ktosa CLI as well as Docker. One of the benefits is also if you do find an issue, especially with a specific test, you can just share a YAML config with the other client teams and then helps us coordinate with each other as to what the bug might be, help us reproduce it and so on.

Um yeah, so this allows us to also create a rapid feedback loop. This is something that's been very useful at least in the last year. Um before I would say 2024, the typical way we would handle inter interop is that the ETH pendops team had to get involved. We had to create these devets. Um you'd have to find issues in the devet. If I'm sleeping or someone from my team is uh is on vacation, then tough luck. Uh, and this gives us a different angle to the to the to the approach. It just empowers the devs way more.

Um, just a I guess brief summary of how this looks. This actually isn't even spun up. This is with parallel curtosis running. And the end of this loop is how fast it takes to spin up a DevNet. Uh, the other massive massive tool that we tend to use very often is Hive. Um, this has been out pretty much since the beginning of Ethereum's history. Uh it also uses Docker and it spins up clients. However, the easiest way to think about it is extremely fancy unit tests. Um so we tend to know what the input is going to be and we know what the output is going to be. And Hive helps us assert that the input is equal to the output. Um we also have a uh a GitHub action for this. So if you are a

client dev or you're running a fork of Ethereum, then you can always spin up Hive yourself. However, we have a hosted instance hive. with pandaops.io and this runs uh nightly and all the issues that are found especially diffs from what happened yesterday to today um are posted in the ETH R&D discord and this helps client teams actually see if there's regressions and get a notification about the regressions. Um yeah the bugs can be reproduced and fixed. It's it's invaluable for our debugging life cycle.

Uh high level overview this is how it looks. There's a lot of test suites and it's a huge repo. Have a look at the docs. It does a lot of things but uh it's it's probably a good place to try and understand how a client EIP will function over over its internal life cycle. The next big phase is devets. So devnetss are what are only launched once we have about three ele validate this on assertter. We make sure that hive is passing on these clients and then once these things are true it's useful to start getting the community

involved. It's useful for clients to publicly start joining. There's a lot of nuances with public devets versus local devets that make them a bit more fun to catch bugs in. Um you you can have something as weird as a Cloudflare issue that suddenly leads to certain packets ending up in weirder orders that leads to a certain bug. Right? So it's very very nice for us and um yeah attack scenarios can be tried live. So uh simple examples yesterday we or day before we tried a non-finality test on Fusaka DevNet 3 and this is something that we all the client devs could observe and then we let it non-finalize for about 2 days. We healed it and then

everyone can have their local monitoring setups and try to see how the how the how their individual client is doing during recovery. Um yeah, this is uh usually accompanied with this nice uh front end. It's uh Dora, it's our explorer. Um it helps us visualize how the chain looks, uh how the chain health looks. There's a lot of tools that go built into this. Um and I'll talk about like a few of them later, but at a high level, anything we feel like we can use to probe the network, we put it in Dora.

Another thing uh that happens is we then move to the test net stage and the test net stage are long-term devets. So you might have heard the name Hoodie, Sapoleia, Girly, we had Robston in the past, Rinkabe. So these are test nets that live for a few years. We're trying to get better at like giving a life cycle for a test net, but it it's a very difficult problem in itself. Um we try to have a purpose for each test net. So if you're a validator you should go to ephememory or to hoodie. If you're a DAP you should go to sepoleia. And this gives us sort of a staging scenario for different things that Ethereum is trying

to take care of. And you might imagine that you want some amount of load here. So besides the load that anyone from this room or anyone else might be generating on their staging setups we also have a tool called spam. and uh spam can create these fun scenarios for us that might be interesting. Uh there's also some EVM fuzzing in there. Now we don't actively EVM first test nets but sometimes we do have like shorter bursts of it. Um the other benefit is that test nets since they're running for many years tend to have a larger state. Sipoleia is about a couple hundred GB. Um, this might suck if you're someone who's just trying to launch the test net and join, but it's very useful from a testing

perspective because it helps us actually see how the client would perform on mainet, which has a bigger state. Um, it's it's also getting a bit nuanced for us because test nets initially started as a way for client developers to test their software. However, now it's also a staging environment for all of the ecosystem, which means we can't as liberally break them as we used to. Uh, we're trying to get better at only shipping, let's say, mostly validated software that has a higher bar than it used to in the past and then moving um devets to the chaos style things that we do.

Uh, the other tool in our bucket is shadow fox. So shadow fox are essentially copies of an existing network and then we launch a new beacon chain and there's a fun trick where the execution layer basically just follows whatever chain the consensus layer tells it to. So we create a new chain and then we fork off from the main network. The benefit of this is we can now break this new network while inheriting the state of the original network. So this allows us to actually validate if configs are set up properly. allows us to validate if um if if we have an attack scenario how that's going to work. Um shadow folks have also been very useful for increasing the gas limit because what we've done is we've taken a shadow fork

of mainet we've bloated the state so we made it about two times as big and then we start attacking this two times as big a state. So this can tell us that if we were to increase the gas limit in mainet and we were to leave this for a long duration of time how the performance of mainet is going to degrade over time and then we can proactively start making optimizations figure out what EIPs need to go in uh before we're actually suffering from the consequences of that on mainet. And uh yeah, the last bit is every time there's a fork in an EIP uh it goes through the cycle of um actually oh sorry I think I might have multiple test net sites. Sorry about that. Um yeah

it's easy to join and uh yeah node operators are on test nets as well. Um, another thing that we can do on test nets is because they have a larger state and are running for a while, uh, we can actually do sync tests on them. And this would tell us how easy it is for anyone to join a new network. And in case we're breaking sync in the new network, this is a base primitive of mainnet. And we want to make sure that we never actually break sync on mainnet. And the last step of this entire process is actually scheduuling mainet and shipping on mainet. Uh you might have noticed that for Saka's going through this cycle right now and uh one of the feedback points that we've gotten is that a lot of L2s require some amount of time to implement the changes. Community

needs some amount of time to update their software to validate that the updated software works. So we have a stage release cycle. So you would have a test net release test net fork about a month between each thing and once the last mainet release has gone out we want to keep at least a month before actual mainet happens. So this whole cycle from starting a specification to getting scheduled to tested to um actually deployed on mainet for the first time we're able to do this twice in one year is hasn't been that easy to achieve.

However, we're trying to do a lot more pipelining than we did in the past. So we've already started work on block level access list for example already started work on EPBS and once those testing cycles are more mature then we'd already start with the uh deployment stage of test net one test net 2 mainet and this is a cycle we're hoping more people from this room would get involved with uh help us catch issues earlier help us catch issues uh before they actually become critical and help us get to a safe mainet thank you and if you have questions I'm happy to answer them.

[applause] All right, we have got two minutes. Uh if there is any question, please raise your hand. >> How can I how can I know if my AIP idea is already created? I mean, is there a website where I can check that? >> So, probably the easiest way is to check the EIP GitHub repository. Um, I'd say forecast is getting better if your EIP scheduled for the current fork

or the next fork, but I don't think it's still a generic EIP crawler, so to say. >> Yeah, maybe I can uh add to it. So we do have a website called eip.ethereum.org. Go to that and go to alls. So you can just uh find like add your keyword and try to look into the proposals. Yes. So that is very much available over there in case you want to have a like test whether this kind of proposal is already added or not. >> Yeah. We have another question.

>> We have one more. Okay. Uh what do you think about idea to have something like a shadow fork for the mainet where we do the fork before the mainet fork to test the software and everything and then follow the shadow fork together with the mainet and see how the clients perform. >> Do you mean something that's running longer term or something that we require just for like a week or two before the fork happens? >> Let's say one two weeks probably something like that. Yeah. So this is >> somewhere between the test net fork and the main net fork or in that period.

>> So we this time we have the benefit of bloatnet being around. So we do have this ability. Bloatnet's already running the latest client releases that are meant for mainet. So it is a mainet shadow fork and it's working. Uh once we're all back home from dev connect there's also another shadow fork planned uh with all the client teams. We tend to do one before each test net fork and shadow fork. Um it's I would say semi-useful. If anything, it helps us make sure that there's no config issues. But in itself is it's very fork dependent. Um an example is Fusaka is mostly a networking change. Unless I have about 10,000 nodes to spare, it's difficult to actually test the thing. Uh

whereas for Pectra, it would it was very useful. Uh it didn't matter if it was one node or two node, you could have still caught bugs with it.