Parithosh Jayanthi and Barnabas Busa - EthPandaOps Fusaka Testing Deep Dive

Transcript

[Music] Hey everyone. Uh, just for the record, I'm also with the hips and off team, but yeah, it's part of the EF. So, uh, today I'm going to be talking about Fusaka, what we have done in the past, uh, year, year and a half. uh Fusaka has actually been uh petra b uh which basically means that we wanted to have pure death in the same fork as uh pectra but then it started to grow so big uh that we just didn't see it as a viable option. We decided to take uh the headliner topic which is uh pure death

out into the next fork and that is uh what what we are going to talk about today. So, Fusaka brings a bunch of other EIP improvements also uh scaling the L1 and the DA at the same time. This includes the uh topics that Mario talked about earlier today, raising the gas limit to 60 million. Uh basically limiting the maximum block size to 10 megabytes, ensuring that we can uh scale the L1 uh as efficiently as possible. Uh we introduced the uh name of BPOS's. So this is basically a light fork. We're going to be able to increase or decrease the target and max blob count uh without needing to uh engine uh uh we can do a

hard fork without uh much uh organization from the client teams like we are not changing anything other than the uh blob counts. Then we have a bunch of uh repricing EIPs. Uh it's we we don't have enough time to go into each of these independently and we have a new uh op codes that improve the UX and uh the pre-confirmations are going to be enabled by deterministic proposal to path. So how could we actually manage to ship two forks uh in one year? This was uh uh just because we started to work on pure death very early on and uh we were pipelining uh shipping of pure death

from Kenya last year which is which was uh around May if I recall correctly we had parallel work on Petra and Fusaka. In the meantime, we had uh CLD, one clv from each team uh working on purees pretty much for like the past 12 to 15 months uh continuously. Uh the first uh pureest implementation appeared in June of 2024 and uh without having a dedicated person from each uh client team we would we would have not been able to ship uh two forks in the same year.

From this uh maybe we can talk about peer and uh what is uh pardas. So currently today uh every single validator downloads every single blob and this blob is then going to be served to the whole network. Going forward if you want to scale up the number of blobs. What we have to do is uh while while keeping the network decentralized, we have to make sure that uh each blob can be split up into a smaller section so that small validators can actually still validate the chain proposed blocks while uh while bigger validators will actually uh

take over the load of uh the heavy lifting. So if you have a small stake of like 32 E for example, you're only going to need to custody a a few uh sections of all the blobs, a few columns. This column is a new concept and this enables us to split up the load uh in the whole network. While the larger operators will actually have to have an increase in resource usage. This uh means uh bit more compute, a bit more memory usage and significantly more uh bandwidth usage.

This will enable us to have uh larger blob capacity while uh keeping the network decentralized and more blob capacity will uh enable air to scaling and cheaper transactions for everyone. what is actually under the hood of spectra. So as I mentioned in the previous slide, blobs will be split up into columns. Uh the nodes will have to advertise how many columns they have available. So this is introduced under a new field called the CGC, the custody uh group count. The columns will be then propagated to uh the relevant nodes. So a specific column will only go there if

that specific node needs it. And the columns then also can be reconstructed to save some uh bandwidth. So if you have 64 columns for example, you can just reconstruct any columns in uh in the network. Nodes with over 248 will become the super nodes only. So if you don't hold 6.4 4 million on your home machine, then you probably don't have to worry about this. But uh some uh node operators uh such as uh the LIDO CSM operators will actually have an issue where uh they don't actually hold the funds but they stake on behalf of someone else. So

they're uh resource requirements will be actually increased. uh but yeah we we assume that significant part of the super nodes are expected to be a sophisticated entities that will have the resources to run these heavier nodes. So what actually helped us uh achieve uh testing? Uh we have a bunch of tools. Uh you probably heard about Dora. We had a uh pure desk column custody overview which enabled us to oversee uh which node custody is which column. And this gave us a very quick overview on like just debugging in the network. Like if

we expect that a node is a super node then we we want to make sure that uh the super node will show up in all of these columns and if we have a nonvalidating node then it would show up only in four separate uh columns. This also helped us uh debug peering related issues. We also added uh desk guardian. Uh this was a collaboration with the um >> problems >> problems. Uh they helped us uh build this tool and then we integrated into Dora as a library function. Uh basically the main fields are the uh new earliest available slot and the column custody

here. So this helped us a bunch to debug early issues with different CL teams. Uh what else? We also added uh it config support into Dora. So we can have an expected uh config value on one side and the actual client uh what what the client is reporting. This doesn't seem like a big thing, but actually clients like to just configure their clients however they want instead of whatever the spec is saying. So this really gives us a very easy way to see if they messed up something. This is also working now for the e

and uh sinker. Uh so sinker was uh built by Rafael from our team. Syncer enables us to basically run uh a sync test uh let it be a checkpoint sync or a genesis sync uh for super nodes or uh non-s super nodes and it gives us uh statistics on how long the sync has taken and we run this on uh many times. So for example on our DevNet 3 we have run it over probably 100 times to see how the state growth grows and if we have any syncing issues altogether and uh I will pass it to party.

>> So I'm going to talk about uh what else helped us test it. So another tool was uh spammer. So spammer was a tool we had that we can configure a lot of custom scenarios on and these custom scenarios can be real life transactions. So similar to like uh unis swap transactions or blobs that we expect L2s to submit. Um we also want to talk a bit about why it took so long. Uh so why what was the requirement? Why was the requirement changing that took 15 months to ship uh podas?

Realistically, even though it feels like a six-month fork, it only is a six-month fork because we pipelined it. Um well, the core EIP was implemented relatively quickly. However, we went through multiple rounds of bottlenecks that we found. One of the bottlenecks is that uh it takes about 100 milliseconds to compute proofs for a blob. And this is fine if you have like six blobs, you can afford that. Uh if you have 40 blobs or 72 blobs, which is a theoretical max of the design right now, uh that doesn't work anymore. You have a 4 secondond attestation deadline and you need to take 7 7.2 seconds to even generate the proofs. That just doesn't work. So one of the bigger solutions was to move the

self-p proof computation during the time the transaction is being sent. So your transaction the blob sender will take slightly longer to submit the blobs but in the critical path of the chain we no longer see this problem. Um this took a bunch of time for tooling support for client support RPC support of discussions about how to maintain backward compatibility and so on. So this was a huge topic for us. Um there was also a lot of peering issues which you can expect in a networking um networking change. One of the larger ones was that if you're trying to request columns from an older node that's recently joined the network, you're you tend to ask for something they might not yet have and you don't

know necessarily if they have it or not. So you're unnecessarily penalizing them. Um the solution for this to was to add some information such as earliest slot. So the client can actually say, "Hey, I'm your pure. These are the columns I'm custodying. this is the slot that I have data up until. Uh so don't try to ask me for something outside of this and don't penalize me if you do so. So this helps a little bit with um especially non-fality scenarios because when you have a non-finalizing network, there's a lot of node syncing at the same time and it leads to these weird edge cases that are just very hard to recover from.

Since we've uh deployed this, non-fality scenarios have been quite good actually. And if you're interested like Fusaka DevNet 3 even today is going through a non-finality test but we're I guess so confident about it we're here on stage instead of actually looking at if the test is successful or not. Um how much can we even scale right so the next big question is we have pas we know the theoretical max of the system we've found a couple of bottlenecks but what is the actual scaling that we're going to ship with? Were we going to go from six blobs to 72 blobs immediately at Fusaka? That was the original idea.

Um, but the longer we looked at the problem, the more unsafe that felt. Um, that's why we introduced the concept of a block blob parameter only fork. Uh, this essentially says don't change anything about the fork, just change the blob number. And so far we've defined about two BPOS and we have a plan to define another three. And this allows us to do it stepwise. So we can do minor changes, deploy a networking change, probe the network, see how it reacts and then do a bigger change. The reason this is also important is as much as we can define devets, we can do thousand node 10,000 node devets, the problem is mainet is always deployed differently.

However, a certain staker chooses to deploy their node will go into the peering logic and there's no way of accurately simulating that at least not today. And that's why we wanted to go with the safer approach. Um, and the next question is what is the right number? Right? We chose we chose that we want to go to a safer approach. We chose that we want to split it up into steps. But someone has to still find out the step and find out how to find out the step. Um, and this is where uh the Zatu tool from our team comes comes very handy. Um, there's a concept of a Zatu sentry and this connects to the beacon API. There's also a concept of uh client forks that we maintain and this is allowing us to get very deep

visualizations into the peer-to-peer layer. We also have um contributor which if you're a homestaker you should be running. Uh this will collect as little information as we can we we want from you that and this will tell us how timing variances work in mainet and all of this will go into uh analyses that uh give us insights into what our blob scaling numbers could look like. We also want to give a shout out to Sunnyside Labs. Uh we did really large networks probably the largest we've done before.

We did 2,000 node networks alongside with them. uh for a scale that is about 10 to 20% the size of mainet. So that's probably the largest we've done in a while. We had to do a ton of optimizations to even get there without everything just out of memory. Um and this leads to this beautiful blog post done by um Sam. If you ever have doubts about how we reach our scaling targets, please talk to him about how he's analyzing the data. Um you can see a clear trend where um the the blob count looks quite good in at the lower we go and the higher we go it looks iffier and that's why we aren't immediately announcing a jump from 6 to

72 rather the next two BPOS go all the way to 21 where even from the time where this graph was taken we have a lot of optimizations already put through um so we don't imagine any issues immediately and then there's another whole round of optimization that will get shipped um like partial cell proofs. There's um a lot of peer-to-peer optimizations that go in that make us feel more confident that we can then schedule the next branch of BPOS's shortly after. Uh there's also the Syncor tool and this was very important for us to actually figure out hey if we go to 72 blobs and we have a homestaker with 50 Mbps internet are we going to make them take 4 days to sync? Um so this sort of

helped us try out full sync um super uh sorry yeah super node sync as well as um varied number of blob targets and non-fality and then throw all of this into nice dashboards and figure out okay this looks like a reasonable target for now. Um, and in general the phase is you have a testing phase. You do these 2,000 node dev nets. You have an estimation phase where you collect all of this information. You look at it. You then schedule a fork. You enjoy the scaling for a bit before people complain that uh scaling is saturated. Then you optimize and repeat. And this is the loop we want to introduce for BPOS in the future. And it's something that's going to keep happening over the next years.

And that's uh our talk. and we're happy to answer any questions you have about um Fusaka. Thank you. [Applause] >> For the uh super nodes, >> yes, >> how do you actually calculate that there's 2048 ETH on a machine? Is it like keys in a certain validator client? So, if you ran multiple clients, could you stay below that threshold? So when the validator spins up, it registers itself with the beacon node and then based on these registrations, you would know how much stake you're custodying and then that goes into the beacon nodes information of how many

columns it needs to custody. So you could take your one validator client and then spread it across, let's say, 20 different machines, each of them just touching one beacon node, and then you would have lower resource use. But the argument is if you have 20 48 ETH, it's probably better for you to have it pointed at one beacon node or a few for redundancy rather than kind of chunking it into smaller ones. >> Yeah, sounds good. Thanks. >> Yeah. >> And it's based on the effective balance, not the validator count. So it doesn't matter if you have one validator.

>> Yeah. >> With 200 or you know >> Yeah. From the perspective of the protocol, we just care about how much ETH is stake, not necessarily the validator keys themselves. >> And this uh value only goes up. So we made this change uh later on like around July uh this year where the CGC value of a beacon node will only go up. So that even if you start removing validator keys uh the the beacon node will not decrease its uh CGC value. This was an optimization we decided to go with it. Uh in case you just want to reset your CGC value, all you will have to do is remove your database uh start fresh checkpoint sync and then uh point uh you know one less validator

to the to that big node. >> Okay. Uh I honestly have a ton of questions. I don't know which one to select. Uh let's keep uh at this topic of a super node. >> Yeah. >> So we have uh two things. We have uh this peer das where we are saying that 200 480 is a super node. At the same time we have people saying a you should consolidate. >> Yes. >> Which means that with the consolidations 248 is just one validator.

>> Yes. >> And it's already a super node. >> Yes. >> Exactly. But that's why from the perspective of the uh of the uh protocol, it doesn't matter if you had 200 validators or whatever the number works out to um split with 32 E each or you had one validator with all of that stake. >> Yeah, that's understandable. >> Yeah. >> Okay. Now to BPO schedules. So you said that two are already scheduled. >> Y >> um does it mean that they are now inevitable?

>> No. Uh so that's also why we have a bigger gap between the BPOS's so that we can actually look at the network analyze if the performance is degraded if it has if we have to ship any optimizations and we still have the option to cancel the BPO we just reschedule it. >> Yeah. So the question that scares me more is not network immediately degrading after a BO but BO coming out we us not seeing a real increase in demand. So if you raise limits but the actual consumption is never raised and then you limit it raise limits one more time you've basically never tried the thirst limits.

>> Exactly. So the solution there is for us to pay a bunch of money to make sure we're hitting the limits at least for shorter durations of time so we can probe the network at the limit. >> Thank you very much for that. >> Blobs are surprisingly cheap so we can afford to saturate them >> when it's important. Now as a CSM validator on my almost super node your test going to cost me a bit of a gigabytes on my drive. >> Okay. Thank you. >> Yeah. >> Thank you. Uh just just one last question here because we already had time for it but uh I don't think your talk specifically covered it but let me ask you which is um Fusaka is scheduled I believe for December 3rd. Is that correct? Yes.

>> And a lot of the conversations both yesterday and here is it is it definitive? Is it going to happen? Is there anything that's going to stop it or is it December 3rd? >> So, the client releases are already out. The blog post has been shipped. So, unless we find a mainet breaking incident in the next couple of weeks, it's scheduled for December 3rd. >> Okay. Perfect. Perfect. >> As well as the BPOS like as we mentioned, two BPOS are already scheduled. So, the first BO will be happening on the 9th of December and the second one is I believe around the 7th of January.

15th >> 15th of June. >> Yeah, but that's why please update your nodes. Uh the moment you notice anything funky, um let us know. We will be probing the network, making sure everything looks good. But the more the eyes, the better. >> Barnabas and Perry, thank you very much for an excellent presentation.