RPC Fleet Mode - Lightweight Reactive Scaling with Besu - Gary Schulte - Besu

Transcript

Uh, it's going to be hard to follow uh, Marius energy there. I didn't know that this was a a comedy venue. I would have changed my presentation entirely had I known that this is going to be that kind of event. Oh, hey, here we go. Um, uh, mayo Gary Schulty uh, Ybaho consensus and BESU execution client uh, Saluros ETH client summit. Uh that's about all my Spanish. Let's go with that. Um so uh I have a presentation for you on uh reactive scaling RPC uh with uh Basu fleet mode. When I ran through this yesterday, I was coming in a bit over 20 minutes. So I'm going to kind of jump

into this and I might skip over or kind of drop some detail here so that I can get in under the 20-minute wire. Uh and with that said, uh let's go. Um, so, uh, why might you want to reactively scale RPC? I mean, you're probably an RPC provider if you want to reactively scale your services. Uh, you probably already know in that case that, uh, blockchain traffic is kind of peaky. You end up with airdrops and memecoins and liquidity farming and, you know, volatility driven, uh, DeFi liquidations, all these things that kind of create this um, you know, really peaky, clumpy uh, traffic that you need to to service. So an infrastructure provider is going to want to scale up to

meet that demand so they can provide good quality of service. Uh but also you're going to want to scale down in the lulls so you're not overprovisioned and you know wasting resources and wasting money. Um unfortunately Ethereum doesn't scale all that well in that way. I mean an Ethereum node typically brings along with it terabytes of data. Uh and if you're into react to scaling you know cloning terabytes of data is not exactly a fast thing to do. Uh so enter basu fleet mode. Uh basu fleet mode was introduced in December 2024 almost a year ago in a blog post by uh Emily from Lineia and Kareem an Amizian from the Basu team. Uh they did a great job on that blog post kind of going into detail

uh talking about the history uh the development uh kind of the benefits and the trade-offs. Um, I definitely highly recommend checking out uh that blog post if you're interested in reactive scaling. Uh, they did a great job, but you know, in the interest of time, uh, and so that we can kind of look at the results of the linear pilot, uh, I'm just going to kind of, uh, speedrun some of the features. Um, so let's do a quick recap of, you know, what, uh, about Basu makes this fleet mode feature possible. Um as always the bonsai storage format kind of features prominently um in that it provides uh a a flat state database. Uh that flat state database sits next to

your state tree uh and it gives you kind of direct access uh to read and write state without having to walk the state tree. So you end up with really fast access. Um additionally we uh Bonsai provides uh trilogs. Uh so triologs if you're not aware they're they're basically like a um kind of a flattened block witness. If anything is touched, deployed, mutated account storage accounts themselves, uh they're going to be present in a trial log. Um in fact, you know, you get a pre-image and a post image uh of the block inside the trial log. Um that triilog um is essentially how we roll the basu state back and forth. the the Bonsu Bonsai uh state

there's only one state that sits on disk. Uh so we roll that back and forth with those trial log diffs. Incidentally, uh the next feature that we're leveraging from Basu uh Marius alluded to the partial state history expiration. Uh we have a blockchain pruner. Uh it's configurable to either prune up to a certain height or prune a certain distance behind head. If you want to stay in protocol with dev P2P, and we all want to stay in protocol, right? Uh you don't prune past uh the merge. I mean, unless you have a good reason to. Um and lastly, the the Basu plug-in API kind of makes this possible.

Uh we've been uh deconstructing the Basu codebase for a while now and pulling out non-mainet features and putting them into plugins and um kind of keeping our codebase really Ethereum core focused. Uh, so as we add new features that aren't core and as we pull old features out into plugins, this plug-in API is kind of getting richer and richer and allows you to extend the client kind of any way not any way you want, but in a number of ways uh to at runtime load a plugin and alter the behavior of the client. Um, so speaking of plugins, uh, Fleet Mode is a plugin um, and it actually required very little changes uh to our plug-in API in order to deliver its functionality.

speedr runninging what those are. Um, fleet mode uh adds some new RPC endpoints. That's kind of bread and butter for the plug-in API to extend Basu with new RPC. Um, it introduces a new custom fleet mode sync behavior. We had to add like our synchronization pipeline hooks into the plug-in API in order to accomplish that. Um, it uh exposes u state tree truncation. That's something that we had in Basu prior to fleet mode, but it wasn't something that was exposed in the plug-in API. And then lastly, we added uh a trial log eventing system. So whenever um you import a new trial log or execute a block, uh you can be notified of it and you kind of react

accordingly. Um so what makes fleet mode fleet? Um our lovable uh Ethereum node over here with all of its state and blockchain. Um what we do to make fleet node fleet the first thing is we drop state. Um if we restrict the fleet mode uh use cases to just read and write of state we can get by uh with just oh is this a laser pointer? Can we use that? No we can get by with just the flat state. Um so we get we realize a lot of savings. I mean the the state tree is an order of magnitude larger than the flat state. So we get like essentially 80% of the storage of state uh back by dropping the

state tree. Um in addition we aggressively prune the blockchain. Uh I think we just keep a handful of epochs of blockchain data behind head and aggressively prune the blockchain. The blockchain is like the biggest consumer of storage on an Ethereum node. So to to aggressively prune that you're getting hundreds of gigabytes back. Um, so yeah, a a lightweight node that has uh very little blockchain data and no state tree. Uh, it's as lean as it can be and still solve the use case for RPC. Uh, in fact, it's so lean it can fit into system memory.

Okay, so um what is fleet mode? What does it provide? Uh, it's fleet mode essentially defines a couple of new roles for an Ethereum node. It defines this role called a captain node. a captain role. Um, and a captain is basically just a regular Ethereum node uh with a few new RPC endpoints. It still has uh a full state tree. It still has all the blockchain data. It's participating in P2P. It's directed by a consensus client. It's what you would expect an Ethereum node to be. Um, but these new RPC endpoints are just some additional duties that it needs to take care of. uh mainly uh it provides an RPC endpoint to kind of register observers of its state. Whenever it executes a

block or imports a new block, uh it's going to push out to its followers via RPC um these new heads and these new trial logs, this new state that's occurred. Um, and also there's another RPC endpoint that allows you to kind of pull these trial logs via RPC uh for arbitrary block heights, block hashes, block numbers. So if you have an observer that gets this new head and it doesn't know about the intervening state between this new head and where its database is sitting, it can go and query its captain and say, "Hey, give me the give me the block data. Give me the the state data. U give me all the intervening changes."

Um, so it's basically a captain is just a regular Ethereum node. Um, the followers though, the followers are where the magic starts to happen. Um, a follower node implements that state database format that I was talking about earlier. It's not participating in P2P. It's not uh participating in consensus. It's not directed by a consensus layer, I should say. Um, it's designed to just serve RPC traffic. So, like I mentioned, it doesn't have a state tree. It just has the flat state database. Uh but with that flat state you can still execute ETH call you can you know do debug trace block uh do ETH get balance pretty much the entire ETH RPC namespace with one notable exception of u ETH get proof if you don't have a state tree you can't

provide a Merkel proof you can't calculate a new state route um that's just a trade-off that you're making with fleet mode but if you're an infrastructure provider uh you already have typically different node types in your setup you've got archive nodes for historical queries and you've got near head nodes that are serving that kind of hot dynamic traffic. Um so if you're making that trade-off, you just kind of route any ETH get proof over to an archive node. Uh and problem solved. Um additionally, like I said, we're pruning that blockchain history, gaining all of that storage back. um a um a a pruned blockchain node and a uh dropped state tree, it results in a

substantially smaller state. Um I think when I last checked a mainet Ethereum node RPC follower was about 80 gigabytes. Um and a linear mainet follower node was about 20 gigabytes. So uh in a typical RPC node in an infrastructure provider that that fits in memory pretty easily. Uh so you get the benefits of you know much much faster state access. Um so a follower node also is subscribing to its captains for state. Like I said it doesn't have a consensus layer. It's not executing these blocks. It is okay five minutes.

I'll speed it up. [laughter] Uh yeah. Um so it's not executing those blocks. It's just applying those diffs and moving quickly through the chain. um it's quering trial logs and the captains to sync and um yeah it's it's devoting all of its resources to RPC uh because it's not really doing the typical duties. There's no um you know transaction pool it's maintaining etc. Okay so operationally how do you get a stack of these together that's going to allow you to reactively scale? Well, first you've got to start with a captain node that is a typical Ethereum node.

You need to sync it. You need to apply the the plug-in. We found in practice that a captain node can serve uh is about 300 follower nodes without really affecting the uh the speed that it's importing blocks or the speed that it's disseminating those new heads. Um when you get a follower node though um you need to sync that follower node from somewhere. Um you can do that with dev p2p like normally and then drop the state or you could query the captain and get all the trial logs if they have all the trial logs since genesis. Neither of those are particularly reactive. Uh they're kind of slow. So the fastest way to spin up a follower node is to clone the database from another follower. Uh Lineia actually defined these source of truth follower nodes that their only job

is to follow the chain head and then infrequently snapshot their database. So when you spin up a follower node with a snapshot, uh you then just sync up to head with a captain and since you're just applying trial logs, it's super quick to sync up to head. Uh the last element of the stack is an intelligent load balancer. You're looking at those queries as they come in. Make sure they're routed to an archive node if you need archive data and if you're near head hot traffic, route them to follower nodes. Okay, I'm going to skip past this since we are running out of time. And um let's talk about an interesting moment.

Uh the 30 days surrounding the linear token generation event. Uh you can kind of see before the uh token generation event, we have this sort of um you know kind of baseline looking traffic. You know, we're coming in around 15,000 requests per minute. uh we end up with this little spike here uh in early September. Another big spike here on the token generation day. Uh this is all if you notice this is all ETH call dominated uh all the traffic here. I mean there's other RPC services but ETH call is really uh dominant in that uh chart. Uh on the token generation day we get all of our uh linear users claiming their airdrops and we get this new kind of higher baseline of traffic uh and then one more spike there on the 15th.

So the way this scaled in production um you'll notice that these follower nodes uh they're scaling on kind of a sawtooth pattern and that's because they're using a continuous deployment strategy uh to kind of deploy this baseline level and then uh in on a schedule they're scaled up to kind of meet the demand that they're anticipating and then ahead of the token generation event they kind of took those downscaler jobs out and uh we have this static deployment and then uh when we hit this big traffic spike on TGE day we deployed a lot more uh follower nodes. Um the uh load balancers, they actually kind of more um reflect the demand. They're kind of reacting just to the spikes like the one

we saw on the 30th, uh the spike we saw on TGE day and the spike we saw on uh the 15th. That's because they're using the Kubernetes autoscaling instead of using a continuous deployment strategy and kind of on a schedule uh reactively scaling. uh talking to the line ops, they're actually working on uh having these scale reactively with Kubernetes. I think there's a little uh infrastructure work that needs to happen before that can work. Um but yeah, it's um it works really well. Uh they scale up and down really quickly. Um we had some um interesting learnings in the pilot and that is um because we're running short on time, I'm just going to go quickly. Uh the ex the ability to load the the database in

memory that was kind of discovered once we deployed the pilot. We didn't anticipate that. We knew we'd be able to uh scale quickly and we'd get better access, but we didn't expect expect to be able to fit all of that in memory. Um, we found that we had a lot of pain points with our plug-in packaging and configuration management. We added some uh build tooling, uh, some prescriptive uh, plugins to allow us to uh, make that a little easier. Um, we also found some RPC performance edge cases. I think the most notable is like really high gas ETH call usage. Um, and that kind of gave us great feedback. uh so that we can go through and and fix that. Uh we're working on that at the moment actually.

Um so the big reveal is uh during the pilot the plugin was closed source uh and as of today uh the plugin is open source. So hey [applause] if you want to uh clone the repo, give it a try um set up your own reactive stack, you know, give us a call. Uh, check out the blog post for sure because that gives great context. Uh, but contact us on Discord, on email. That slide's missing for some reason. Uh, there we go. Yeah. Uh, but again, mayo Gary Schulty, if you have any questions, I can answer.

>> Excellent, Gary. [applause] >> I'll have you go to the center of the stage just for a second. I We We do have a few questions in there, and I think I'm going to sneak them in here if you bear with me here. Um so for the first one can you share anything about space requirements and in particular what is the expected growth database size of the fleet node over time? >> Um well it's really going to depend on the network you're you're talking about like mainet we we notice is um roughly a tenth the flat database is roughly a tenth of the size and that's pretty much where all of this the uh the storage scaling is going to come from. So if you think about uh if your network has large state growth, you might end up with

maybe a 10% uh of that in your your fleet node database. So like a mainet state is coming in around 800 gigabytes. Um a a mainet fleets about 80. >> Okay. And what is the time to sync? >> Um when you are using a snapshot strategy, it's going to be you know commensurate on how long how old that snapshot is. uh but we find you know it takes less than five minutes to get any any snapshot up to head. >> Okay. And then one last uh maybe quick question uh can you talk a bit more about the flat state?

>> Yeah. Um so the flat state is really it's kind of a if you look at the Merkel tree state um the information you're looking for is actually in the leaves of the tree. So when you're walking the tree, you walk down the the state tree to get to the leaf data that you're interested in reading or writing. U if you have a flattened state, it's just the leaf data. You can access it directly by key, write it directly by key, and then worry about updating your state tree afterwards. >> Excellent. So again, a round of applause here for Gary. [applause]