P2P Networking Call #1

Transcript

Kamil Salakhiev: Whoa? Let's give maybe 2 minutes for people to join. Yeah, I rolled. Daniel?

Raúl Kripalani: Nope.

Kamil Salakhiev: Okay, Mark, what kind of change? Yeah, right, I think, yeah, let's just, get started and, yeah, see how it shows. Yeah, I… so… first of all, like, this is now, like, breakout calls, yeah, more or less structure, I think, will be the same, but, yeah, now it's a bit easier to kind of track everything and, yeah, bring up topics, on the GitHub. Also, all recordings will be available on YouTube, I believe, and, yeah, on Forecast. So yeah, but I think the time is the same. I also… just in case anyone still uses the old calendar invites, yeah, I also updated the link… them with the new Zoom links, so that, yeah, I hope we don't…

have any mess there. Yeah, and with that, I think we can get started. Yeah, maybe start with… That's the last time with, client updates, yeah, maybe with, lighthouse? Daniel, are you here?

Daniel Knopik: Yeah, yes, sure. On partial messages, there's still work in progress on Gloas support. It's almost done, just pending some… Review, self-review, and then peer review. I don't expect this to be merged that soon, but it's, yeah, ongoing. We also, ran some nodes on Hoodi, with partial messages, it's… As expected, pretty unexciting, because now they have, like, 5 nodes supporting it in the whole network. They are appearing with each other, but That doesn't really make an impact, but the few things that… are, you know, meaningful, like, how many of those cells being sent are useful, look pretty good, so I'm happy with the findings for now.

We also thought a bit about roll… like, bigger rollout or release plan, but we can talk about this after the updates, I guess, maybe? Or however it fits in. Besides that, Joao isn't here, right? Okay, there's… some internal refactor in rust-libp2p, basically switching back to a different… Stack of, protobuf deserialization. But yeah, nothing, nothing too relevant, for the… For, for everyone. Yeah. I think that is mostly it, yeah.

Kamil Salakhiev: Hmm. What is this protobast deserialization issue? Is that… I think we discussed that, with goalie P2P, it was, like, some… yeah, basically, a member of… Yeah. Messages are more…

Raúl Kripalani: It's something… yeah, it's… it's something related to the payload, yeah, it just doesn't… it's not worth going into it.

Kamil Salakhiev: Okay, okay. They're just goops. Fine. Alright, good, thanks, I don't know, like, do we have anyone from Prysm, or… Okay, no one, yeah, any other CL clients? Yeah, if not, yeah, we can just go into EL part, yeah, I see Bosul, so, yeah, how things are sparse blob pool, yeah, we've been discussing last time, yeah, with this, get both V3 performance, I believe, like, yeah, you said that we might do some, CPU profiling, maybe, and yeah, see how… Exactly the time spent there, yeah. Any news there?

Bosul Mun: Aye. about GetBlobs V3. I… I couldn't take care much. about that problem, because I was working… also on sparse blob pool, and its 71 is also going on, so… But I will invest… I really want to investigate it. But for now, It's making sparse blob pool more solid. The implementation is my more priority. being more prioritized, I'm sorry.

Kamil Salakhiev: Yeah, yeah.

Bosul Mun: I also wanted to ask about the timeline of That's what I love about Delta. Like, how urgent is this issue? Is there any date deadline? that I should solve this problem until.

Marco Munizaga: I… I can speak a little bit, So, I think this is basically the last thing before we… at least before, I would feel comfortable deploying it onto mainnet. Like, the change itself is backwards compatible.

Bosul Mun: Yes.

Marco Munizaga: But, like, this extra latency with get blobs, and I think this is something we see in all the Get Blobs versions. This is just, like, I think the last thing? I'm…

Bosul Mun: Yes?

Marco Munizaga: Yeah. Hopefully that helps.

Bosul Mun: And… Hmm, and also… when I saw the profile, it's mostly about JSON, it's mostly about encoding, time. So, the… delay. So… but right now, in… in Mainnet, it's, the blob utilization is not very… super high, like, in the cell-level data.net, so… I think we can delay… No? I'm, I'm sorry.

Marco Munizaga: So, I'm not convinced that it is just the JSON encoding, because if it were that, we should be able to see that in a localized benchmark. I think there's something more subtle going on there.

Raúl Kripalani: Yeah. So, everywhere.

Bosul Mun: Hmm.

Raúl Kripalani: Yeah. We were… we were going into the… I don't know if you've seen Discord, Bosul, I tagged you in… in a thread. I posted the… the findings here. I think what's happening is that, Geth and just like many other Go clients, are using KB stores, that are, usually SMT that have… Where the keys are stored in index and memory index, but then the values are actually stored on disk, so the way that the databases operate is usually they will return you… when you fetch a value, they will return you a map slice. So really, the moment… the query, the latency of the query to the database is not indicative of the retrieval, of the latency of retrieval of that value, because really, that value is going to be consumed when you actually read the slice, right? Which is when the data is going to be paged from disk into memory, and then fed into the process. So this, I think this is what's happening.

it's a red herring. I really suspect it's a red herring, what we're seeing here, in the sense that… it is occluding what's happening behind the scenes. So, the traces from Tempo are actually stopping at, I think at the serialization, right? And you don't really see what's going on under the hood. And I think the only way you'd actually be able to see it is with a CPU trace, with a CPU profile. So the idea here is we… I asked Barnabas on that On that disk, I don't know if it's gonna happen and when, but the best way to debug this and to get to the bottom of it is precisely doing that, just running a CPU profile.

Okay, and this would.

Bosul Mun: Let me see.

Raúl Kripalani: high variability we see in the response times with the same blob count, right? Because the cash situation is gonna be different for… is highly, highly volatile.

Bosul Mun: I see, I see. Thank you. Huh? Yes.

Raúl Kripalani: I think the next best step here is to ask them to ask Eatman ops, or whoever is running the DevNets, to, to, do a CPU profile while we do a load test.

Bosul Mun: Yes.

Marco Munizaga: I think, yeah, I think I'll try to get something similar set up, try to get an environment where I can connect to the DevNet and reproduce these issues on a machine I control, and then I could really dig deep on… On the exact timings here. Yeah, I should be able to get that done this week and share some insight there. Thank you. Yeah, I think this is probably an issue that probably affects both get blobs V2 and V4. But V4… Can be a bit more efficient on what it returns.

Bosul Mun: This is good.

Raúl Kripalani: I think this…

Daniel Knopik: Nope.

Raúl Kripalani: Generally affects anything that… that retrieves… That retrieves, that retrieves data from stores, but specifically data that, can be sitting on disk. I think… Yeah, state trees are less likely because they tend to be hot, but blobs definitely are flush. Quicker, because they are larger slabs, so they flush quicker to disk.

Marco Munizaga: Also, a side note, that if this is an MMAP, slice, We may want to re-evaluate our use of MMAP in that case. And I can go into more details later about why that is, but basically, it blocks the go routine in a way that is, like, a hard block rather than a… Yield to out of their routines.

Raúl Kripalani: But just to be clear, this is just a store that has been adopted. Is it Pebble, Bossel? Or what is Geth running on these days, Pebble or BroxDB? Used to be a rockstadian, right?

Bosul Mun: To install… to store gloves.

Raúl Kripalani: Yep.

Bosul Mun: is, Billy.

Raúl Kripalani: Billy. Oh, it's a different one, okay. I know we… there's been a few over time. Billy. Well, we can… we can research that, we'll look into the code. It should be fairly… fairly simple to know what they're using under the hood, but it's a fairly common practice in Go databases. Like, Badger, you look at Badger, Pebble, this is the way that databases… that KB stores work, and specifically LSM bad… LSM stores, LSM trees.

Csaba Kiraly: Billy is a very simple flat structure.

Bosul Mun: Yes.

Csaba Kiraly: So… but I think it's memory map, actually, I seem to remember.

Raúl Kripalani: Should be a memory map. Yeah.

Csaba Kiraly: So that, that might be…

Raúl Kripalani: I would assume that it would play well with the go-round time. That's because it's such a common thing to do.

Csaba Kiraly: lawsuit.

Raúl Kripalani: But definitely.

Csaba Kiraly: worth looking into.

Kamil Salakhiev: Yeah, alright. Also, I think you also were mentioning that Nethermind, I think they had some issues with the sparse blob pool, I think, like, treating, like, this bitmap, incorrectly because of, like, endianness, something like that. Doing it… yeah. Do we need to.

Bosul Mun: So.

Kamil Salakhiev: anything, I mean, I don't know, EAP or anything, like, to maybe make it more clear, Yeah, what do you think?

Bosul Mun: How about… First bottle, update, recently, Camille's, PR is merged. I reviewed it, and I think it's better, it's more clear, and… So, please take… refresh the… take a look, and… about the… that issue, the problem was that we didn't really, specify which Encoding this, Feed map will have So I… I also have a change for that?

Kamil Salakhiev: Yeah, I was curious, like, whether we need to specify that, or it's, like, implicitly maybe… Already kind of use the… well, we kind of use everywhere, like, the same. NDNS, around the spec, around the EAPs, and maybe Engine API. I would, like… Is it worth clarifying or not? Just…

Bosul Mun: I think we should specify it. in Engine API, and also this, VIP.

Kamil Salakhiev: VP. Yes. Hmm.

Bosul Mun: I… I will open a PR, but if I open the PR, it will be merged automatically, so…

Kamil Salakhiev: Hmm.

Bosul Mun: It'll look like this.

Kamil Salakhiev: Okay. So, basically, yeah, the issue was that, yeah, Nintend was having this engine API implemented incorrectly, because they were treating, like, bits, not in the right order, And… yeah, now I believe.

Bosul Mun: And also adopted a few, yes.

Kamil Salakhiev: And also the P2P, yeah. Okay.

Bosul Mun: Hmm. This is…

Kamil Salakhiev: messages.

Bosul Mun: Very simple clarification.

Kamil Salakhiev: Yeah, okay.

Bosul Mun: about the… Abnad. I planned… we planned to launch this, and… Next, glamsterdam-devnet, which was glamsterdam-devnet 4, I… if I remember correctly, but the problem is that they don't have blocked transactions on their there were some problems with sending global transactions to Glamsterdam devnet, so… I… I don't think it makes… much sense to deploy a sparse blob pool on there, because there's… there will be nothing to be tested. So, the… So, so, the plan is to… I think the… Plan is to first, per step.

contacted Lighthouse, Because they are participating on WAL.net. So… about implementing the cat gloves before, and… And also updated the Herp Trace update. Engine API. if that takes too long, then we can also have a, like, NFT.vnet, like we did for It70. Yes. So…

Kamil Salakhiev: Yeah, I think… I don't know, Daniel, can we ask you, like, do you know, like, if your team has capacity to implement these updates, like, get blocks before and… Yeah, for questions updated.

Daniel Knopik: This is actually on my to-do list, so I hope to get to this soon, but right now, I'm still in the finishing touches of partials, especially partials for Gloas, but once that's done, I'll… I'll start. So that's planned to be started soon.

Kamil Salakhiev: Okay, sounds good. Okay, nice. Yeah, I think, yeah, that's all for the basketball pool. Yeah, any other updates? Yeah, Chaba, maybe anything with Discovery V5 and other things on your side?

Csaba Kiraly: Yeah, so, this V5, this V5. Everyone is claiming to have it. I have some doubts. So… I'm clawing now the mainnet just to see… who's doing what. We still have a few clients. Not up to speed on this V5. So, we just need a bit of… bit of nagging, and then… then we'll have the SP5. Now, Besu has this issue that they don't have the unsupported parallel stack, so they cannot have the Swift and Display 5 at the same time. And, But the solution is easy, at least in the Go codebase, so I hope it will be easy for them as well. Oh, it's Java. It should be. There should be classes, objects, everything, so it should…

It should be okay to live in parallel to stacks. Which we just need for the transition. So, I guess they can do that. We have a problem with less. Which is… Yeah, they… they just sneakily use a different port, so they have basically the same issue. They don't use the same port, they use a plus one, which we don't expose in the analysis, so it's, kind of… no one knows that they are listening there. And there is another mind, which has, Yeah, they just forgot to turn it on, so they have it on testnets, but not on mainnet, so… After that, I think we'll be ready with this V5. Yup. So I think we did that,

I don't see too many blockers, but we have to… Start… Having the testnets Just be 5 only, maybe? So the biologist. you know, we figured out the things that are not clawing. I'm also doing a bunch of clawing of the discovery networks and the network as a whole. Not yet ready to present anything on that, but, just for you to know. I have some new enhancements on top of the collar that I had. Published in 2024, the Fast Team Crawler, so now there's an advanced version which is doing much, much, much, much more. Yeah. That's on the SP5 side. On the still EL networking mempool side, I have a nice tool which is giving you observability.

You can look into the manpool and its evolution. Excellent. More of an explanation, so it's in a… on a… somewhere on a side bench. In gas, plus, an extra tool. This is from my side. See?

Kamil Salakhiev: Yeah, alright, sounds good. Yep, I think that's probably it for… client updates. Yeah, we can just proceed with the general discussions, if anyone has anything to bring up. Yeah, I don't know, like, I don't think, like, Sukun, you have anything. I think you presented, on last ACDC already, the, proposal for the Hegota, for the, yeah, For the attestations improvements. But yeah, if you want to add anything, yeah, feel free to.

Sukun Tarachandani: No, I think, yeah, I've presented it twice now, so thank you.

Kamil Salakhiev: Yeah, yeah, yeah, yeah, yeah, that's fine. And, yeah, yeah, on my side, yeah, yeah, sorry.

Daniel Knopik: Apologies if you want to go in first.

Kamil Salakhiev: No, and I'll just briefly, yeah, just in case anyone is curious, yeah, there was… yeah, I did this, yeah, thanks to AI, some improvements to just how to run simulations and shadows, so that, yeah, if anyone uses that, and you want to use them on your MacBook, so now… Yeah, there is, like… I did a fork which actually supports that, at least it runs on Docker. Which was pretty useful, at least for me, because, yeah, like, the performance was better, so that, yeah, simulations now, yeah. More convenient and easier and faster to run sometimes. Yeah, just, simply… I'm sure that. Yeah, so… If anyone needs that, feel free to use that, that's… was very… I don't know, I'm just using it now, from now on, yeah. So far, I didn't notice any much difference, from… with,

original shadow on IMD. Yeah. Yeah, Daniel, you wanted to add anything?

Daniel Knopik: Yeah, yeah. Back to partial messages. partial columns. Putting the current issue of the latencies aside, Lighthouse's implementation is pretty close ready, it's in our unstable branch, and we intend to do a release in the coming weeks anyway. So… Unless we find, like, we're also heavily testing this, so unless we find any further blockers, we intend to release, Partial message with that. The thing is, so, of course, we won't encourage people to run this on mainnet just yet. The idea would be for people to try it on Hoodi. However, people are slow to update anyway. Especially on testnets. That's even worse there. And having partial messages feature-flagged a way where we would put out a blog post explaining what it does, we would need people to update and figure out this blog post and decide to enable it, and personally, I don't see the adoption rate on Hoodi to be

Large enough for us to gain any meaningful insight from that. So… We actually consider enabling partial columns on Hoodi by default. While keeping it disabled from mainnet by default. I'm not sure how you guys feel about that. If you say, no, no, that's way too risky, or if you have any concerns about that, please let me know.

Marco Munizaga: I'll say that sounds… that sounds great. Do it.

Daniel Knopik: Awesome. Cool. Yeah. yeah, in the next few weeks, it's pretty vague, I know, but I guess sometime in June is kind of realistic. Gotta keep you updated.

Kamil Salakhiev: Do you know, like, what is the ratio? I mean, what is the portion of lighthouse nodes in Hoodi? Like, once you have this enabled by default, so that, How many nodes will be there?

Daniel Knopik: I mean…

Kamil Salakhiev: I'm just curious.

Daniel Knopik: I don't have any stats in mind, I'll need to check if any of the… Scrapers also scrape on Hoodi. Probably someone does. But that being said, it's… it's really not really in the… as I said, people tend to update slowly, especially on testnets. Of course, like, the hosted nodes by… if EthPandaOps shall, would probably be updated rather quickly, and the Lighthouse internal nodes are already running it 100%. I'll just…

Csaba Kiraly: Have the stats.

Daniel Knopik: stats. Yeah, yeah. Oh, you have the stats?

Kamil Salakhiev: No, it's.

Csaba Kiraly: So, we'll see. Searching… what's the question on Lighthouse nodes on Hoodi?

Kamil Salakhiev: Yep.

Csaba Kiraly: Just a moment. My, yeah, I don't see it immediately.

Kamil Salakhiev: Yeah, okay, no problems.

Daniel Knopik: Yo.

Kamil Salakhiev: Nice.

Daniel Knopik: Like, regardless, it will be… it's like…

Csaba Kiraly: 22%.

Daniel Knopik: Of course, it can be toggled off. Anyways, so…

Csaba Kiraly: I don't think so.

Daniel Knopik: I…

Csaba Kiraly: safe.

Daniel Knopik: Don't think this is a risky thing to do.

Csaba Kiraly: Good. Nice.

Kamil Salakhiev: Alright, yeah, other than that, do we have anything else to bring up? Yeah, Raul, anything on your side?

Raúl Kripalani: I'm happy to give maybe some thoughts and, like, a short presentation. I haven't prepared anything, up front, but I do have some material to show on, specifically batching attestations at sorts. Some of you were present already in the discussions in, in Svalbard, so maybe it's redundant for you, but for the ones that weren't, I think this is… this is a useful thing to have. In context, so… Okay, so what I'm gonna do is… I'm just gonna quickly share my screen, if that is okay, and… I just have some material here to show, and maybe to guide the discussion. Give me a sec. It's happening, is it here? Yes, this is the one. Alright. Do you see my board?

Right now. Yes. Okay, sweet. Cool. So I just wanted to talk through the rationale here of why this proposal makes sense, and why it's something that we should, consider for Hegota. So… We're currently at 1 million attesters, we have 64 subnets, and there is… this much. So this is what, basically, the architecture, network architecture looks like. It's a schematic. I'll walk you through this very quickly. What you see here is just a schematic of nodes. These are beacon nodes, and how they are arranged into subnets. There's 64 subnets. And I think it's every 256 epochs, or, like, there is… there was another number, I can't remember. Subnet… beacon nodes rotate which subnets they're actually giving structure to and servicing. So you can think of…

subnets as being the backbone, the circulation, kind of like, you know, the veins through which attestations are circulating in the network. And as we all know, committee… attesters get assigned to one committee per epoch, to… to attest, once every epoch. So, as an attester is, calculating its, look-ahead epox, it is gonna say, I'm gonna, I'm gonna be pushing attestations on this particular subnet forever at whatever slot, and, it will need a way to be able to deliver those attestations to that subnet. Now, what happens in reality is that, we can't have So our testers are gonna be rotating committees every… Every, every epoch? And… we can't… so if attesters were the ones that were creating and giving substance to those subnets, we would be having a lot of volatility in the network, because basically we would be having a network. The backbone of the network would be reshuffling, every 32 slots, and that creates a lot of instability. So the mechanism that we have is we decouple the backbone from the traffic that transits over that backbone. So, the backbone is those 64 subnets.

nodes based on their PRID and based on a bunch of things, that's a deterministic formula. Calculate two subnets that they're gonna be giving structure to, so they're going to be subscribing and moving messages and forwarding messages, does not mean that they're actually going to test. These are beacon nodes. They might not actually be running any validators behind them to begin with. So these are just beacon nodes, they give substance to those subnets, and then testers are sending emitting attestations on those subnets, right? So we have this dual structure. And what happens is that, if you run the math, the chances, of, of… having attestations come from inside the mesh or inside the subnets for a particular committee is tiny. It's just a 3%, right? So, for every given slot, you look at the nodes that would be servicing those subnets, and the nodes that would be attesting, and it's just a 3% overlap. So, in reality, most of the attestations are coming from outside those subnets into the subnet through

what in at least the Go implementation of GoSubSub is called FanOut, this fan-out mechanism. And just to give you some magnitude of, some, some magnitudes here, currently, if you take the 1 million attesters, you divide them into 32, 30, 32 slots, and you divide them into committees, you're gonna get to the point where you have around 515 attestations per subnet per slot, so… 515 attestations per committee. These attestations are tiny. They are 180 or 200 bytes once you compress them. They don't compress too well, because the one… a big part of it is the signature, but the rest does. And there's actually a good compression factor if you're able to, batch multiple attestations at once, because those attestations will be having common fields across them. So…

Just to give you a sense of magnitude here, currently we have for every committee, we would be having around 100K of unique data transiting… transiting over the, over the subnet, for that committee. So starting from that. I just wanted to go back down here. So we've got, a million attesters, and the straw map says that… let me see where I put this… so the Straw Maps has this little box over here, which is 1208K attester cap. Why is that? Can… Do people in the call, like, have an idea of why that could be? To make it a little bit more fun and interactive. message in the chat. Consolidation, yes, yeah, but that's a sec… that's… that's one way to get there. Why would you want to have that? Why would you want to have fewer testers?

Well, I think people are probably not able to answer now, because I'm sure that most of you know the answer. But the idea, the thing is that the network is sized in a particular way. We've got 4 seconds to propagate attestations, it is sized for 64 subnets, for the current volume of attestations, and so on. And the idea, if… What you want to do is, the things that come after the attester… the attester cap are things like… faster slots, and faster finality, right? So that means, or they're actually come together, they're stacked together. So this actually means, if you're kind of, like, naive, you can go and say, okay, if I… if I need to reduce my protocol, if I need to slash my timing in the protocol by, like, 8,

Then I must need to have 8… a factor of 8 less messages, right, in the network. That's kind of like the naive way of saying it. So really. the causal effect here that this is pursuing is really the reduction in traffic and the ability to move those attestations, in time, to where they need to be. So… this is the physical… so going from 1 million attesters to 128 attesters is kind of like the physical way of achieving that, right? But what if there is a virtual way, right? So you just keep a million attesters. But instead, you allow them to consolidate and pre-aggregate their attestations at source. So this is something that the protocol should have been doing since the beginning, but it was way harder for different reasons, and they weren't good mechanisms to defend against permutations that could appear in the network. The problem here is

If you're gonna be allowing messages in the network, attestations in the network, do not represent a single attestation. But to actually group multiple attestations within them, you could… an attacker could come in and try and dynamically recombine the attestations that they're seeing flying about in the network to form new, unique messages, right? So you have this, like. infinite… not infinite, but, like, like, very highly bounded universe for permutations, right, of those attestations. And that basically translates to spam in the network. So, there's a proposal, it's called Batched attestations at Source. We might rename it to Pre-Agregating attestations at Source. We don't want to call it Aggregating attest… don't want to call it aggregate… Aggregating. Only because that has a very specific term in the protocol, and this is not… not what's happening here. So yeah, so we may… we might back out of this name and just call it pre-aggregating, because that's really, really what's happening.

So, the key unlock here is… okay, so how do we prevent, so the idea here is that instead of sending a single, every validator in a committee, sending a single, attestation to the network. You allow those beacon nodes that are running multiple validators behind them to pre-aggregate at source before they emit the message to the network. So this is already happening today. If you look at all… like, it's already happening that beacon nodes are collecting, are collecting the attestations from their validator clients. So really, this would just introduce one step that would be transparent, almost transparent to the validators themselves. So the idea here is that you would have… For those validators that are attesting at the same… at a particular committee at the same point in time, you would pre-aggregate their… their attestations at source, and then sign over them.

And there is a two-way binding, to make this secure, so that that attestation, that aggregate attestation, cannot be recombined with other attestations in the network without it being very obvious that there was, some malicious actor, doing… doing something, because this cannot happen in a… in a… in a benign way. So the way that that mechanism works is you have the notion of a batcher, so the batcher is… one of… the validators that are, say, for example, you have… you're an operator, you're a large operator, a validator pool, or dbt, and you are… managing, let's say, 50 validators that are gonna attest on… that happen to attest on the same committee. So, normally, you would send 50 messages to the network. With this, you send one.

So what you do is you find out who those validators are, and from those validators, one of them is gonna be the batcher, so it's gonna be an endorser of the batch. It has to be, and this is an important invariant here, it has to be one of the participants in the committee. In that signature, in that pre-aggregated signature. So then, what you do is you take… Every attester is gonna create a batch seal. Consenting to be included in the batch that is going to be signed by the batcher. And then, once the batcher has collected the signatures it wants to include, it is going to aggregate them, and then they're gonna seal their whole… they're gonna put a signature over that, that aggregate.

So now, this has some nice effects. In that, with this data structure, the validation becomes Pretty, pretty easy to be able to stop, any malicious permutations, because they just simply can't happen, because in order to create permutations, you would need to be one of the validators that are participating in The permutation, and then you would need to… you can't strip signatures from the permutation, and you can't add signatures to the permutation either, because they wouldn't be… the validation wouldn't check. So, that is basically the key mechanism here. The important piece is that early proposal… early… earlier versions of this proposal were actually, Seeing the need to… so we… we thought that we would be needing to introduce new slashing mechanisms.

So that we could slash permutations that appear in the network, and this kind of two-way link was something that we came up with, More recently, in a later version of the proposal, that basically just made the need for slashing just go away, because you can validate at the network, and you can be sure that whoever transmits an attestation to you is acting in good faith. And if not, then you would just go through the normal gossip scoring mechanisms to penalize them. Another important invariant is that And this is how we bound potential, because there is still the possibility that there could be, permutations generated in the network, that could… not permutations, but kind of like different aggregates aggregated in… or different forms of these… of aggregates sent.

But they're actually upper bound by… By another mechanism that we introduced. So basically. Every validator in the committee is able to, today, sign once, right? For one payload. If they sign twice different payloads, then that's anyway an equivocation, and it is slashable. They're only able to sign one payload. And they're only able to act once as a batcher. That's it. So that's the mechanism to… that's the other mechanism that allows us to prevent to upper bound potential permutations. So in the worst, worst, worst case scenario, totally worst-case scenario, you would be having a single operator running all of the validators within a committee. So say 515, we said. Imagine that's the case, hypothetical scenario, right?

They could, with this proposal, though the worst possible kind of like, you know, offense, would be Each of those validators sends a single attestation, and each of those validators acts once as a batcher. So, the spam is bounded, and that's it. And there's no possibility of sending more permutations, because each validator can only act once as a batcher, and that's it. So the idea is you would, you would have the… yeah, so you're upper bounding. We're trading off, kind of like, perfect… Crypto-economic, like, security from… from the perspective of A slash condition here for just a, an upper bound on the spam, which is very, like, a very… in any ways, nothing, nothing off a stretch, because we're talking, 100 kilobytes as a baseline. We would… as a, like, if we just 2X that, it would be 200 kilobytes. Of course, there's an amplification here, because,

this is a gossip network, and we are amplifying traffic, but then, on the other side, there's also other proposals that Sukun and I are working on to reduce the overhead of traffic, so really the net would be, like, it would just net out. Anyway, like, once you even apply, like, if you put yourself in the worst-case scenario, and you apply that upper bound, and you're… imagine you're, like, you know, hitting the most adversarial scenario here. I think that's pretty much it. The one thing that I do want to do these days is… Speak with operators? To… because we have good… intuition from implementers, from client implementers, that this is a very simple proposal, or, like, relatively simple proposal to implement.

And they kind of, like, have already sketched out, some of them have already sketched out how it would happen. So, the next thing we want to make sure is that dbts and, pools and large operators and so on, like, is there any operational concerns here, right? The reason why I personally feel really good about this, versus the validator consolidation, MaxEB, which was… I think I didn't mention this. But we had one of the… one of the goals for… or kind of like the… what went in, I think it was in Petra. was, a… the validator consolidator… the balance consolidation EIP, which allowed validators to consolidate multiple 32 ETH validators into a max 2048 ETH validator, the uptake has been rather slow, and…

This is because it requires, like, a dedicated action, planning, there is, you know, risk in moving keys and doing things with keys, outside of the ordinary, so, of course, people, like, tread very carefully. This proposal is entirely transparent, right? It just happens behind the… behind the… behind the scenes. Validators don't need to do anything in particular, it's just the client, handling this for them. So that's another… another benefit. But of course, something that we need to check is in the different architectures that… validator pools have and operators have, is there's one more interaction, right? Which is, you need to collect the… well, there's two interactions here, right? Depending on how you look at it. But one, you would need to collect the batch seal from the validators, which can be done ahead of time. So this is… does not need to happen synchronously. Once the validators know, that they have been scheduled for duty, for attestation.

they can just… and they know, and, like, there's a policy that determines which is the validator that's gonna act as a batcher. They can just omit the batch seal and just consent to be included in that batch, and that's it. And even in the worst case scenario, if they… they could send their own attestations as well to the network, if they have any… if there is a fallback path, the wor… like, the… Disaster recovery mechanism is for the validator to just send its attestation anyway, because it's not slashable, right? So, I think from that perspective, we're really good. The one that I am less… that I need more signal on is the dbt, because DVTs do run their own consensus internally, which is already a synchronous step.

So my understanding is that the batcher and, like, the collection of those signatures could be done anyway through that consensus, because they have They had that, like, those interactions. So… so yeah, so that's, but, but yeah, we need to, we need to definitely go into… Have those conversations and understand how this would be done. Cool. I see there's some questions in the chat. Who is the batcher? Is it a committee member? Yes, it is a committee member. And it is a member… It is one of the validators that you operate, so it's not another operator's validator, and it's not a protocol role. It's just an implementation detail. And what exactly is the DOS vector that we avoid with the seal? Yeah, this is… this is a good question. We want to arbitrary, but we do not want that because of the traffic and because of compute, needed to verify them, yeah. So, there is a possible simplification here, where we…

Remove the bat seal. We discussed this, in Svalbard, but I haven't had a chance to think carefully about it yet. But I do think that there is a possibility that we remove the bat seal. The reason… is that… I think there was a very specific, edge case, which I can't remember right now. I think it was… let me just think through it. So the… the… BatchSeal creates a authorization for a given batcher, And… this… okay, yeah, so this would… this allows… 4… and… a batch… yeah, so I think this allows for… this prevents a… A scenario where a batcher So say, for example, I'm a validator, I'm a deep… I'm a loan validator, right? And I've omitted my single attestation, and so I have, like, another 100 validators.

You could have a malicious batcher that batches their own attestations. And also collects into that attestation. the signatures of the single… the leaf signatures of the other ones, and because they haven't consented to being… so there's… that could happen if you don't have the mechanism of the bat seal, because there is no way in the aggregate to verify that those validators were authorized being included in that batch. It is a non-rational… it is an irrational attack if you have a validator that's actually valid… that's actually emitting a valid attestation for itself, because then that… that one would be intersecting with the single attestations, and you would need to, like, evict those… those ones. I think that was… that was the main reason. This gives you perfect…

This gives you, It simplifies the tracking logic, because if you don't have this tracking logic, if you don't have the seal, then you could have this scenario, and in order for you to be able to accept The aggregate that also included leaves that were not part of that signature, or that had not been consented to be included, you would need to evict those leaves before you include the aggregate, because BLS does not handle intersections. It is a super edge case, and I think it's irrational, right? But I wanted to start from, like, the strongest proposal, and then we can weaken it if we want. Yeah.

Gottfried Herold: Actually, there is a question for… from the crypto side. One thing, sort of… I was sort of wondering, or some comment, because you said, that you don't like, or that you think it's a better alternative to sort of consolidation. I think yes, kind of, for as long as we do BLS. I'm not sure how much that still holds once we eventually converge to a post-quantum signature scheme. Where basically things… sort of the… sort of the parameters change quite a bit for those.

Raúl Kripalani: Yeah. That, that is true. And… at one point, it just becomes, yeah, there's, like, this is… this is very dependent on the network architecture, but yeah, that's… that's a good point. I think what… This is probably our best way to get to… like, an effect similar of what we are actually needing here, I think. unless you consider the post-quantum heartbeat as well, to have all validators participating, I think this needs to be scoped, right?

Gottfried Herold: Not necessarily, not necessarily. And I'm just saying that, I mean, I fear that, I mean, the benefits you get from this don't really… I mean, they're… they are basically somewhat specific to the BLS case, so… They are. My… my feeling is that we need this kind of consolidation effort to make, sort of, the post-quantum things feasible, or more feasible, or sort of to lessen the burden we have, sort of, on the… whatever happens once we get to post-quantum, and I mean, the point is the earlier we start consolidating, the more uptake there will be. Yeah. It's more of a political thing, that if you have some alternative mechanism, then maybe all the validators won't start consolidating, and so on. I mean, just some bit of… I mean, trying a bit devil's advocate here.

Raúl Kripalani: Yeah. Yeah, I think, I think, yeah, that makes sense. I think the alternative way that I look at it is, yes, there's probably some… a stronger cap that we're gonna need by here, right? Which is when post-quantum attestations really would kick in. But this, would allow us to stretch the timeline to physical consolidation, right? Because it is… it has been quite slow, right? Again, I think it's been a year, and we haven't, like, seen a big amount, I think, like, maybe 58,000 validators or something like that, I can't remember exactly off the top of my head. But it's been… it's been rather slow, so the idea here is you stretch the timeline, and you actually, like, there's more justification in this case. It's, like, a hard justification, because if you either increase the bandwidth requirements and the network requirements for everybody here, if you don't increase…

If you don't reduce the attester cap, or… or… or you reduce the… sorry, if you don't reduce the number of attesters, or you actually, manage to do that. But it gives us… gives us more time. And it's a fairly, fairly simple proposal to implement.

Gottfried Herold: Yeah, I mean, yeah, it's, like, kind of nice, because it's sort of the… the consensus there doesn't even need to know much about the details, that's nice.

Raúl Kripalani: Exactly, yeah.

Gottfried Herold: It's kind of using the fact that.

Raúl Kripalani: Yeah.

Gottfried Herold: are indistinguishable from non-aggregates, in a way, in BLS.

Raúl Kripalani: Exactly, yeah. Yeah, and it's using the metadata to reconcile, and to make sure that the alignment is right.

Gottfried Herold: Yeah. What does the timeline, or sort of the validity time you have for these seals, are they just for a single, for a single slot, or are they more…

Raúl Kripalani: Yeah, let me pull up the data structure, it does include the Yeah, it's hard to…

Gottfried Herold: read, it's hard to read, possibly due to screen size, so maybe I have to zoom in a bit more.

Raúl Kripalani: Yeah, maybe that's why I'm also… my eyesight is becoming worse, because I have such a tiny, tight muscle. It's great. Wait, so the… yeah, so this is just the pre-image, includes the exact slot, so at this time.

Gottfried Herold: Okay.

Raúl Kripalani: Yeah, so, you know, validators find out that they've been scheduled to airbox ahead of time, right? So, they would just, emit the image, and that's it. They could actually emit more than one if they wanted to. Okay.

Gottfried Herold: I was just wondering if you could sort of have a bit of a more permanent sort of, authentication for these kind of binding, on-chain, the on-chain registry for more long-term… but that's kind of very similar to what a kind of a soft… soft consolidation, in a way, if you sort of make this…

Raúl Kripalani: Yeah, but this is not…

Gottfried Herold: In record.

Raúl Kripalani: So this data structure does not go in-chain.

Gottfried Herold: Yeah, yeah, I understand, I understand. It's just sort of… it would be an alternative that… I'm not sure whether you… probably you considered it, because it's kind of an obvious thing to do.

Raúl Kripalani: Yeah, the problem is that it replies another transaction, or another object, or, like, another…

Gottfried Herold: Yeah, yeah, yeah, yeah, yeah, yeah.

Raúl Kripalani: This is really, like, this is really an inline… like, I like to think of it as an inline authorization. It's an authorization token, basically.

Gottfried Herold: I mean, it's not so clear that it requires something inline or some new transaction token. You could just have the token, sort of, basically have a range of, slots, and then use on the networking layer some kind of caching, if you don't want it, sort of.

Raúl Kripalani: Yeah, one thing to note is that this object is actually never transmitted, it's just the pre-image, right? Because you already have this data within the attestation itself, so what you're.

Gottfried Herold: Yeah, yeah.

Raúl Kripalani: compute the object, and then, verify that this thing matches, and that's it. So… so yeah, I mean, I think there's honestly little to squeeze here. The one thing I was… the one thing that I still go back and forth about, and I think we just have to get… like, more precise about the attack model here is, the threat model is the… is whether we need the seal mechanism to begin with or not. That was something that we discussed online, that we discussed in Svalbard. And yeah, we need to, we need to just study it a little bit more.

Gottfried Herold: Yeah, okay. Is this written down? Yeah.

Raúl Kripalani: Yeah, it's an EIP, it's currently in a PR, as I need to.

Gottfried Herold: Okay, yeah, okay.

Raúl Kripalani: Same changes, it's, this one.

Gottfried Herold: Yeah, I mean, it's just neat to sort of point to cryptographers at this also, I guess.

Raúl Kripalani: Heavy. There you go.

Gottfried Herold: We're not… we have not been aware of this, just to make sure that some of our team actually.

Raúl Kripalani: Oh, yeah. Yeah, sweetie, that would be great. Yeah, I really appreciate it, got it. Whoa. Yeah, thanks a lot. If you have any further questions, feel free to reach out. Happy to… to answer them.

Sukun Tarachandani: I had one question in the chat. if you have.

Raúl Kripalani: Oh, I missed it. Can the batcher send 2N aggregations for each of the validators it has a batch seal from? No, so every batcher, can only… every validator in the committee can only be a singular attester and a batcher once each. So, a batcher can… once it's sent a batch, it cannot send another one. It can send conflicting batches, that is true. It can send conflicting batches, to the network. And, like, if they take different network pods, then it might end up, you know, in a place, in a weird place. I don't know, this is maybe something that we need to analyze.

Sukun Tarachandani: Okay.

Raúl Kripalani: Not to… not to… not… yeah, it's bounded… yeah. It should be bounded, yeah. I do think that we're gonna need some… So there is a… I think we are tracking the arity of the attestations? And that would be the mechanism to de-conflict. But no, that wouldn't work either. Yeah, we're gonna need to check. The principle at play here is that in order to be a batcher. you're actually attesting as well. You are an attester in that attestation. So therefore, you have something at stake. If you act maliciously, then your attestation would not be included. But then we have to counterweigh that with the potential, like, benefit of running an attack if it's possible, because, you know, the penalty of, like, an… your attestation not being included is tiny… could be tiny, you know? So we need to kind of… or relatively small to… to the… to… I don't know, the…

the impact of the attack, so then that's… I think that's the piece we need to check.

Sukun Tarachandani: That makes sense, yep.

Kamil Salakhiev: Okay, good. Yeah, thanks, Raul. Yeah, so I think link in the chat, yeah, feel free to read, and yeah. We can discuss further if you come up with more questions, I think, in Telegram group, or… I don't think we have Discord about, Discord channel about these budget decisions yet. Yeah, alright, good. Anything else? From anyone? Or we call today. Alright, fine. Yeah, thanks everyone for joining, and yeah, I guess see you in two weeks.

Sukun Tarachandani: Bye.

Raúl Kripalani: Right.

Kamil Salakhiev: Bye.

Jeff Chung: I'm…

Anton Nashatyrev: Bye, thank you.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.