P2P Networking Call #10

Transcript

Daniel Knopik: Hello!

Kamil Salakhiev: Oh, boy.

Anton Nashatyrev: Hey, guys.

Kamil Salakhiev: Hello, hello. Yeah, maybe let's give another minute, and then you'll start.

Yeah, hey Chase. Yeah, I pinged, actually, some GF guys, so yeah, if they can join, here we can. Yeah, discuss it first, but… For now, I only see Csaba, maybe when others are here, yeah, we can bring this up. If it's okay for you.

Chase Wright: Yeah, yeah, whenever, whenever. I don't… Monarchy. I just don't want to be the last, because I won't be able to…

Kamil Salakhiev: Oh, yeah, yeah, noted. Here we go. Put it earlier, it's okay. By the way, I have a question. Is this time convenient for everyone, having these P2P calls? I mean, for me, it's like 8.30pm. But I'm fine, I mean, I can make it, it's just here, How much people would have that in some other places, and… Is it possible to make… Push this call to be… Well, to happening a bit earlier. Yeah, it's kind of… we have this because, like, yeah, we have some people in America, and people in Europe, Central Asia, and yeah.

Okay, I guess, yeah, the order, I think, yeah, will change, as we go, of what we present today. Anyways, yeah, let's get started. Yeah. Fine, brothers. Nope. Oh, whoa. Good for you, Daniel. Yeah, okay, maybe I'll just… create a poll in the Telegram group, and then, yeah, if, yeah, we decide to move it earlier, yeah, we can do this. Anyway, so yeah, let's get started, and yeah, just as usual, yeah, let's start with client updates, so… Maybe begin with Lighthouse?

Daniel Knopik: So, on Monday, we had a release, and that release finally, finally, finally enabled partial messages on mainnet by default. So, the usage rate should go up on mainnet. As we speak, I haven't looked at any stats yet, but mainnet has not broke yet, so that's good. Yeah, I will… oh, probably… Look at some metrics on, like, once people have I had the chance to deploy this. And post some interesting stuff somewhere. I'll talk about it in the next, networking call. Yeah, besides that, the usual, usual stuff, nothing exciting to report yet. Yeah. Draw anything to it?

jxs: No, no. On the network. Whether that I recall.

Kamil Salakhiev: Yeah, okay, good. Yeah, I see… yeah, there was a question from FLCL, but yeah, I think we responded. Okay, nice. Do we have anyone from Prysm? I saw Satyri, but he just left us. I think we don't have anyone. Yeah, I don't know, Anton, do you have anything, maybe, from JVM, jvm-libp2p, or…

Anton Nashatyrev: Yeah, no, nothing major from jvm-libp2p.

Kamil Salakhiev: Okay, good. And, yeah, from the Geth side, Csaba, anything? I'm not sure, like, how much we had to work on EL recently, but yeah, if you have anything.

Csaba Kiraly: Yeah. Actually, it's on Prysm. So I implemented, message validation in last week, in… in go-libp2p at different levels. A post is in preparation on that. I have a few slides, which I represent, I think, the next occasion. Yeah, not exactly, gas on gas, I see no particular changes this week.

Kamil Salakhiev: Okay, yep, FLCL, do you… Have anything to share, or… We're good.

Alexey Osipov: I guess, some… Not very… related stuff, but we, added, Consensus layer plugin, so we had to bring, P2P, We did not yet mesh it, but we plan to do so. And we are working on a light client that will hold Kind of partial state. For the purpose of optimization. Maybe something per contract, or maybe something sharded across such lifelines, or for nodes, not sure yet. It's a kind of research. and exploration. And as for partial data columns, do you mean, like, it's, about gossip protocol, or… Oh… Like, blob data columns? What?

Kamil Salakhiev: I think it's EL side. Yeah, if you're asking about, like, about what Lighthouse released, it's basically, you know, now we have Partial data columns enabled by default, so yeah, if there are… I'm not sure, like, if I need to recap, like, what… Partial messages, they're… what are the partial messages, but if you want, we can do that.

Alexey Osipov: Well, let me ask in games, something like that.

Kamil Salakhiev: No, it's fine, I mean, yeah, I don't know. Daniel, do you want to elaborate, like…

Daniel Knopik: Yeah, yeah, I mean, it's a… it's a CL thing, so… so… so basically, So, in… as most blobs on mainnet are in the public mempool. Most of the nodes have them already when the… Blobs arrive. when the block arrives, or… yeah, right? So, we have getBlobs V3 and V4, so we can… Retrieve, these blobs. And… If there are some missing, like, let's say we have 15 blobs and one of them is missing, we previously had to re- gossip the whole column. Which is very wasteful, because 14 or 15 are already, known by Some or most of the network. So with partial data columns, we can basically, let nodes just… Communicate the delta.

So whatever is missing on the other node. Yeah, through a mechanism that is, you know. kind of gossip, but a bit more akin to an RPC-like structure where we pull By communicating, what we have and what we're missing.

Alexey Osipov: Oh, this sounds great, thanks. I didn't know about that.

Kamil Salakhiev: Yeah, I think as long as, your EL supports, Get blobs V3 or V4, yeah, so, yeah, CLs can take advantage of Yeah, of partial messages, so I think you should support it, so it should be already okay. But before it's even better.

Csaba Kiraly: Excellent.

Kamil Salakhiev: That means you can also, like, probably you support reduces sparse blob pools, and, yeah. Yes, that's all.

Csaba Kiraly: Yes, it only needs the Get blobs, and then, depending on the version, you have different efficiencies, but, the CI just has to surface it, and then it's the bumping it, basically, into the peer-to-peer diffusion. I, I forgot a few things on the Geth side, because it wasn't last week, but the week before, I think. So, actually, it was earlier, but I never said it. I was also working, so Geth has a, a CL light client, called blsync. And, We sometimes use it, so we, we updated it to, to be, I got, Glamsterdam compatible, and also, yeah, I just made, made updates to it, and also made a few efficiency updates to it. It also has an experimental way of attaching to

to, all the P2P topics and, and just fetching stuff from there, so it's, it's, it's getting, A quite interesting, one, but experimental, definitely not for production use. Not… not merged on… on main. Yeah, but the part which is, which is merged is… is the, the old, BLC client, which one can use as a CLI client, with… Updated to… to run with the… with the new forks.

Kamil Salakhiev: Okay. I assume this, light client could be used as kind of backend for the wallet applications, for example, right? Something like that.

Csaba Kiraly: Yeah, yeah, yeah, that's kind of how it happens to run. And also, if you need a quick deployment, like, I'm using it, not because I was usually done as EL, but when I just need to test individual EL things, but I want to test on mainnet, I often use this variant.

Kamil Salakhiev: Okay, cool. Yeah, for the interest of time, yeah, I think Chase kind of requested to, Take a look at his issue chest, maybe you can quickly make an overview, like,

Chase Wright: Yeah, sure. So, I posted the issue in the chat, and, basically, so I was doing, like, a cross-client analysis, or whatever, of the ENR records, and Geth is the only client that publishes a snap key in the ENR, and it publishes it as an empty array. And so, I was looking at all the other clients, none of them set it. It's not specified in the EIP whether or not it, like. should be set, or how it should be set. And so, I kind of came up with a proposal of, like, okay, I think we should set it, and I think it should be versioned. Gary commented that he thinks it just might not be necessary, and I just don't have the technical background

on it to really know what to do about it. So I think there's basically, like, 3 options, and one is just we don't put SNAP in the ENR record at all, and then the other two options is we put SNAP in the ENR record, but we… Put the version that is supported in the value, or Or we leave it as Geth currently is, which is it just reports an empty array, and I'm not sure what the best approach would be, and I'm also not sure if, like, it really matters. Like, I mean, what happens when Snap version 3 comes out? I don't… I don't really know. So… that's kind of where I'm at. Like, I'm happy to do whatever the clients want, and I can modify my PRs as needed, or close them, but I think we just need to…

have a decision on… on should SNAP be in the ENR record, and if it is, how should it be, valued?

Kamil Salakhiev: Okay… Yeah, I'm not super deep into… Yeah, yeah, ENRs, Oh, yeah, I don't know, maybe any EL can comment? The FLCL replied, LGTM, okay.

Chase Wright: Yeah, I mean, Nethermind actually already merged the PR that I… that I brought up, because it was put in draft, right? The actual EIP Change? from Tony was… was put into draft to actually update the EIP with that wording, and so I opened PRs to… to all of the repos and clients to implement what I had proposed, right? And so. I mean, if that's fine, then… then I don't have to do anything, and the clients have to update, and that stuff has to get merged. But, you know, if… if it's not the right approach, then I'm happy to change and do whatever you guys want, but I just don't know. I don't know enough… To know what the right decision would be.

Kamil Salakhiev: Right, yeah, I'm curious, so you have PR to devp2p? If I'm not mistaken. Or it's just an issue.

Chase Wright: Yes, there is… there's an… so I opened the issue, and then I opened a pull request to devp2p to specify the proposal, and then the EIP itself has a draft to update the EIP. And then I also open PRs to every client to… to push it. And it's kind of weird, because the reason why it came up and was weird to me was, Reth… doesn't support SNAP 1, and with Amsterdam, they're going to support SNAP 2. And so, that puts it in a weird position where, like. Reth doesn't support one, but then if it advertises SNAP, then what does that mean for that particular client? And then, like, Nimbus, their execution client, I believe they can sync Snap, but they can't serve Snap data, so…

That puts it in another weird edge case.

Kamil Salakhiev: Right. Yeah, I can bring this up, maybe, to the Geth team, and yeah, we can discuss, like, and get it either merged and fixed, or… Get back with some… No useful feedback, but yeah. For me, it seems like, yeah, It's not like… should not be hard to fix that, and I mean… Yeah. I think the main issue is, like, we have different behavior in different clients, and we just need to unify that.

Chase Wright: Yeah, yeah, exactly. That's all I'm really looking for, and I don't need an answer on this call, but if more people could comment or chime in, and we could get a rough consensus, then we can move forward.

Kamil Salakhiev: Okay. Yep, I think, yeah, we just need to keep track in this issue, and Yeah, I mean, thanks for escalating that. Yeah, we'll take a look. Okay, nice. If nothing else for this, yeah, maybe let's proceed. Yes, Sukun can you be next? So, I think we wanted to bring up… Need a couple of Consensus updates, and anything else? I mean, we can discuss 8334 if time permits, but yeah… Let's see.

Sukun Tarachandani: It's a bit unfortunate that the CLs I wanted to pitch 8334 at are not on the call, so…

Kamil Salakhiev: Yeah, we can skip it to just…

Sukun Tarachandani: Yeah, and just talk to the person. Discuss the decoupled consensus.

Kamil Salakhiev: Yeah.

Sukun Tarachandani: I'll zoom in a second. Yeah, so, we had this workshop in Cambridge, and I presented these slides there. I guess it would be useful for the wider group as well. And, so this is about simulating decoupled Consensus, which, is… proposed to be headliner for I-Star. And, the idea is that we decouple the votes from… So currently, our attestation votes does two jobs. It has the head vote, which determines the head of the chain via LMD Ghost. And, wow. And then… It has the target and source checkpoints for finality and justification, and it would be nice if we can keep these two votes separate, and let both of these processes run at different rates, if possible, which is the availability chain, like, the head vote is simpler with

We can do with a nicer algorithm for the head vote, which is goldfish, and have the consensus, like, the finality vote be a different algorithm, which is Simplex, which has some nicer properties. Now, the slide does not have details of the consensus system, but only about, like, the part where we intend to improve the time to finality by 4X by… just… actually just increasing the amount of voting we do by 4X in an attestation subnet. So… Yeah, basically the idea is that we would have four times the traffic in attestation subnets, and then that way we will be able to get everyone's vote four times as quickly, and then we reach finality four times as quickly.

Okay, so moving on. So this is like a stubbed-out Prysm where the details of the consensus algorithm have been removed, but all the objects that are propagated on-chain are… Like, of the correct size. And basically, the major change that is, like, when I say stubbed out the details of the consensus, what we have done is basically saying that the latest block received ever is head, like, and since this is not a Byzantine network, we are running this in a cloud system with 500 nodes, who are all trusted, and they're all running my Prysm fork. So, yeah, basically, we don't need to go into fancy fork choice, but we do record whether goldfish votes reach on time on every node and these sorts of metrics.

So yeah, that's basically the test network. We have 500 cloud nodes, and the way we chose 500 is the fact that there are 447 nodes in the subnet, in a single subnet, and so… we have 500 nodes that we want… that we were given with our budget. So… Every node is basically in two subnets, and these are the configuration for super nodes and home nodes. And, We are testing with a slight slack of basically about 25%, so we have 2,500 validators per subnet, as opposed to 2,000 that you would have on the mainnet if we were to reduce, time to finality to 16 slots, yeah. Now, this number 447, like, if you have done the math, this might look actually a bit higher than what you would expect when you have 10,000 nodes each subscribing to two subnets each.

So this number actually is higher because Prysm and Teku nodes subscribe to their aggregator's duties well in advance, so they do this to epoch in advance. one epoch in advance. And so, yeah, basically, you add the aggregators for all of those 32 slots into the subnet, so the rough number comes out to be 450, basically. I think Teku has an even more lax policy where they keep on being subscribed to for the entire epoch. I think Lighthouse is more aggressive here, and it only subscribes one sub… one slot in advance. But yeah, net-net, this number comes out to 447 nodes in a subnet, and so we are simulating with 500 nodes in the subnet. Just as a point of, like, how to… how we think about structuring the simulations, it's that every node should be processing as much traffic as they process on mainnet. So that's, like, the goal of, like, how to get to these numbers.

This was, like, more relevant for the audience there, not probably as relevant here, but I just wanted to highlight the fact that, like, from the Beacon nodes perspective, we have no control over I have and I want messages. And in theory, we can add this control, but then that would require extensions to gossipsub and would likely be a different protocol. Yeah, anyway. So, again, to elaborate on, like, the main proposed round structure for decoupled Consensus, it is 64 committees over 64 subnets. So, this number stays the same as it is today. Sub-rounds align with slots. I will explain subround what a sub-round is. Every Beacon node subscribes to two subnets, and basically this keeps the number of beacon nodes in the subnet the same as today. And the idea there is that we don't want to change the security assumptions that are related to how many nodes are in a subnet, and

basically, this also gives us how many nodes would you need to DOS to, like, censor a committee. So that number sort of stays the same. Yeah, so now this is the slot structure. Basically, now what happens is that you publish a block. And when a goldfish committee for that block receives the votes, they send out a goldfish vote. Which says that I have received the block on time, and if it does not receive the block on time at the 3-second mark, it will send out a timeout vote saying that the block did not arrive, and my head is… with whatever is its head. And then there are 56 of these FFG subnets which are voting.

And we basically artificially stagger the voting to these 5 interval… 5-second intervals, So that we don't flood the network with votes initially, and what happens then is that the goldfish vote loses out to competing FFG traffic. So… I think Csaba will talk about in the next call, where if we do topic prioritization, these problems can be alleviated, and we can potentially publish all the FFG votes right at the slot start, or, like, discuss other slot structures. I'll also, like, mention that when I discuss the EIP8334 later, that if we just do EIP8334, then also, like. For the network that we have today, we… you don't even need to do prioritization. That saves enough bandwidth that there is enough slack in the network to propagate the votes.

And basically, yeah, every subnets… every subnet has one committee, and that committee's aggregates are published in 9 seconds. These numbers were actually chosen out of thin air, so the idea was, like, it seemed like Let's see if this works, and then tweak the numbers around, and turns out it works, so yeah, it's fine. Yeah, so this is, like, the first slot structure that we tried, where we have, basically 64 committees and 64 separate subnets, which, to our simulation, maps to two committees over two subnets, and each having 2,500 voters. For contrast, current slots have… current committees have 500 voters, basically, so we are, like, doing a 5x test, which should bring finality down to 4x at least.

And basically, yeah, at round start, you have a block, and then you have a payload, and with the payload, you have data columns, and you have a goldfish vote. There is a problem with the simulation, because there was a bug in my, stub, so the PTC vote is actually happening at 9 seconds, and not at 6 seconds, or when the payload is received. Yeah. You can give me just one second, someone's at the door.

Kamil Salakhiev: I'm not sure if it's a bug, I thought we actually voted second 9.

Sukun Tarachandani: No, so you're supposed to vote at, Basically, whenever you receive the payload with the data columns, and.

Kamil Salakhiev: Oh, good.

Sukun Tarachandani: you are actually supposed to vote by 6 seconds, I think, if you do not receive both of them in time, say, below or not timely. 9 is actually when the PTC vote is due, as I understand. And, like, yeah, that was sort of the bug that had picked up the line. From there. Yeah, anyway, so, moving on, okay. Yeah, Vlad does correct here in chat. anyway, yeah, so this was the slot structure, and like, we simulated it for, like, 200-ish slots for all of them, like, mostly, like, 300 slots, I think, for each. specific simulation. It mostly worked fine, like, there were no blocks missed, there were actually no votes dropped either, like, almost everything worked fine. There were a couple of nodes which just weren't working fine, but they had, like, separate issues, they never was booted up, so I don't think it was, like, related to what we did.

And most of the votes reached on time, there was a bug in my aggregate logic, which I will show that reduces the aggregate traffic a bit. Like, shows the aggregate traffic logic a bit low, and the bug was basically that We have this optimization where if you… have that aggregate or a better aggregate, you don't forward things, but the bug is that you should at least forward one aggregate, basically per subnet. And since we weren't forwarding that one aggregate, aggregate propagation takes a bit long, so… And the other… the reason why everyone had the aggregate was the fact that FFG votes were, like, just propagating in time, and every node was in both the subnets. So everyone, like, just had all the votes.

Yeah, and as you can see, like, payloads arrived on time, data columns arrived well on time, goldfish votes arrived on time, and, like, the block also arrived on time. So, in general, it looks like the system can support it if we stagger the votes. But there is, like, a small caveat, so this is, like, we look at what the data is flowing in the system, and this is for a super node, so, like, net-net super nodes are fine, they have a lot of data column traffic. And these attestation traffic keeps on going, but I will now go to the home node view, which is, like, the more interesting of the thing, and I also don't want to take a lot of time.

So, if you look at the home node votes, basically, there… there is a potential of things bleeding into the next slot, because, like, yeah, it is just taking, that much time to clear out all the queues, and you keep on receiving duplicates, because we don't have, like, I don't want or something of that sort of structure here. Yeah, so… the problem here is that basically our system is right at the edge, so if you look at the bandwidth used, it's, like, almost… So, our peak bandwidth requirement is roughly, like, 40 Mbps per second, and what we have is 25 Mbps. So, there is some queuing, the queue does clear up before the next burst of votes arrive, but it would be nice if you can… if we can reduce this.

Yeah, so basically, the next round is with the EIP8334, which is, like, the bundle attestations, and… And the EIP is fairly simple, like, currently we send 4, votes. the whole attestation four times, attestation data. If you have four validators voting for the same head vote, or, like. the same target and source. We'll send 4 different attestation votes. What we can do is we can just wait for, like, 10, 20, 30 milliseconds, and we'll get a bundle of, say, 4 or 5, or maybe even 10 to 100 signatures in, like, the middle of the slot. In, like, the middle of propagation, and that would basically reduce our… our data used by almost half, right? Like, so for this 4-vote case, we see that 4 attestations, that would have taken 960 bytes, now take 552 bytes. And this also does not include the fact that our topic name itself is 50 bytes.

Yeah, so this is based on partial messages. We are porting it to Waggle, but I think the numbers would be even better on Waggle, because it gives you slightly more control to the application. So, yeah, this is the case with partial messages. And here, the numbers look actually pretty good. It's the same slot structure as before. The numbers I will show are slightly worse than they would be in practice, because my queues were a bit too tight, so some messages were dropped, but there was only a bit too tight, so all nodes received all the votes, like, there were no vote drops happening here. So this is strictly a lower bound on performance. So arrival times are… actually have improved. I will show one comparison towards the end for goldfish votes, that goldfish votes actually propagate faster with partial messages, because there is just slack capacity in the network to send out the goldfish votes faster.

Otherwise, they are just competing in the first second of the slot. So quickly, this is a super nodes view. As you can see, like, the… attestation traffic has reduced a lot, I'll show a comparison for both towards the end, and the same as before, like, data columns make up a lot of data on super nodes, basically. Like, if you would subscribe to all 128 data columns, it's, like, a lot of data. No pun intended. But yeah, so… This is, like, the interesting view for home nodes, where you can see, like, votes have significantly reduced. I actually have a bug in my implementation which sends more metadata than I need to, like, it does a lot more syncing than I need to.

I can also, like, improve that, but even without, like, that sort of improvement, and, like, just a dumb share your metadata with whomever-you-want-to sort of implementation. Now, basically, the data used is well within our 25 Mbps limit. Like, it's, we have, like, yeah, almost half, like, more than half the data requirement here. And you can see the benefit for it, basically. The latency does not improve much because we are, like, very close to… the graph latency of, like, what the latency in the system is, because, like, the word propagation delay is roughly off the order of 500 ms, like, it's not a lot. Yeah. And also, like, the queue was short, so some of the improvements that was… would have been found is not seen here. But since we have saved up a lot of bandwidth, Goldfish votes actually are faster by almost 20% at P99, like…

So, Scenario 1 has 1 second, almost, to propagate goldfish votes, and Scenario 2 takes almost, like, 730 ms. So, like, it's more than 20% faster. And that sort of is the advantage of this system, that basically we can rely on other traffic to get the priority that it needs. To show as just a requirement here, like, this frees up a lot of space, and, like, that can be used for things like, bigger payloads, 8411, and other things, like… So, the graph here, basically, the dark color is for EIP8334, where, like, admittedly, like, you would expect lower data usage, and the lighter graph is for EIP… for basically non-EIP8334. So, most of the traffic is FFG votes, and it just takes a lot more traffic, as you can see.

And as you can see, the PTC votes are basically overlapping completely. towards that. So this is the other simulation that we did, and this is, like. sort of why I am pitching for 8334 to happen is because… Like… This is where, in a single subnet, we schedule 3 committees, where this is the first committee, votes happen for the first 3 seconds, then we have aggregate for that committee. Then the next committee votes, then we have aggregate for that committee, then the next committee votes, then we aggregate for next… that committee. So, on a given subnet, there are 3 committees, and in total, there will be 64 into 3 committees. So, basically, imagine 64 subnets in parallel, and 64 into 3 committees happening at these timings.

What this gives us is this nice property that, finality, things are completely decoupled from the round things, and you can Even imagine moving the last two… these two columns, rows around. And aligning them just wherever you like in the slot, like, and just completely decoupling the availability chain, which just keeps your, Chain after finality running, and the finality chain just keeps running in parallel with, like, some sort of a minimum required bandwidth. This actually works fine also, but, like, all things receive one time, but we are, like, yeah, we have now crossed the goldfish votes into… 1.1… 1.2 seconds, as opposed to 1 second before, right? And so, things are a bit slower here, because there is… the burst at the slot start is bigger, and so maybe with the topic priority work, we can reduce these bursts, reduce the…

these bursts affecting goldfish votes, and then basically actually have this very completely decoupled system design, where the finality things happen completely in parallel to block propagation. Because the nicer thing there is, then, if we do quick slots. You can imagine a block being propagated at 3 seconds and basically at 6 seconds and 9 seconds. Hi. Yeah. So, for numbers here, yeah, this is super node usage, which I will also quickly move on. And, yeah, so here, basically, we are, like, right at the edge. So, as you can see, all of the FFG traffic that we pushed right at the start has caused a lot of clogging, which makes Goldfish votes sort of take a… low priority. Now, it still works fine, everything has reached in 2 seconds, so in principle, we can do this, but it does feel like if we want to do something like this, it would be great to have topic priority, or at least to have EIP8334. Just to be clear, these numbers are not with EIP8334, this specific.

Yeah, I think that's it for the slides. I have a few more, but, like, yeah, I think they give enough background to the problem here. I'll just say, like, This is for, basically, the audience there, and for everyone to think about, like, how should we think about staggered publishing versus multiple committees. Which is basically Scenario 3 versus Scenario 1. And, like, we have been chatting about this, and it feels like we need to decide on the set of properties we want, and then make this call, because… Considering the network can support both, at the moment, the choice feels a bit arbitrary, which we don't want to do that, and it would be nice to write down the properties that we want, and why whatever design we choose is aligning with those properties. I'd just say, like, if we do EIP8334, we can also do 4-slot rounds with 8-slot finality.

And, yeah, so this is just for a… This is the simulator I have, which can run on Shadow and subnet, and basically 1,000 nodes, 200 super nodes, 1 million validators, 64 subnets as we have. And, basically, we publish this way, at different points in the slot. And this has fits the PTC votes as well, by the way. So, actually, everything reaches well on time. There is a lot of aggregate traffic which flows over into the next slot, but as you'll see, like, it is a worst-case simulation, which in practice would not happen, because I have not implemented the optimization where you only forward the aggregate if you don't have a… like, superset aggregate, which, in practice, we have, and, like, on a given subnet, we only get, like, 4 or 5 unique aggregates, and not 16.

But this is the worst case scenario, and it still works fine, but the aggregate traffic bleeds into the next one. So anyway, for the numbers, this is the supernode time, this is the home node time, whereas you can see, like, the aggregates are, like, getting towards the 10-second mark, but they're still well short of 12-second mark. Yeah. Goldfish votes finished by 1.5 seconds, which is good enough for us. Same way, payload takes up to 2 seconds. If we do 8411, it would be much lower. This payload is much bigger, this is a 256KB payload. Block and data columns are the same as before. Yeah. So, actually, FFG vote traffic flows through for just fine in this, and we have to optimize for aggregates to be propagated within the time. I don't think this would be a problem with the optimization, and if we say we will only get, like.

Eight unique aggregates, which effectively house these, but we do see some bleeding of the aggregate traffic into the next slot. But just to be clear, this is… twice as much load as we want to push for in I-Star. This is… 8-slot finality, the numbers I showed before on the Prysm run were 16 slot finality. And that's it!

Kamil Salakhiev: Cool.

Sukun Tarachandani: This…

Kamil Salakhiev: Yeah, looks nice. I'm curious, these shadow simulations with million validators, did you run them on, Prysm nodes, or you had some binary.

Sukun Tarachandani: No, no, this is like a custom simulator which just sends the correct size objects and responds and correct size things. So, when it receives the payload with the data columns, etc, it will send out a PTC vote and so on, but the data itself is garbage, like, it's just data.

Kamil Salakhiev: Okay. Yeah, it would be just interesting to maybe scale it to… around 10,000 nodes, which is kind of a bit closer to what we have, I think, in the real network.

Sukun Tarachandani: I try, I think…

Csaba Kiraly: Yeah.

Sukun Tarachandani: So, this one works till 4,000, so yeah, I think 4,000 is good enough for us. But yeah, yeah, sure, I can try. But just to be clear, the load in total is the same here, like,

Kamil Salakhiev: Yeah, in terms of amount of messages, yes, yes, yeah.

Csaba Kiraly: So I know it looks good on the headlines, but it's actually absolutely irrelevant in some sense, because what you should do is just understand the scaling behavior, and from there, it's, but it's… If you know how it scales from 1,000 to 2,000, you… and you just check that. And also the eastern a little bit, you just know how things scale, and there are no surprises, it's at 10,000, so… I wouldn't… I know there's a big push for these things, but I don't think it… we should… There are so many other things we can waste resources also on. When it will be the final design, yeah, definitely, just do all the checks we can do, but but to understand these properties, we really don't need it.

Sukun Tarachandani: I completely agree with that, yes.

Kamil Salakhiev: Yeah, we can.

Sukun Tarachandani: It's much better to understand the scaling law here than to run it on a specific size.

Kamil Salakhiev: Right. Yeah, any other questions? Yeah, I had a quick one. You had a slide about FFG votes propagation, what is, like… how many votes are propagated there? Is it, like, for the one group of validators, voting, like, at some particular second? Or it's just,

Sukun Tarachandani: Okay.

Kamil Salakhiev: That's the question.

Sukun Tarachandani: So, in this, not, let's look at this one. So, in this scenario, you will have two 2500x3 votes in this committee here, here, and here. So, sum total of these 3 would be 2,500. And just to compare, like, when we run this on mainnet, this number would be 2,000. So, we are taking, like, this 25% extra slack to understand. Here, these numbers would be 2500 by 7, basically. 1, 2, 3, yeah. So, this burst would be 2500x7, this one would be 2500x7, and so on.

Kamil Salakhiev: Hmm. Okay. Yeah, okay, yeah, I'm just assuming, yeah, like, in real network, you're not voting every slot, like, it's just, yeah, you have assigned one committee, and yeah, you're still voting just once, if I understand correctly,

Sukun Tarachandani: Yeah, but,

Kamil Salakhiev: Yeah, you still can't respond.

Sukun Tarachandani: Comes from that, basically. But it's still kind of subscribed.

Kamil Salakhiev: I mean, the rules are the same, you still subscribe to two subnets, where you propagate. And, yeah, there is just one committee where you send. Your vote, and that happens… like, you have 8 slots, and yeah, you vote, basically, one time every 8 slots, if I understand correctly. Yeah.

Sukun Tarachandani: Not that.

Kamil Salakhiev: And that, yes, and that makes one epoch, and then, yeah, we have two epochs of 16 slots, and that's four times less than what we have today. Basically. Yeah.

Sukun Tarachandani: Happen on mainnet, yes.

Kamil Salakhiev: Yeah, good. Unless we have some consolidation, maybe, and we reduce number of validation.

Sukun Tarachandani: Interesting. Yeah.

Kamil Salakhiev: Which is the whole.

Sukun Tarachandani: That would obviously be great, yes.

Kamil Salakhiev: Although there we have… then we have another question, like, whether we can, like, decrease number of Subnets, or, like… Y-yeah, kinda like… Whether we have less… Subnets and committees, or we, Yeah, my final question. Yeah, I think…

Sukun Tarachandani: Basically. So far, the reasoning, at least of the… from the decouple side, has been… we'll change the subnet structure at PQ. And only change as little for this as we need. Because, like, the consensus itself is changing a lot. And for PQ, we definitely need to change subnet structure, like, the votes are bigger, and, like, aggregation is a different beast completely, etc. Like, here, aggregates are free, right? Anyone can keep adding them. Right. Which won't be the case with PQ Signature. So, Yeah, I mean, if we consolidate a lot, 8 round… 8-slot finality and, like, 4-round slot… 4-slot rounds is a very possible target, but then the question is, can we do this with post-quantum signatures, which is, like, which are much larger?

So, nope.

Kamil Salakhiev: Yeah.

Sukun Tarachandani: Okay, I will stop sharing.

Kamil Salakhiev: Okay, thanks Sukun. Any other questions? Okay, nice. If not, then, yeah.

Sukun Tarachandani: I actually have a question for Anton. Sorry.

Kamil Salakhiev: Yeah, yeah, that's cool.

Sukun Tarachandani: Anton, I understand you have also done the simulations which had the similar things, so did the numbers seem familiar to you? Do you think there was a lot of discrepancy? Sorry to put you on the spot, like, we can discuss separately if you want.

Anton Nashatyrev: Yeah, yeah, actually, the numbers look pretty similar. Actually, I would love to, To, look at your, charts in more details and, compare.

Sukun Tarachandani: Sure, yeah, okay.

Anton Nashatyrev: Let's shared,

Kamil Salakhiev: Okay, Yeah, Sukun, if you can share your slides, I think it would be good, maybe, to answer others. Okay, good. Yeah, next in agenda, yeah, we wanted to discuss, informed diffusion from Csaba. Csaba, are you here?

Csaba Kiraly: Yep, let me find my slides. In just a moment… Where did I put them? Just time to open the iteration. Give me a sec… Yeah, I'm starting to present the right version. Okay, so… Share…

Zoom is accessing my screen. Nasty Zoom. First clean, first clean. Almost there. Do you see it?

Kamil Salakhiev: Yes.

Csaba Kiraly: Okay. So… Yeah, I was publishing, Informed Diffusion, a week ago or so, which is itself is built on, on many things, including FullDAS. But now I decided that the protocol should have a name, and the name is zcast, so… What I'm presenting is, is, is what I call as fast and lean broadcast for ethereum. Which happens to have an executable spec and, some formal modeling. Lean… I had to have lean in it. So there's lean in it, lean here means, not using that many bytes. So, why we do this thing? because we have all these big objects coming up, I don't have to explain this too much to you, I guess, but, like, the first objective is obviously payloads, which are growing,

People tend to underplay it a little bit, but we are adding the bars, we are boosting the gas, and if we can, we boost it even more, so these will be big. But we will also have execution proofs, you'll have objects with post-quantum signatures, and so on and so forth. So these big objects are just, just coming, And then, yeah, big objects are working today, and that the reason behind that is mostly that we have a particular deployment of Ethereum, which we are lucky to have, a bunch of nodes in data centers, lots of bandwidths. It's just not what our protocol guarantees, want to say. So we say we have better guarantees, and we should have them as well.

When we are speaking of this broadcast, this is broadcast, so you are sending out from a link, Kind of the… the limit is, obviously, you have to send out at least one copy, and and that's kind of the minimum that you can do, and that's the minimum time, and And, when we want to have a protocol, we want to have protocols which are kind of close to minimum, and what we have in gossip-sub is just way, way, way far from that. For… for various reasons. So… Yeah, let's dig into it a little bit, if I can. So… In zcast, we do… a few sinks for this, Molly. Yeah, I might do the long, slides. So, in zcast, we do a few things.

We take, so we apply a bunch of techniques. Let me see if I have that slide, because I think it's just… Just better. So… The funny thing is that, that, these things comes from, from, in many senses, from FullDAS. And… and one thing to realize is that broadcast is, In many sense, just a special case of segmented, sample broadcast, the special case where everyone is sampling the whole thing. So, I was taking a bunch of ideas from past research, and a bunch of ideas from, from FullDAS, and combining them into a single protocol, which can do broadcast, which can do also. sampled broadcast. There are a number of techniques, I'm not going through them. I have another slide which is describing the techniques. But here you see a few of them. We are segmenting, we are doing erasure coding.

We have lots of control over the, what in gossip sub you would call, I have and I want. So the information flow between the, The entities. We have the sampling, and we have kind of all these components together. The goal is to be able to do… Broadcast to be able to do. sample broadcast as it is done in, in, FullDAS, or to be able to do what RowDAS is doing, or what Food DAS is doing. So… as I said, the kind of the… the thing is not new, it's a combination of a bunch of techniques, lots of techniques, from the, 2020s on, fast chunk diffusion. Lots of techniques on how you manage recording, things like Epidemic gossip, Plumtree, ideas from BitTorrent and CoolStreaming, ideas from peer-to-peer streaming research, ideas also from FullDAS on the sampling side.

And, yeah, some of the ideas are new, some of the ideas are reshuffled from literature, some of that literature is my own papers, some are from others. Anyway, there is a background document which is having all these literature references on the techniques. And what you get is blazing fast, and also using, only a few bytes over what is the bare minimum. So, how I built this up is, is going basically technique by technique, although, I mean, the development process was a bit different, obviously, but the evaluation… in the evaluation, I was doing an evaluation where I evaluated, the performance adding technique by technique. And, What you see here in this slide, which is quite an important slide, is two dimensions of the evaluation, and there are many other dimensions, I would say, but this is the performance dimension.

You have the latency on the… I don't… do you see my pointer?

Kamil Salakhiev: EF…

Csaba Kiraly: Yeah, so this is the latency axis, in milliseconds, and this is the, the byte, the bandwidth utilization. The, the how much more we send than the bare minimum, sending out one copy of the, of, of the data. I mean, it's sending… This is overall bandwidth utilization network, so if we would do just pure pull, for example, what we would get there? And what you see here is that, this is… this blue point is where gossipsub is today, and this is how we are sending payloads today, so if you would send one message… one megabyte message, you would get this point, or this point, or this point. And the difference between the three is the scenario. So I have three scenarios. One is, let's start with the red one. That is kind of what we have today. The BlockBuilder is building a block, and it's sending it out, and we have this huge network with

a bunch of 1 Gbps nodes and another bunch of 50 Mbps nodes. I was parameterizing it with some numbers, it doesn't make… it's not that important. I was seeing 20% 1 Gbps nodes, 80% 50 Mbps nodes, the builder is sending out one Gbps. We can argue about these numbers, anyway, this is the point. Then the second one is the same network, just the builder is sitting at home. So the builder has a 50Mbps uplink. And you already see the huge… the touch between the two, which we currently… have for our home builders. So, our home builders suffer a lot. Here, there's a break in the timeline, which is an… Which is located at mixed scale, both are logged at mixed scales.

And then there is this more theoretical point, like, what our protocol tries to guarantee, and our claim is that we have minimum nodes requirements, and it would be nice if that claim would hold not just at the node level, but at the network level. So this is the theoretical point where if we would have nodes, where every node is sitting on the minimum requirement, this would be our network. And that's, like, it just doesn't work. And then you start to apply techniques. Here you have the… in smaller techniques, some of that you know, like segmentation, and you bump heal. You increase the, the… bandwidth utilization, but you have huge latency gains, so we are able to…

Down, down, down the line. Then you apply, batch publishing, and and then you do some more. And then you apply technique after technique, as you're cooling at some point, and you can get to these points, the 17ths. And this is the, kind of, the… the performance, zcast is giving you. this sub-700 millisecond, performance over, a network where the node is… has only 50Mbps uplink, V's… Only… Less than twice the bandwidth utilization. Which is… Half of what we are doing now. Yeah. Okay. Next. By Zcast. So… is not just fast and, and, and… lean, but it's also fast and lean in composition. So I was comparing it to what I have.

I have the gossip sub implementation. Which is this. I have the ecbroadcast implementation in its current state. And I'm not saying it cannot be better. This is kind of the current ecbroadcast implementation and the current zcast implementation. And you see that zcast is good, ecbroadcast is good, it's just not as good. And we can speak of other protocols, zcast is a combination of techniques, and these techniques are… we can apply them in other protocols, and then they become better, obviously. But this combination is a particularly good combination. There is another advantage of zcast, I'm not ready with the final figure. zcast is elastic. So what that means is that, it's a probability of a good networking protocol, that if you are

Start to suffer, and you are, under bandwidth limit. And you… your timeline, obviously, extends, but you also start to be more bandwidth conserving. you don't explode. And some of our protocols just don't have this property, so they… they tend to explode, and that can lead to critical failures. What is good is if you don't have those issues. So here, depending on where we are, how constrained is the bandwidth, zcast has very good scaling properties. So… The less bandwidth you have, obviously delay is increasing. But the bandwidth utilization is… is… is not exploding. It can handle the… This, and it can handle it like we model it here, where the whole network has these things, and it can model it in a heterogeneous network scenario, which we always have that.

Again, some numbers, not the most important things, When you are reading 1.6 copies, take it with a grain of salt, which is, it is compared to the, to the uncompressed. So I'm compressing it, and so it's, it's, it's, it's two compressed copies. So we can do better. We can do, just pull, and then we are down to too much less than this, but our delay is obviously much, much, much, much, much… Hiya. Okay, as I said at the beginning, zcast is not only for… just for broadcasts, but also for sample broadcasts, so thinking of the future, thinking of the past, whether we did FullDAS. Thinking of other ideas around it, and thinking of the future where we have other sampled objects, most,

well known being the block, that we want to have, for which we want to have a sample diffusion once we have ZK proofs. So zcast can handle this sample diffusion, and it gives, very nice, times, and… bandwidth characteristics. This is the… One specific, simulation, and I should, learn much more, 32 blobs… 128 columns, 500 nodes. The source is, is, happens to be a… 1 Gbps one. There is at home nodes. So, yeah, we can, we can change the scenario. This was the one that I was running quickly. We have… The, 50% of the nodes having the sample in less than half a second, or around half a second, and in 3 quarters of a second, everyone has the whole sampling.

Yep. maybe one more thing to say about it is that it has a Python specification, so it's not just, Oh, by the way, I never said it, it's a Go implementation built, Delight from, go-libp2p and, and go-libp2p-pubsub, and, so it has code-wise gossipsub abstractions, but it doesn't have to be… it's not tied to gossip sub in, in any way. And it's… it's… part of it is implemented as part of the PBS code. So the application code on top of, of the Go implementation of P2P taken from Prysm. So that's the… that's kind of main implementation, but I was also doing a Python specification, and what I was doing, so on the Go implementation, I was running two types of simulations.

One is, the framework of Malco, used in a larger simulation framework, so it's native Quick, and the other one is Shadow, and in Shadow, I was running it with TCP Quick, and different backends. So those have matching performance, and then I was… there's the executable Python implementation, and that has itself a toy Python simulator, which is… which can run the toy model, the specification, and… and compare results, and… to some extent, they match. Obviously, the toy one is… it doesn't have the sophistication than the others, so there are differences. it's nice to have all these, because you find bugs through these. But I was also doing, with more formal specification. So I have, some of the… Properties, modeled, sequined.

And as for my specification, and I have some… Checks already on that? the gaps, the things to solve, it's not finished. This is, version 0.5 of the… of the protocol. That will be, version 0.6, I'm sure. But it goes into the direction of having a formally verified, network implementation. Yeah, so that's where I am. This is zcast. Next version is, is, is in the works. needs a bunch of testing still. I'm learning lots of explorations of different properties of it. happy to kind of try it in testbeds, DevNets, whatever, compared it to other solutions. there's the post on Ethresearch, there's the spec, and then you can find the series of specs out there.

I think that's it for my slides. Yeah. Then, lots of details, obviously, in the posts. Every technique has a… has its own description. And the papers behind it. That's it from my side.

Kamil Salakhiev: Yeah, it looks interesting. Yeah, I understand it's a… It's basically how we can basically… maybe… I'm not… like, maybe we should not call it an endgame, but something… Where we could iterate towards starting from 8411, and then… Yeah, because, like… Yeah, 8411 is super simple for now, and, yeah, just trying to figure out, like, how do you envision, like, how we can take advantage of that, like, whether it's, like, next iterations, or, like, it will be a future EIP for some other forks, or, like, yeah.

Csaba Kiraly: Yeah, so it's kind of codifying this gain. as a minimum, the segmentation gain, which is number one. And everyone has batch publishing, so… so you can already look at the point two. And then, I was going in a particular order here, but there are techniques we can just apply, as a local policy, from all these techniques. without changes to 8411, just, I already know, see if the techniques applies, just on top, as a local implementation policy. Saliev was just a number, but I've seen a few which already apply, so… so we can… we can go to better points than this, just by using 8411 as it is. And then we can… we can make the jump down there, then we can make the… the sampling,

For the… for the future proofing a little bit, and also because we… you have builders. So, to be a bit more serious about the DAS design.

Kamil Salakhiev: Yeah, I guess I'm just curious, like, what actually the zcast is? Is it, like, a bunch of techniques? Is it, like, concrete protocol? Or it's, like, some kind of framework, so we can maybe take this optimization, this optimization, and then combine them and, yeah, come up with something else.

Csaba Kiraly: So zcast today is a combination of these techniques, which you always have a combination of techniques. Even gossip-sub, you have an combination of techniques, it's just some of the techniques are not even defined, and you dig in the implementation, and you find the technique. And the other one is… Is applying it slightly different. So here, some of these are codified to… To be the one which performs well. So, zcast as it is today, is a combination of techniques. It's a protocol, it's… we can embed it on top of libp2p, we can put it on top of Native Quick. If we put it on top of Native Quick, we can unlock a few more Benefits? we can play around with these things, but this is a combination, and that's, is consens a bit diversioning, obviously, because I'm obviously iterating a little bit.

Kamil Salakhiev: Is it only me, or we lost each other? Jebel, maybe your mic… It's not working anymore. Yay.

Csaba Kiraly: The ACD was the ethp2p implementation as it is today. Obviously, that can also improve, like, these techniques are universal techniques, it's not that, this is the P2P technique, and that's the something technique. I was… Taking a given combination… exploring combinations in a given way based on my past papers and other things. because I had ideas about what works, and this is how far I arrived, but that doesn't mean there is nothing better than Delhi's by definition. So we can iterate, obviously, on these things. And we can call it different names. It doesn't really matter. It's, We should, we should improve our networking, that's definitely…

Kamil Salakhiev: Right. Yeah, so it doesn't assume, like, so it could be integrated into ethp2p, or into some new transport. Well, basically networking stacks, if we have them. Yeah.

Csaba Kiraly: Yeah, so there is a specification we can, we can just codify it into the P2P specifications framework, if that's what you want to do. We can decide what to do. And there are other aspects which are… which are less, evolved. The… for example, the peer scoring aspect, on that aspect, Definitely. So I'm exploring it, and I'm doing part of the formal verification on that aspect, specifically. But that, obviously has… I mean, it already has weaknesses, as it is defined P2P, there are ideas to change it. We changed it in a given way in the execution layer, so I would… I would explore going in that way, but it's not a part that I was focusing on first. There are some…

punishments in the… in the… in the protocol, specifically because we are going, We are doing stuff like Announce and I Want, Let's use those names for this. and we are not asking from everyone, we do need to be really active when someone is not delivering. And that comes with punishments, which interacts with peer scoring. So, there are different… punishments at different timelines. But this… this was focusing on… on the, on the short timeline. So how can I… I do, kind of, short punishment, or short moving on to something else so that I'm… the single payload is… is surviving and being delivered? And not some long-term effect of, after some time I'm dropping you.

Kamil Salakhiev: Alright, Daniel?

Daniel Knopik: Yeah, great presentation, thank you. So, So, as I understood, you mostly focused on… At the, payload propagation in your simulations and research. And I'm wondering, if you intend… Like, if this is meant to be, like, a single solution for, like, all possible objects, like you mentioned, attestation are getting larger as well. And so I… I'm… kind of wondering… so right now on gossip, we have, like, this one dissemination strategy for all the objects. like, what do you envision? That these different techniques are kind of Easily pluggable for different, let's say, topics, or… That this is, like, only going to end up being used for payloads, or what is your reason there?

Csaba Kiraly: So, to be honest, Part of that is more a marketing question than anything else, in the sense that We can name and parameterize a combination and make it so open that at the end, the essence is in the parameterization, not in the protocol. I think… So what I was focusing on is… is the, The use cases we have for single-source to multi-source, diffusion, and that includes sampled diffusion. That also includes the, the kind of out-of-band delivery that we have in… currently in… in… in PeerDAS, so the, syncs popping up from the EL as additional sources, I don't think it includes attestation aggregation, because attestation aggregation is more in-network compute. So that's, that's a different regime of networking techniques.

obviously, there are things that transfer, but there won't be… there'll be much less in common than between sampled and non-sampled diffusion. Sampled and non-sampled diffusion has just, they are just simply very close, if you just look at it as, the non-sampled being a special case of the sampled. This doesn't really hold for the… for the many-to-many traffic patterns that we have in the… or the many-to-one traffic patterns, or the in-network computation traffic patterns that we have in the attestation aggregation. So… so I think that's, that's a different protocol. Now, we can use the same name, we can use a different name, we can… call zync is P2P and put these protocols inside, maybe that's the best thing, from the…

from the, like, from the ethereum protocol perspective, when we do this urge, we should give names to things so that we can speak about them and, and this combination… giving this name to this combination.

Daniel Knopik: Okay. Yeah, cool.

Csaba Kiraly: That's how I see it.

Daniel Knopik: That's how people suspected.

Csaba Kiraly: Yeah, there are aspects which are… which are kind of… which I was keeping outside, like, for example, the peer scoring, to say, to some extent. I'm exploring, but that was not my first focus. So, obviously, these things combine, and and… And, and, you know, no one has the one single, thing that works. It's a combination of many ideas from Discussions, past literature, just, yeah.

Daniel Knopik: Okay, sounds good. Thank you.

Kamil Salakhiev: Okay, Anton?

Anton Nashatyrev: Yeah, Csaba, great presentations, like, looks fantastic, Yeah, so one question I have, did you test it, with respect to Byzantine properties, like, how many… malicious nodes, my… my presence in the network, so it's, It will continue to propagate messages.

Csaba Kiraly: Yeah, absolutely. So I have, like, so I have a… in the post, and, so there is this post as the previous post, and the previous post has a background document linked, which was just too much for this, for, Yeah, so that's… that's on my notes, and that has this multi-dimensional explanation, and I was doing, like, 6 type of different technical models, and we can argue whether they are… how much they are realistic. There are simple withholders, and then there are exploring, like, the percentage of withholders up to 50%, and then there are malicious nodes which have active behavior, and but, you know, the active behavior which I was defining, so it's not, like, bulletproof in that sense.

And, that's where the formal verification line will be coming in on the slightly longer term, but for that, I want to speak with formal verification experts, because what I did was a bit hand-wavy. So, So that's not a lot to do on that front. The… for me, the important path was to start on the formal verification, because that's where we… we can close some of these gaps, where we have the… the unknown Malicious node behavior. And you want to have some system probabilities guaranteed. I did what we usually do, which is, you know, this is… this is how I imagine an attacker, let's… saw it in a simulation, let's put a percentage behind it, and It still works. I run it, I don't know, 10 times, 20 times, and I didn't bump into a failure.

But that's very far from knowing that there's no such attack which could compromise this and that probability of the protocol until they are more than 30%. And that's what I'm expecting from the formal verification branch of work. And the properties will not be the, like, I think, too alive to the point where the popletes are really the ones that we… that are making it bulletproof. It'll be a longer game, but we can… we can start chopping off some of these dots.

Kamil Salakhiev: Okay, Great. If no other questions, yeah, thanks again, Csaba, it was great. Yeah. I think that's all for today, unless anyone has any other things to bring up. Okay, nice. Then, thanks everyone for joining. Yeah, it was good, so see you next time. Right.

Daniel Knopik: Yeah, bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.