Consensus Improvements Roundtable: Low-Latency P2P

Transcript

Let me start with a little bit of context. Uh when we are speaking of networking in Ethereum, uh there are a bunch of things there uh which we are dealing with and we are doing improvements on on all of these and when we are speaking of scaling uh any of these can go bad and any of these can be the bottleneck. Um but today we will focus on on more on these things. So when we are speaking of low latency, we are speaking of block diffusion at the station aggregation and block dispersal. Uh for example in B dispersal what we have we have sharding and data variability and what we have is is kind of we are just doing partial delivery.

So the nodes are only receiving part of the of the actual data and this is we are optimizing bandwidths right uh with speed build we are going down to 12% of the usage with folders we are going down to uh I mean these are theorical numbers obviously had a bunch of overhead uh for validators like 1.5% and for someone just sampling we are at 0.1% of the data that they are getting and in full we already including a bunch of techniques to do with low latency Um okay uh changing s times what we are doing there we are kind of keeping the sut the same transaction per second or whatever per second uh is typically same but we are getting a better granularity

in the in the system. So for example we are getting shorter inclusion delays if you have a transaction which you sent to the meool uh it just have a gets a chance to be included faster and other benefits. uh and we all know that this requires better diffusion latency. So we already got to go improvements and uh well part of the go improvements that we are doing are advantage efficiency related less duplicates typically. Uh some of them are about better diffusion latency and where we also want to keep the protocol robust. Yeah. and and I was drawing that nice triangle there because people like that uh because that represents that uh you take have to make compromises and tradeoffs and uh but I

think there are some free lunch here uh others are compromises uh unfortunately it's the angle is not always the best model uh we also have privacy and I think we could if we think more we could add other dimensions um so yeah we have to take compromises and uh different techniques achieve different Uh so part one uh we all know propagation speed is the problem here uh with slow frequency right. So this is a uh measurement we did in uh 2023 uh on on blocks how we did this uh to make it difficult. This is alabetic scale but this is kind of delay here you have a 4 seconds and these are

different size blocks. So this for small blocks everyone is happy for large blocks uh was a lineish this was the situation uh and um well since then we developed a bunch of low latency field techniques uh I call them the low latency field to build toolbox and I want to speak of this toolbox so what are in the toolbox uh well there's one thing uh which is kind of u an obvious choice uh which is segmentation or sometimes we called chunking. Uh it's just about uh the fact that we are using gossip sub for something which is not was not meant to be uh like large messages are just you just don't send large messages is like

this. If you do large message in a network you are chunking it. Uh that's the way to do it. Uh and you can latency. So uh there's this thing called store and forward uh delay which is if you have multihop you are always getting the whole message and then you can forward it. uh if you have smaller chunks then you can fast forward uh and we are doing it so uh we are going in this direction we have this called a thing called now sellable messaging called partial messages uh which is u doing adding sub message support um I think we will also have something like substream support uh it's not something that yet uh this however requires verifiable segments so if you want to do this all the segments have to

be uh kind of verifiable uh somehow uh we have it with KCG in uh uh in D but we need something there. Okay. Our second tool is uh I was putting it together is scheduling and prioritization. So if you assume that you are band limited and you should assume otherwise the game is easy just send it to everyone and uh and it's done. So if you assume your band is limited, you have to make choices. Who am I sending it to? Uh what am I sending? And this is the uh scheduling and palletization and the gossip stuff is quite naive in this. So uh we were introducing things like patch publishing uh and that's already in some of the clients and uh you can do things

in uh forwarding uh scheduling qualities and I also I was putting quick in uh in this category because what we achieve is quick is is validated. So we have independent streams with quick uh we can prioritize between the them and then we can also do with quick uh something which is unreliable transport. So I'm sending a piece and then it's already timed out and I don't need it. Uh I don't have to resend it. I can send something else instead of that. So this is kind of scheduling and prioritization as part of our toolbox. Next one uh encoding uh we love this. So encoding uh lead encoding fountain quotes random coding uh these are techniques uh that we usually use in

communications and was not really used here. Um and what you achieve here so you achieve here basically what I call it typically you are cutting the tail of the delay distribution. So there are those cases when the delay becomes high for some reason for someone and you are trying to cut chop that off. So in the sense that those are getting it faster. So instead of waiting for a timeout and asking for a missing piece, you are just getting it uh with for getting an encoded piece. This is for collection or instead of sending duplicates uh you are sending another piece which is a useful piece because of the encoding. So you what you achieve is kind of you are reducing the delay because we are cutting the the

tail of the delay distribution. Um okay we can use it own quotes we can use fountain quotes. You can use London coding. Uh there's a small note there that this is adding coding delay and coding decoding delay. So you have to be a bit careful. Uh but it's a nice technique. Then we have network coding. Uh network coding was there since 2005. Uh it's has been there for long. Uh we might apply it. uh it's basically about in the network we can do recording and with this uh we can basically achieve a situation where anything we send becomes useful so there are no duplicates in the

sense that whenever I sending some something to someone he can use at least part of that it comes with compromises it comes with the compromise that then you you have to collect all the pieces before you can get anything useful out of it uh but it's it's a technique which combines combines very well with the nominal codes. Uh you can try to do something similar with other codes. Um it again adds coding delay. Uh but it can speed up and make more efficient uh network.

Okay. So we had four tools but we have more. So we have typical tools here. Uh one is the kind of the overlay optimization box. So you have this graph over the way and we do it sometimes random graph and then you can imagine that the data is ping ponging back and forth through the ocean and that seems like useless. Uh there are questions on whether that's actually useless or not uh for the robustness. Uh but you can do latency based optimizations. Uh again you have to be careful because you are reorganizing the graph and you are losing glass properties. you might have a bigger distance in the graph. So more multihop and and so on and so forth. Uh

but we are working on this. Uh this is this WFR gossip that we are working on now. So we are working on on a variant of this. Okay, that was number five. Number six, I should have put the numbers uh kind of traffic pattern specific optimizations. So we are using gossip sub now for like a generic tool uh which is good for everything and it's not. Uh so we have different use cases when you are sending the block these are isolated broadcasts of large messages when you are sending us attestations this is concurrent broadcast of small messages uh and there's also a question why why are we even broadcasting those which was also raised by this uh and I could continue list so for each kind of

use case you can tune the the protocol and get better numbers so the numbers uh and uh you might have noticed that I was not putting numbers. So I'm put showing two things here. One is this is from the uh explorer from the labs from from pandops uh kind of similar to the to the test 2023 test that we were doing. It's just a much nicer interface and much more modern. Uh so what you see here is one block, one block which was first seen after 1.72 seconds. So maybe someone was doing delay games or whatever. Uh anyway, let's assume that's when it was sent and then you see how it was diffused uh and

the different curves are different continents and notes uh at that continent. Anyway, what you see is that after 2.2 seconds everyone had that block and that block happened to be a 65 kilobyte block and I think there are also blocks and actually seven blocks and I think that's included on those curves. So this is fast, right? Uh that's cool. That's uh positive. uh it seems we can do much faster. On the right you see uh actually from this is from yesterday evening uh when I was running my experiment from 2023 on at the station propagation. So this is my node getting all the attestations for for a given uh slot and this is

filtered for those uh which were missing slots or slots which were were very bad. So until 4 second nothing happened and then at 4 second things started and one thing you can notice that is that there are good cases where it's very fast. Other thing that that you can notice is that they are kind of all different and this is the problem is these things. If I if I would draw four blocks one after another, they would be all different. And it's really hard to say like this is the mean material, this is the uh it's really hard to how to give guarantees uh on these things because there are so many variables in the system uh that it's hard to understand and actually even if you're simulating this, it's really hard to get to the point where you can say uh I have 10 to the minus 6

that it won't go above 4 seconds. uh because you just can't capture all the complexity of the system. Okay, but I mean the numbers are are nice. Uh we have these small numbers. So let me go to the part two of my presentation uh which I was calling is propagation speed the problem. Um and here I want to play a bit uh a naive view of the advocate and for that I have to go back a little bit to how episodical is working. So there's a sender sending a message that is a receiver receiving it. Uh and then it's sending an acknowledgement and then when the sender is getting the acknowledgement is deciding whether it

has to desend because it was not getting the acknowledgement or it sends the next message. Right? This is kind of how a a simple network protocol works. the the most simple uh but then you realize uh that you have to choose this timeout and when you're choosing this timeout there is some uncertainty and on how fast the message is propagating and how fast the acknowledgement is propagating so you have to add slack to it right and then you have to choose a timeout which is which is bigger stay with me and then you realize that message size also matters like bigger the message bigger the time you need even bigger timeout So the question is kind of how we define

the slot uh when we are speaking of this slot type and here I'm just kind of mapping out this figure and transform it into a a network instead of just a simple communication. So we have the builder it is sending the block and then in this communication which is not to a node but to all the testers we have to have bunch of slack the worst case size the worst case propagation multi of uncertainty uh and you can go on and then these atesters are sending out at stations and then that's again a multi of propagation this aggregation you have a bunch of slack there uh so the worst case becomes like huge right So the real problem is that we have to deal with

this third case uh because it's too hard for us to sacrifice a block uh and this is kind of my thesis here that uh uh we should make this this much simpler we should make it simpler to sacrifice a block in the system and then you can see scale and actually this thing that I've shown to you at the beginning this we call stop and wait and this is from the stone age in in networking technology like from the 60s or I don't know And then networking was evolving and we we developed a bunch of techniques. So we know how to paralyze.

Uh we can send a bunch of messages on the fly and then we have something which is called selective acknowledgement which is a bit map of of which one was acknowledged and then you see here bit maps and then you can think of oh maybe I can merge bit maps and that helps. So this is something kind of delayed execution and then in networking we know how to do capacity exploration. So how many messages you send we know how to scale this up and scale this down and we do front congestion control and then we also do something like timeout tuning like we we also know how to play with this uh I do exponential back off which is I'm extending the timeout and for example vitalik has now

a a post to exponential backoff I think and then we know how to do further recollection so we anticipate state losses. We do a bit of encoding uh and then we don't have to well we don't have that many zeros or even if we have two zeros it doesn't matter because we are encoding and we are just going forward fast and then we know how to pipeline. So it's not that we have to always wait uh we can just pipeline it. So I don't know how to map these things into blockchain space. uh I'm just saying saying that there's latency which will not go away from the system and in networking what we do is we are working around it uh and

sometimes I feel that here we are trying to do this worst case singy uh which doesn't allow us to to work around it so I see some ideas to work around it uh and maybe we just need to work around it that's all thank [Applause] questions. Uh yeah, maybe just on this last point, you mentioned that we do this worst case thingy.

Yeah. Uh and uh Yes. And I mean, I guess that sounds somewhat political, I guess, between working being able to work around it or not. Maybe you could say more on Yeah. So the uh currently the the cost of of sewing away a block which is quite big and lots of effort and then the network is doing lots of effort to to make sure that it's actually checking with everything. It's so much that that then you don't want to sw away and if that means that you have to tune your delay such a way that you have with basically 100% you are not slowing away 99.999 you are not throwing it away. If you find a way to make it cheaper to slow it

away then you can set this threshold to 90%. Right? Because then you are throwing away 10% but but you're fine and overall you are going much faster. Uh that's kind of the underlying thinking like the curve typically has this long tail and then so if you want to set it to 99.99 90% then you are going 10 seconds if you want to set it to 90% then you go 4 seconds. Uh right and yep which one is is the better strategy RLC or RLC? I mean might not be black and white but uh what is your take on it onc and uh

versusnc doing it at network level or doing it at a layer ah it's different usage so soc is is okay soc is very good for when you have a a big thingy that you want to to send over like the whole or nothing so like the block feels like that but then you you should think that maybe I'm actually better streaming the stuff so chopping it off to small pieces and and streaming it and then other starts to to have issues. I seen that people are using it in in streaming contexts. I always have problems using it in streaming contexts.

So, and I'm designing uh so so it might not be that good for that but I think it's it's it's a nice technique. It has issues bunch of them uh but uh but it's it's it's something to explore differently. And then LLC uh we can just use it for for example in D uh we can use it to to enforce the same thing node so that the sample it takes is just much stronger. It's not taking a cell but it's taking a linear combination of cells. So when you have to reconstruct uh you have a much stronger reconstruction guarantee.

On one of your first slides you had like different aspects of networking. One of them was privacy. What do you mean by privacy? It's like exposing the address of the nodes. Uh yeah, it can mean anything but it can it can mean that and it's always like uh scale. Uh so we have some level of protection in the system currently not that high. Uh but we do have something and sometimes it's a question because it's not that high maybe we should sort it away and make the network more efficient. uh or maybe we should enforce it and then we have so it's it's part of the compromise space and and who we want to protect uh at the network level.

What's your opinion on that? Like should we target to still kind of try to keep as not as private as possible or I don't know because it's so easy to uh to undo at the moment. So it's like it's it's always those questions. I don't know. I I'm asking because like we keep investigating and testing the gossip uh I think partially correct me if I'm wrong but partially because like this privacy assumption and if we don't have it then we could optimize it how propagation works.

Yeah. So if we solve it away we can we can make uh lower delay solutions. Uh my take on that is is that we need a draw pass. Basically we need a system which is which is have a slow pass for for private syn and maybe a fast pass for uh for fast transactions and maybe on the slow pass you are doing zik aggregation and on the fast pass you are not doing z aggregation and these have to be kind of go together but then the dimensions increase and then you have a problem but so it's privacy it's z aggregation because zik aggregation as well is is adding a delay Right? It's when you are doing these encodings then it's always you are

adding a delay to the whole process. So but how you apply it is is always like if you want to be really fast like I want my transaction included in 10 seconds then z aggregation is a bit in the way. Uh yeah. So can we do a zik proof of for example you being a stator and not giving up privacy? I think that works on that uh st you can be a staker and not give up privacy.

You you give a z proof that you are a staker so you can form a very efficient grade. No the problem is that you are giving it up uh when you are generating the sic. So if I can observe the safic I kind of corate it with uh with the fact that you stated and then I can pinpoint that who who are you because the schedule is is somehow public. So so you have to hide the schedule. You have to hide a bunch of things. It's just very complicated and and it always comes with with delay and and bandwidths overhead like huge. So you have to mass the traffic, you have to add extra traffic, you have to delay everything.

Uh it's it's it's just not efficient. But it's cool to have it. It's just not default. I think for privacy like that might be a place where node specialization is particularly helpful. For instance, one reason we want privacy is we don't want home operators to be exposed and and dust. But if we said privacy is really helpful for like these nodes that participate in censorship resistance so that we they're kind of more invisible to the network but we have stronger assumptions on nodes that make blocks or that test. I think I like the idea of having two types like one which is the faster that serves like certain purposes

and the other one that is yeah for slower more resiliency oriented services I think. Yeah. So also on the user side you have this like someone needs transaction someone don't. So you you have to think of this tradeoffs in that context. Can there be a dual role where some some entities will say that okay we don't need privacy but we will help the network uh propagate faster. Yeah I mean I I thinkators could be such role. So they basically give up privacy but they can uh everyone instead of just gossiping transactions to each other

they just send the transaction signatures to each other. They send signatures directly to aggregators and they do aggregation much faster. Yeah. Yeah. Like all the other GS are doing it already. They gave up long time ago. It doesn't mean we have to do it. It just it just you don't need because it's so hard and so expensive to do it right. Uh You might want to have these classes separated. Thank you so much.