Post-Quantum Interop Call #45

Transcript

Thomas Coratger: Hello, hello.

Shariq Naiyer: How's it going, Tomas?

Thomas Coratger: Right? And.

Shariq Naiyer: Very good, very good.

Thomas Coratger: Great. So let's get started. I think that today I have… we just have, like, the usual team update, and then I have only as a discussion topic, like, DevNet 5 stabilities. and stuff that need to be fixed or improved, so what I can propose is that after the team update, we can do, like, a discussion on the current difficulties that we have for DevNet 5. What remains to be done and all of this. So, yeah, let's get started with client updates. Zeam, first.

Gajinder Singh: Thanks, Stormach. So, we have been working on, fixing scaling issues that we are seeing in Zeam client while we are running the shadow sim test. So, after resolving some of the issues, basically, some of which were More towards how we want to account. the computation for Shadow. So after that, we are now successfully able to run 32 node, 4 subnet, all-Zeam Shadow network. to finality. For whatever number of slots we decided, so now the plan is to… Run it for longer number of slots, as well as to keep increasing the note count, and keeping it nodes per subnet. And basically doubling the note count on each iteration and seeing what is the max… max network that we can run.

Obviously, this is constrained because of, this is… this gets finally constrained, because there is limited memory in a particular server. And the server that we are running on has 96 GB, so we can run a whole lot bigger network over there as of now. And then we have, 300GB of space, which is on NVMe Gen 4 disk, so hopefully we should Be able to scale it quite a lot. Till we see… problems. So, that… that is one of our current focus right now. In terms of, Normal testing and node operations, we merged QUIC P2P, and we have seen some of the issues over there. One of the things was that our quick library wasn't multi-threaded, so we multi-threaded it, and basically are now running tests and resolving the issues that we are seeing, so we will have stable

client operations soon, maybe in a day or two. And, then, so while we are doing this, we are also running an all-Zeam network on… this particular server that we are already using for ShadowRuns, because in that server, we get to make, Proposals within time, because it's, 20 core… And 28 threads, i7. 14 gen, so it is able to make a proposal in times… in time, so we are using that server to basically scale up the network, and we have run, again, about 32-node, 4-subnet network over there, so we are seeing some forking, which we are trying to resolve, but we are hopeful that our QUIC P2P fixes will resolve those issues, and then the next step will, come to multi-client testing. So.

By the end of the week, we will start running multi-client DevNets and start testing them. And start skilling them. And apart from that, we worked on, We discussed the spec PRs related to Fork choice and let us justified, so in that, we… we basically approved the latest PR that says that, restation store should be had, let us justified, but then this invalidates, or sort of… not invalidates, basically, this makes redundant the previous PR. that, ETHLambda team added, that was basically, while block building, you, you, you start by adding votes that are even before Stores later justified, so that you can move move your… justified… you move your new block justified to the current, and then move beyond it. But with this PR, that is no more required, because now the votes in the new round of the voting, when everyone is going to be using state's head's latest justified as a source.

Those votes will then progressively move the justification and finalization forward. So that PR is not needed, and as I messaged in the group, Anshal will do a PR to revert that, because I think the spec should be easy, and that PR Complicates it in the sense that we already have multiple contentious contentions running while we are doing the block building. So, one of the contentions is finalization, second contention is justification, third contention is basically block building, and we want to simplify the block building, in the easiest manner, so that So that, basically, most of the times, you can easily do a proposal, even if you can't move justification or finalization forward, so that there is no liveliness issue. And in that respect, there is another assertion that we want to remove

after today's PR that has been merged regarding latest justified, so assertion was that when you have minted a block, when you have produced a block, its latest justified should be greater than stores later justified. Now we… We'll remove that assert, because… That is no more required, just because that now we are basically saying that, okay, a chain can… Continue… voting from its heads, let us justify the note from the stores, let us justified which could be heard. So… so if we remove that assert, then block building will become much simpler and easy. So we will have a guaranteed block building, which will… ensure that we… there is never a loss of liveliness. And if somebody still wants to, add… pick votes from behind the heads, let us justify it, so that it can move justification forward, so those are…

client optimizations, and those clients can… so basically, this is, like, client-wise optimization and client-wise strategies that all of the clients can build in. All these strategies are good, but again, in terms of spec and in terms of most of the contentions that block building has, I think they are adding complexity, so we will do a PR to simplify that. Yeah, that is Roma's side.

Thomas Coratger: Okay, great. Thanks a lot for the complete update. Rim?

Shariq Naiyer: Alright, hi guys. We've been, we've been implementing the optimizations and the fixes proposed to the spec, and we have been, we have been working on bringing some of the… some of the improvements that we… we found in our DevNet 4 runs to DevNet 5. Now, what we're finding is we're interoping successfully with ETHLambda in one subnet. This was just an initial test, now we will, now we will increase… now we'll add Zeem and Lantern and Grandine, I believe, as they're ready, and, we will, we'll have, like, 24 validator, DevNets. And I'll keep sharing updates on that on the… on the DevNet 5 channel. Let's…

Thomas Coratger: Okay, great. Clean, I have seen that in the chat, you said that there is no updates, so, okay, cool. Lantern?

Mihir Faujdar: Yep. Thanks, Thomas. Hi, everyone. Yeah, so, basically, since last week, we've just been testing Demo 5 runs on an Intel Xeon server with 8 cores and 62GB of RAM. Still, still been testing a small DevNet with two ETH Lambda nodes and one Lantern node, with Lantern being the aggregate… aggregator. And, yeah, basically the average Type 2 proof generation is taking about 2.8 seconds, Where the block is carrying one distinct aggregate, aggregate attestation. But what we've been seeing is that finality is not stalling. It is increasing over time, but yeah, there are some slots where block building takes over 10 seconds, and that just creates gaps in finality. On the M4 Max, Laptop, like, finality is definitely a lot snappier, like, like, there are a lot less gaps, for…

for finality. So basically, like, what we've been noticing is, like, as the number of distinct, aggregated votes increases, the time to build a block increases. But in terms of next steps, what we want to do is, we want to increase the number of subnets in our local DevNet. So, yeah, currently, I've just been testing one subnet, but in the next sort of testing, we'll be increasing the number of subnets, and as well as adding more clients. But so far, we've seen stability with testing with ETH Lambda, but hope to add more, clients. And given that the PR1166 was merged to lean spec, we will be adding support for it as well. Thanks.

Thomas Coratger: Great, thanks a lot. ETHLambda.

Pablo Deymo: I know. Hello, hello. We changed it at the stations to use the current has strutifier as source instead of the global certified MAX. That's because we found some edge cases that we reported. We made a PR to change to the spec that, I think that's the one that Gajinder mentioned. And our latest image has it PR, merge it. We updated how we stationed our score to prioritize newer targets in blog building. We noticed that, The Syngate mechanism forced non-finality when there are multiple missions loads, so we added a flag to disable it. We are also working on the pre-build optimization to start building blocks on the last slot interval. We're also working on improving consensus debugability.

Looking into state diff as a way to keep as much history as possible. And in parallel, we continue working on Goldfish and execution layer integration. Both are prototypes for the foreign demands to have proposals. And we saw that Hive tests were not working well. We fixed another Hive hardness bar… bug. Because we haven't seen any successful hybrid for a few weeks. So that's it in our side. Thank you.

Thomas Coratger: Okay, great. Gean.

Bankat | gean: Thank you, Thomas. Hello, everyone. So, following on from last week on Gean, we were reviewing several… and running several multi-client runs on our end, and Gean continued to… But speak clearly, proposing, importing, and finalizing alongside each peers. So, with no spec or SSZ layout regressions on our side at the moment. And also, we've also begun, work to bring Gean under the Shadow Network Simulator. which will give us, more reproducible and deterministic interop and network testing. So that is still in progress. There is still work in progress, though. And, the harness is to… sorry, is to harness and build, parts All the building parts are coming… the NHS and building parts are coming together, rather. And we'll share more updates on that as it matures. So, looking ahead, we'll just keep testing the multi-client runs, tracking, lean spec changes on the specifications, and also progress with our shadow integrations. Thank you.

Thomas Coratger: Okay, thanks. Grandine?

Mercy Boma Naps-Nkari: So… Yeah, this… last week, we've been running, several, validators with different, subnets. We did 32 with 4, and then 64 validators with 8 subnets, so… The only thing we noticed was the block building and then the type 2 verification was still low, and then finalization was a little bit chumpy at some cases, so… We're still looking out to, optimize that, and also follow up with the lean spec PR. by the Lombard seams, and then… Yeah, that's open mind.

Thomas Coratger: Okay, great. So, in terms of spec update and cryptography update, not that much things are new. I mostly finished the cleanup of the spec, meaning that right now it is pretty clean, at least from my point of view. I don't want to introduce more changes for you guys to check. Even so, like, I haven't introduced any spec change purely, but mostly cleanup, re-architecture things, but right now, I feel this is pretty clear. We should have a meeting internally next week, I think, about the spec topic, within the EF. So that is about the spec. In terms of cryptography. we are, exploring right now, some stuff around binary fields, as a proving system in order to check if we can get rid of Poseidon, maybe in the future.

Nothing is sure, of course, like, we still keep researching on the existing Prover modules and Prover architecture, but we are looking into binary field as well. Yeah, maybe Emil can say a bit more about this, but yeah.

T. Wambsgans: Hello everyone, yes, mainly working on binary fields for the week. That's it.

Thomas Coratger: Yep, so maybe now not that much thing's new from our hands, so… I think that maybe it could be cool, maybe to list the status and problems that you have with DevNet 5. Maybe we can have, like, a document, if someone is able to do this in order to share. In the async, in the DevNet 5 channel, so that we can have at least a list of the problems that we have with DevNet5. I noticed last week, that we mentioned that the time to aggregate proofs Was a bit too high for you, so either we should add to change the machine, or to increase the slot time, or to find something. But now that you are more advanced, I would say, in the debugging process of DevNet 5, maybe I would love, like, to hear from you guys about…

Issues and stuff that remain from you?

Gajinder Singh: Yeah, I think once we sort of stabilize a little bit more on DevNet 5 and do the PRs that, for example, we merged today. And remove some of… the asserts that we added. So, basically simplify things, then… Still… That, building, the merge proof. Is one of, one of the issues that we need to figure out, or basically say that, okay, let's do testing on our bigger machines, or… Say that, okay, block building interval is bigger. So… so we still need to… Still need to do that, and then definitely… We need to start scaling now, again, to see What is the stable and scalable network that we can run for a longer time? I think the best thing would be to…

sort of have a month-long running network this time. That is the target. And, so for that, obviously, proposing problem… so we need to basically take a call on how to deal with the proposal problem.

Thomas Coratger: Okay, so what I think could be cool, then, is that you guys continue debugging the DevNet5, merged a couple of PRs that remains, and everything like that. I can tell that the spec will not change that much, except maybe some small bug fixing, or some test additions, or stuff like this, but no major change to expect. So, once you have this debugging process done on your hand, maybe we can discuss a list, in the DevNet 5 channel of problems that you have. For example, like the… proof aggregation, if this still is a problem, but I suspect this will be a problem. So… yeah, we can list the problem like this. If you encounter other problems.

I don't know, DevNet instabilities. It could be cool also, like, to have the number of validators that you run for the DevNets, so that we can have a… some… some view on the scaling, because this is very important to try to run before the end of the year, I think, kind of a real, DevNet with, like, if possible, like, more than a hundred, like, validators, if possible, thousands of them. So, yeah, I would love to have a view on this number of validators. To be sure that we are operating in the right way, and that we are scaling, so… Yeah. I think that we can… maybe next week we will have, like, a clearer view,

On the pass, and on the debugging process once this is done. And maybe we can have a list of problems. The Omega?

Gajinder Singh: No. Okay, so before, Mega, I just want to say that we… this… so, in this particular respect, shadow sims are great, because it is totally simulated time, right? So, the time that… you logged onto Shadow is a simulated time, and so it doesn't matter whatever real… block building time is, and that way shadow, sims are great, so… we are shadow compatible, and I'm not sure which other clients are shadow compatible, so we can… so right now, all of our Shadow sims are Zeam only, but we are more than happy to include Other clients that are shadow compatible and have this particular, function have this particular configuration where they can simulate, basically, they can log the time that

That they need… they can log the time to shadow that You know, one basically configures that, okay, this is your… This is… this is basically the time you need to… configure for merge, so… so if you… if other clients can do that, basically, then we can start running interop in Shadow itself, and I think that would also… progress our, efforts forward before, basically. We are… we basically get stuck on this particular problem, saying that, okay, we need bigger machines, or… a bigger time interval. But yes, I think, cool, we can start using ETHLambda as well, so we'll start So it lambda message that they are Shadow compatible, and we'll start We'll start using them in our interops, and hopefully have some interops before the next call.

But as such, I think the easier solution to this particular problem is we can just increase Increase the time interval for block proposal. And so the right way is to say is that, okay, now we have like, okay. 3 seconds for block proposal, and then… there are normal intervals, because they shouldn't be taking time. So, for this particular problem, if we have one second interval, then we have like, right now, we have 5, right? So, we have 5 plus… 3, we have 8… Second slot, in which… 4 seconds are for the block proposal, and then… rest for intervals are for whatever normal intervals processing that we are doing. So we… we can basically transition to that and try running this network on a bigger scale using smaller machines. I think that is a more viable solution to testing on

On normal machines that are out there.

Thomas Coratger: Have you seen, I think in the chat there is a question of Mega for you also, like, do we need to add a slip instead of true aggregation for this simulation? I recall that Zeam having something like that. I think you are muted, Gajinder.

Gajinder Singh: Yeah, yeah, yeah. So we do need to add that, and that's what I mean by simulating So the sleep amount is basically the time you are logging on to Shadow, and saying that this is what you took. For doing that particular competition, so you need to introduce those slips at all those places which are computation heavy, in which We are tracking, we are tracking the rate of doing those competitions, so I think, Parthaka and Partha or Kai can provide you the config that you need to implement. So, once you implement that, basically, you will have implemented sleep at all the points that we need. Basically, all the major computations that are there, so that we can log those computations onto Shadow, but we will definitely not log the real computation time, we will log assimilated time.

So that's where the config is needed. Basically, you will say that, okay, in the config, you will say that, even though my real time is 2.5 seconds to do, proof merge, but I will only log 0.4 second element. So… so that is… that is, configurable through the config that you will need to consume. And… Yes, so you will need to add that sleep over there.

Thomas Coratger: Okay, so yeah, my plan… yeah, please go ahead.

Pablo Deymo: Yeah, I just wanted to add that on our servers, we are seeing that the CPU resource is mostly the limit. We have memory to spare on a 32GB server, but… We have, like, I think 32 threads total, and we are seeing finality lags a bit, with only 16 nodes already. Just wanted to add that. We'll later post it on Telegram, but…

Thomas Coratger: On 16 nodes, just each Lambda node.

Pablo Deymo: Yeah, just ETHLambda.

T. Wambsgans: Please do, number?

Gajinder Singh: Smaller and not the same, right?

Pablo Deymo: Tori?

Gajinder Singh: This is… this is just a normal run, not a shadow sim.

Pablo Deymo: Yeah, it's just a normal DevNet run.

T. Wambsgans: Could you… I heard the number… could you just repeat the conclusion? There was some lagging?

Pablo Deymo: Yeah, we saw a finality increasing, I think it was, like, in… 30 slots at a time. Which is… Yeah, it's weird. Mmm… That's with 16 nodes running on a 32-core machine. 32 threads. I think it's 16 calls.

Thomas Coratger: And I guess this is due to… Proof aggregation.

Pablo Deymo: Yo.

T. Wambsgans: Like, the bottleneck was not recursion, it was, XMSS aggregation?

Pablo Deymo: No, no, it's regression. It's aggregation on the blog proposal, which is mostly… Okay, I see. And type 2, type 1, or…

Thomas Coratger: Yeah, I think that you were not there last week, Emil, but basically, like, we discussed with Zeam team, and I think, if I remember correctly, Grandine as well, that we… they both identified, like. That's recursive aggregation, not row signature, but recursive. Was, like, a problem with the timeframes that we have, especially on small machines. Probably less performance than your laptop. And also, I think that right now, we are… if I am correct, we are still using, like, the… let us say the proven XMSS variant, and not your light XMSS variant, so… That is even higher in terms of cost. And so both of the team were reporting problems. On this, and so either they had to increase the machine requirements, or maybe we discussed the fact of increasing a bit the slot time that is inside the spec right now, that is 4 seconds, maybe 6 seconds, or 8 seconds, or something like this.

So yeah, they reported this problem, but there are many things that we could do. Maybe if we switch to the lighter XMSS variant, this will be better, once we have a proof. Maybe increasing the machine requirements to… a minimum of MacBook M4 would be beneficial. Let us see what goes with binary feed techniques, maybe it will be better as well, so… I guess there are a couple of stuff we can do. I… last week also, like, since you were not there, like. We discussed a potential other option, that is squeezing the performance. With the Collaber classical scheme. But it wasn't there, but my perspective is that we cannot, for example, reduce 2X the performance, or something like this. We can gain some stuff, but not 2x, or 3X, or 10X, right?

T. Wambsgans: Yeah, likely. Maybe, but it's not easy. Yeah. Okay.

Gajinder Singh: Yeah, so, we also still need to drop one PR that is a bit of optimization, in the sense to start, type 1 aggregation in the previous interval itself, so we will drop this PR this week.

Pablo Deymo: Yeah, I just wanted to add a suggestion. We noticed that, when… Doing a comparison of block… of block proposal times, or block building times. depending on the number of attestations, we saw, like, a two times increase, I think, between having zero attestations and only one attestation. I think we could, like, get a two times improvement there. If we don't agree the proposal signature to the rest of the attestations, and just have it, like. Like, an addition to the block proof. It's just something to consider, I think.

Thomas Coratger: You mean, sorry.

Gajinder Singh: American. Because I didn't understand.

Thomas Coratger: Yeah, maybe, can you repeat the end? Not sure I am.

Pablo Deymo: We currently aggregate the block… the proposer signature for the block with the rest of the sign… of the aggregated proofs for the… for the attestations. We could instead have them, like, separate, the block proof for the attestation signatures. And then the proposal signature itself. Since it's only a single signature, it doesn't add much, either size or verification time.

Thomas Coratger: Yeah, yeah. So you propose at the end, we have a block, and instead of having Be, like, only a single proof. For each blog posted on-chain, we have a proof that is 200 kilobytes, for example, plus the individual signature of the proposal corresponding to the block, right? That is… That is the number, right? No, no, DevNet 4 was, like, we didn't aggregate Attestation with different messages. I think that here the proposal is to aggregate everything, and just having on the side, not proposal, right?

Pablo Deymo: Yeah.

Gajinder Singh: I mean, yes, so these are… yes, this could be one of the resolutions that we can look for in terms of look… of running a large-scale network on commodity machines, but I think the better thing is just to increase the interval. Because, as such, then we would be testing the final, format that we are proposing for… lean spec hard fork, right? So, for post-quantum. So I think… This is, like, This is… this is one of the recourses that we can take, but then… we can… I think it is still better to increase the slow time. So that we can deal with it. As of now. I'm assuming that a lot of optimizations will… Bring down this time over…

Over a couple of months, or over a lot of months. To about less than half of… What it is, and then maybe even further.

Thomas Coratger: Okay, what I can propose is that… between now and the next week, we continue debugging process, and we start simulating DevNet 5. if we still have, like, the problem of, signature recursive aggregation, which I guess we will still have. What we can do is, inside the spec, we try to increase the bid the slot time. If that solves all of our problems, that is perfect. Everyone is happy, and we can start scaling and adding more validators. And then, later, within a couple of months, we can re-evaluate, seeing if performance are better and stuff like this, to decrease, again, the slot time. But this is something we can do. And in the meantime, it could be great to have in the discussion channel a list of points that are problematic for DevNet5. For example.

Clients not passing tests, or… a lack of interoperability between clients, or, I don't know, no finalization, or, like… and also the number of validators that we are using. Could be great to have some sort of bullet points, or a document, or anything like this could be great. So, yeah, if this is okay for everyone, and no questions, comments, or stuff like this, I think we are right. Perfect, guys. Then, see you next week, and don't hesitate to ping us, like, on the Telegram channel with any debugging stuff, yeah.

Shariq Naiyer: Have a good one, guys.

Thomas Coratger: Alright.

Mega | Lambda: I'm joking.

Gajinder Singh: Next one.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.