Fast Confirmation Rule Breakout #10

Transcript

Mikhail Kalinin: Hello, guys.

Justin Traglia: I don't think.

Nazar Hussain: Morning, okay.

Mikhail Kalinin: Yeah, let's Wait for, I don't know, four more minutes and get started.

Hello, Goma.

Nazar Hussain: Thank you.

Mikhail Kalinin: Yeah, I think we can go ahead and get started. So welcome, everyone, to the false confirmation rule breakout number 10. Here is the agenda. Yeah, it's likely to be. a short call. And, we will start with, client updates. great. Let's Yeah, let's start with Grandine.

Mercy Boma Naps-Nkari: Hello, so, there's really not much, updates, since the last, call. But I already pinged, Thomas on the group for review on the PR itself, so… Hopefully… Very soon, we'll get it with you.

Mikhail Kalinin: Cool, Thank you, and to confirm, you pass all the, Reference tests that we have so far.

Mercy Boma Naps-Nkari: For the FCL test, yes, but the general tests, I think Grandine is not passing all the tests for the spec test. But for FCR-specific text, yes.

Mikhail Kalinin: Okay, cool. Alright. Thank you. Yeah, that's great. Looking forward to reviews from you. From you. Team. Next, Lodestar.

Nazar Hussain: I agree. We released the first confirmation rule, and our public release is 1.44.0. And… During our testing, we found some performance edge cases around the people boundaries. And there was some research, around the April country, so. We released it to the public, but with some… Banner or kind of a notice that there's an experimental feature, so. If you can't be joined, rely on medical counselors or something. This feature, yes. And in addition, I found one fix for… Performance related to this EPOC Foundry edge case. And this fix is working and deployed on our feature group for, like, 4 days now. I deployed on Friday. And it feels like everything is more smoother, and there's no reset afterward.

Since I deployed this fix. So we will be testing this fix further, and then, releasing this fix in our upcoming release.

Mikhail Kalinin: Yeah, awesome. Yeah, that sounds great. Maybe you want to share more of the — yeah, sorry.

Nazar Hussain: Yeah, I did some analysis on… One of the reset cases. And I shared with you. If you think, like, it's a good enough case study for anyone to share it, then I can share with everyone via Telegram.

Mikhail Kalinin: Cool. Yeah, I think makes sense to share it. And yeah, it's… Do you want to share more detail on what was the issue that could cause resets before?

Nazar Hussain: So, like, the first observation that I had, and first… the impression that I've had was that because our safety threshold It, like, keep increasing with every slot, Let's say we had a slot… 28, on index 28, which was confirmed. Boss confirmed, but… When the head moved onto the slot 31, on The required… spoiled for… Reconfirming all the slots were… higher, but Forms were not received enough for the last few slots. So when we crossed the epoch boundary, we went back to reconfirm all the plots. And during this, we had to reset and fall back to some other, like, lower blocks to confirm. And so far, my analytics, like, first impression was that this is a performance edge case.

On the Epoch Boundary and FCR was taking too much time, but later, while digging in further. gossip sub-channels and other. I found out that this particular node had low… Like, lower the ceiling of our test station, so, compared to the other ones, so… It did not receive enough votes for the vlog 29 and vlog 30. And that was one of the reasons that, It had to reset and solve it. Just some roadblock. This analysis is in one of our documents that I will share with everyone to tell you.

Mikhail Kalinin: Yeah, I see. Great. So. So it was about the node did not receive enough attestations in one in the last or one of the. Last slots of an ebook, right? Yeah.

Nazar Hussain: Yeah, it was slot 30 and slot 29, which received very low loss.

Mikhail Kalinin: Got it. And you said that this was just with one node, an issue with one node. And not the others you have observed.

Nazar Hussain: Yeah. Like, I observed other notes. having the reset on different times, but not in this particular case where I was doing the our search. But on this particular time scale or time window, I checked other NORs and they had different and higher attestations received compared to the particular NOR which I was just waiting for. This, like, my analysis was that this was a network effect. That we did not receive enough. support for this particular chain which was moving with the head and that's why it has to fall back. Reset and go back. To the finalized, and then started confirming the finalized. Up to the… let's see, it was slot 25 or 24. So it falls back around 4 or 3 blocks.

Mikhail Kalinin: Yeah, I see. And one another related question. So is it like because. something happens with the nodes that you observe? I mean, like, you've mentioned that you saw also resets on other nodes as well, but at a different time. So the question is, basically, is it, like, a general problem with the network? that happened at that epoch, or those epochs? Or is it just, like, with, with the nodes you're… you run and you… you look at?

Nazar Hussain: So one common effect for all the nodes were that most of such incidents happen around the epoch boundary. And, and again, it's not, It's not the issue with the… specs, but it's just the case that on the EPOP boundary, we go and reconfirm all the blocks from the last gesture I checked on, so… I, I think it's, I'll see. Based on this poor performance, I would say it's still, like, doubling. Closed, or very narrow. Edge case, where you actually have a very low traffic, or some problem with the network. But I cannot, yep. conclude that this is something related to the FCR. If you have a low attestation, then definitely we will see this effect.

Mikhail Kalinin: Yeah, yeah. And yeah, this is what I'm trying to understand, whether it's like a common problem for the mainnet. I mean, like, or whether it's a problem with the… The nodes that you run, Or, you know, like other clients, it would be good to hear from other clients if they. Observed anything like that.

Nazar Hussain: So within our server stack, I think… It was just those two particular nodes, which was… I'll have more research. or… But, Yeah, other than that, like, since I deployed this fix from Friday, I've never seen a single reset. On the main deck.

Mikhail Kalinin: Okay, cool. Great, so I think… yeah.

Nazar Hussain: There's one other… one other observation I would like to share is… It's the actual behavior during the node syncing. Okay. Maybe this is something we should also, like, think on other clients as well. So, on one of my notes, which was syncing and taking too much time, and it was, like, out of memory. For some reason. I saw node reset count increase, 250 plus.

Mikhail Kalinin: Oh, yeah, I see.

Nazar Hussain: So, like, 250 plus was, like, a very weird number, like, very high number, so… and this keeps happening, like, several times, because… north started sinking, and then it crashed, and then it started sinking again, and then it crashed. So my graph was, like, kind of a… Jigsaw, where it was going up and down, up and down, and it was going up till 250. So, during this.

Mikhail Kalinin: Yeah.

Nazar Hussain: In the process, I was thinking that we might… Like, fine-tune this, like, this particular edge case that… We should not trigger the FCR. During the shrinking.

Mikhail Kalinin: Yeah, I think it makes sense to not trigger the FCR. Unless a node is fully caught up with the head.

Nazar Hussain: Yes.

Mikhail Kalinin: And yeah, and then just start from, you know, from the closest restart point. After after the sync is finished.

Nazar Hussain: I was working on this one.

Mikhail Kalinin: Yeah, great. I think it makes sense. We had a discussion with Nimbus team as well about this. And, yeah. come to agreement that makes sense to enable FCR only after node is fully synced.

Nazar Hussain: Good.

Mikhail Kalinin: And related to… To the issue with the participation. I think we, yeah, I think we should just ask. In the chat. The Telegram group, if I went here. Anyone else observed the same? I mean, like, other clients. Okay.

Nazar Hussain: Sure, I will post some questions, okay? I'll also share my analysis.

Mikhail Kalinin: Yeah, yeah, great, yeah. Yeah, that's awesome. Anything else, Nazar?

Nazar Hussain: No, I'm just sorry for myself.

Mikhail Kalinin: Yeah, thanks for the great update. Looking forward for you to release those fixes.

Nazar Hussain: Okay.

Mikhail Kalinin: Cool. Any other client updates? Yeah, I have some. Terence basically posted it in the, In the call issue and, yeah. They are blocked by their passing inspectors. They have a limitation, but they're blocked by human review. At this moment. So… Also… From, from, from Lighthouse, There was a regression that we mentioned in the previous call. It seems they have, resolve this. The problem found the issue, and Have a fix for this, and they do mainnet runs. To… to check, and… and as far as I'm concerned, they… Don't observe the resets anymore.

On their notes, which is great, Also looking forward for them to to get it released. And from Teku, they. They are certain their implementation as well, the production ready implementation. So that's I think that's all for the client updates. Barnabas, what, what do you exactly mean?

Barnabas Busa: Like, how can we, like, if I would want to roll this out on concert on the net, whatever, like 6. Let's say that the clients have support for that. How could we verify that their implementation is good?

Mikhail Kalinin: You mean like that they support the FCR?

Barnabas Busa: Yep.

Mikhail Kalinin: or that the FCR is actually making some progress.

Barnabas Busa: Oh.

Mikhail Kalinin: on the chain.

Barnabas Busa: Yeah, so like, I want to verify it on chain that it's making progress so that I can.

Mikhail Kalinin: Yeah, yeah. There is, yeah, one thing that you can observe is the event stream. The pause confirmation event is on the event stream for a beacon node. If FCR is enabled, it should. Basically producing output every slot.

Barnabas Busa: Okay. Should probably then add this feature to Dora to be able to visualize it.

Mikhail Kalinin: Yeah. Okay. And also… At. From what I know, Sam is running, This lighthouse or maybe other clients as well. And, yeah. You can ask Sam about it.

Barnabas Busa: Does any of the clients support FCR on top of latest Gloas?

Mikhail Kalinin: That's a good question. I don't know. the answer.

Nazar Hussain: So you will post the question.

Barnabas Busa: The question was whether FCR on top of Gloas is something that you guys have.

Nazar Hussain: In Lodestar, I merge my PR in the morning. Which one are they making? Our spec tests for the FC are compatible with the Gloas. Hopefully there is no problem.

Mikhail Kalinin: Arnabas, are you in the Fast Confirmation Rule Telegram group?

Barnabas Busa: No.

Mikhail Kalinin: Maybe I should add you there? Do you want to?

Barnabas Busa: Okay. Feel free to add me, but Yeah, I'm not really interested in… The… like the implementation details. I'm mostly just interested in whether we should be able to test this or not. And, like, if… If we subscribe to the event stream and we visualize it somehow. What kind of application would be able to use this, like, right away?

Mikhail Kalinin: I see. Okay. And yeah, yeah, the event stream is for a beacon node. And, basically, The safe block is… is now… yeah, if the fast confirmation is enabled, the save block will point to the most recent confirmed execution payload. That's the other, part. This is where, like. This is where… Blockchain data consumers basically can. use it. This is how they can use it in their applications. Yeah, it's fully merged. Yeah, it's in the consensus packs. And, another thing I wanted to mention is that. Some clients, like Nimbus, enable the FCR by default. Other clients, as far as I know, use a feature flag, so you have to Pass the feature flag to enable FCR.

Barnabas Busa: Did anything break by enabling this? This is just a new event stream, right? So what?

Mikhail Kalinin: It's.

Barnabas Busa: Indeed.

Mikhail Kalinin: Yeah, so yeah, it's an event stream. The event in the event stream and the safe block hash. In the… book choice updated. Engine API will will point to a most recent block like more recent block than it is pointing now because now it's justified. With FCR, it will be a fast confirmed block. Yeah, it's, it's, it's definitely worth trying on, on Gloas because in Gloas we. We have a payload, the payload, before Gloas, the payload from, from the previous slot could be confirmed. But, since Gloas. The payloads of the previous. Slot will not be able to get confirmed. So we will confirm the payload of the slot before the previous one.

Alright, guys, yeah, let's, let's probably, discuss this in Discord. After the call, I mean, like, sharing branches and all this stuff. Well, let's work. For you all. That's awesome. Barnabas, do you have any? Yeah, yeah.

Barnabas Busa: Do we want to ship this before loss on each sale? Or what's the… Yeah. What's the expectation here? Yeah?

Mikhail Kalinin: Yeah, the expectation is as soon as it's ready, it's fully compatible. It's actually compatible with all previous forks. So and it should be compatible with Gloas as well. But yeah, it's a good point. We must test these in Gloas DevNets. That would be very valuable.

Barnabas Busa: Yeah, I mean, usually how we go with testing is if you want something in like we could get keep on that. So that's why I'm asking it, because right now the current scope of Gloas is already pretty big. So if we're gonna be pushing for Fcr implementation also on top of loss in order for it to be pitting We lost around the time. we do the main network, then I think it would be very good to start testing it on the DevNet.

Mikhail Kalinin: So it's, it's not, related to POS. It's not like tied.

Barnabas Busa: I know, I know, I know, I know, I know there's no, like… consensus-related issues or anything like that, but, like, it would be nice to know the current scope of the fork, even if it's not a breaking change for anyone.

Mikhail Kalinin: Yeah, I see.

Barnabas Busa: So that the client teams can also make sure that they allocate a certain amount of resources for such features. Especially if they're expected to ship this before close.

Mikhail Kalinin: Yeah, yeah, I think, yeah, yeah. Actually, time allocation was one of the main issues for FCR implementations. Yeah, but I think it's kind of in parallel. Anyway…

Barnabas Busa: Yeah, sure. But if you have a team of five and everybody's working on Gloas and nobody's putting time into FCR because they don't assume that they need to ship it before Gloas, then It just won't be done.

Mikhail Kalinin: Right. That's okay. Totally true. Boma, did you have a… guys, I saw your hand up. You want to ask?

Mercy Boma Naps-Nkari: You know, I wanted to, asks about, like, is there, like, a specific timeline for when PR should be reviewed, and then… so that clients can start, Each band of can start testing or something. That's. or my question.

Mikhail Kalinin: I see, yeah. You mean like the test, the comparison test between the implementations?

Mercy Boma Naps-Nkari: Yeah, okay.

Mikhail Kalinin: Yeah, yeah, I was asking Sam about this, and he said that they basically can do One more, or even, yeah. More test testing on that. But yeah, I would. I think that we need to wait for Lighthouse and Lodestar to Cats. All the things released. And then, yep. After that, if you're… I don't know how much time we expect these to happen. Like. But… If your team can allocate resources as soon as possible, that'd be great. But maybe in a couple of weeks so we can coordinate on the next call, next FCR breakout call. Just coordinate this testing.

Mercy Boma Naps-Nkari: Okay, I'll try to, like, see if they can review, because I don't want… us to test, and then there is a review, then I make a change, and then… After the change, there is no longer… we are no longer testing it, which we don't know if something broke or something is missing.

Mikhail Kalinin: Yeah, I see. Yeah, makes sense. I, I also think that, For this testing, we could use a branch, not… even if it's not merged into master. All right. I think it's all for the client updates. No, please let me know. Yeah. And yeah, let's move to the spec and testing updates. It would have, there are no spec. Related things, there is only this PR with, a bunch of, new tests. It's… it's been merged, there are tests for some edge cases, and, With this PR, like the manual test,

Suite is for FCR can be considered as ready. So we covered all the cases that we have planned to. Also, I tried to. I tried to… yeah, a few calls ago, There was a discussion about whether we should. have a mainnet, a test with mainnet configuration. I try to generate it, to basically generate test vectors with mainnet. Configuration for some of the tests, and it took a lot of time. It was like about 3 hours for just one test, which runs for, you know, for maybe 5 or 6 epochs. So I think it's a big issue for test generation and. Yeah, it would require faster. Oh. spec to actually produce those test vectors. But I think we are pretty much fine with the minimal.

Preset. All right. That's all for spec and tests. Any questions here? Okay, then metrics and APIs, there is a metrics PR that was merged, Which is also great. I'm Yeah, just wanted to share that this PR has been merged, and… Yeah, it would be great to see those metrics implemented across all clients. Oh. So we can do more. Comparison after… After release, it's on the mainnet. Okay. and yeah, now to the research updates.

Roberto Saltini: Yeah, so, we… Finished reviewing the proofs, the algorithm, they'll check out. But while reviewing the proofs, we noticed that The pseudocode in the paper has some difference from the Algorithm as specified by the Python specs. And so, we just need to add an appendix to the paper explaining that these What these differences are, and… Showing that, that don't affect the correctness of the protocol. Shouldn't be too long.

Mikhail Kalinin: Great. So is this basically one thing left before? releasing the paper.

Roberto Saltini: Sorry?

Mikhail Kalinin: Yeah, is this just the only one thing that you have mentioned now? Is this the only one left over? Before we can publish the paper.

Roberto Saltini: Yeah, this seems to be… The only one. Thing that is left over.

Mikhail Kalinin: Yeah, sure. That sounds great. Thanks, Roberto.

Roberto Saltini: Who else?

Mikhail Kalinin: On the research. topic. I would like us to think about. The… Late block improvement. That we did not introduce because of… Some proof work, Was not, was kind of hard and difficult to. come with a proof to the, the proposed solution that we had before. So why, why I'm, like. I think it's valuable because with this proposed solution to recover from late logs, we observed like 1 or 1.5%. But… Performance increase. This was observed on a prototype in Teku. And, comparing to other clients, Teku, yeah, confirms. Oh.

Like one slot confirmation in Teku happens. More frequent than comparing to other clients that do not have this implementation because, sorry, this optimization, because this optimization is not in the spec, basically. So we should think about whether we want to put some efforts into that. This is not only a Paperwork, not only a proof work, but also will be a spec work and tests and climate implementations work as well. But this change should be relatively minor. At least it seems so.

Roberto Saltini: Yeah, I mean, from a proof side of things, I think could be. Quite time-consuming, too. Prove that this change is correct. yep.

Mikhail Kalinin: Yep.

Roberto Saltini: So, and then, then, you know, that Two time. It will take time from other dedicated time too. Work on Future Forks, like on the Casper Consensus, for example. So. That is my concern. You know, in terms of… yeah. Whether this is worth it.

Mikhail Kalinin: Yeah, I see, yeah. Boma?

Mercy Boma Naps-Nkari: Yes, I wanted to ask, is there like a particular link to get the details of this optimization? So I'm not sure if you do understand.

Mikhail Kalinin: Yeah, sure, I'll try to find it. Yeah, I'll try to find a place in the code in Teku and share it in the chat. In the group, in the Telegram group. Yeah, Roberto makes sense. I think, yeah, we should first on the research side, think about whether we want this change and how much time it will take to. To get this done on the proof side and then decide basically whether it makes sense to. To make it.

Roberto Saltini: What is the improvement? What is the improvement that we we get? With this change in terms of performance?

Mikhail Kalinin: It's about 1.5111.5%.

Roberto Saltini: More blocks confirmed in one slot, you mean?

Mikhail Kalinin: Yes, yes. Yeah, I think it's pretty valuable.

Roberto Saltini: Yeah, no, I mean, yeah. I mean, I think like the spec change from what the record wouldn't be. Massive. Yeah, hey. Yeah. Nice.

Mikhail Kalinin: Doesn't seems massive. All right, I'll post more about this improvement in the group. All right. Are there anything? We want to discuss today. Great. Thank you so much, guys. Great progress. And it seems like all the things are. Conversion. And yeah, see you in a couple of weeks.

Roberto Saltini: But yeah.

Barnabas Busa: Peter B.

Mikhail Kalinin: Bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.