All Core Devs - Consensus #173

Transcript

Alex Stokes: Hello. Let's see, let's give, maybe just another minute or two, and we'll get started.

Okay. Josh, if you're helping with the stream, then I'm ready when you are.

Josh Davis: Stream's live, go ahead.

Alex Stokes: Okay, thanks. Hey everyone, welcome to ACDC number 173. It is issue 1874 in the PM repo. There's a link to the agenda that I just put in the chat. So… Yeah, today, just one quick housekeeping thing with Fusaka and the IPs, then we'll touch on Glamsterdam and Hegota. So, yeah, otherwise, it's a pretty short agenda, so yeah, let's get into it. To kick things off. Pooja has a request, to adjust some of the statuses of a few EIPs from Fusaka. I don't know if Pooja's on this call, but let me just grab a link to this comment. It was EIP7607, 77723, 7594, and 7918, and I think they just need to have, the status changed.

But yeah, if you were involved with one of those EIPs, please take a look. And yeah, we'll get that… Cut it up. Okay. Next then, we'll turn to Glamsterdam, and yeah, so I think the main thing for today is to discuss EPBS and the progress there. There was also a request to mention an engine API, issue, let's say, for BLs, but… First, let's have a little chat about EPBS. So… Yeah, I mean, maybe a good place to start would just be to ask, where clients are at. I know we had the spec changes that we were working through, even into the end of last year. And, you know, the next step now would be the EPBS DevNote 0.

I don't believe we have the 7.0, and yeah, I think it'd be good to have a conversation about how we want to get there. So, any CL clients want to chime in?

Stefan Bratanov: I can speak from Teku. So… we have, basically, we're passing the spec ref test from Alpha 4.1. Apart from the folk choice tests. So, we're pretty much in the same situation as we were before the holidays, but just passing the new specs. So, yeah, we're working towards, DevNet Zero. But, still not sure exactly. Of any timeline.

Alex Stokes: Cool, thanks. Any other clients want to give an update? Okay. Barnabas, you put some notes here on specs. Do you want to say anything else about that?

Barnabas Busa: Yeah, so you basically, dial down the specs to be 1.7.0-alpha.1 for DevNet0, and most likely it's going to be Alpha 2 for DevNet1. I included the CL spec PRs that are attached to each of these scopes, and hopefully we can discuss that. There's a couple of open PRs still, 4843. Fuck.

Alex Stokes: Yeah, I can grab them. It's 4843, and it looks like 4826.

Barnabas Busa: Yep, exactly. So the 4843, seems to be… pretty much good to go, and pretty much good to be merged in. And the other one is, really the questionable one.

Alex Stokes: Correct. And yeah, I mean, the way that you have it here, those would be in DevNet1, so we have some time to resolve those, and then there's a solid target for DevNet 0.

Barnabas Busa: Right, but, yeah, it would be very good to… to have some timeline discussion, regarding EPBS, because it's been delaying for a very long time, and based on new inclined updates, it just doesn't seem like it's going anywhere. I'm hoping we can, like, We can discuss specific deadlines for DevNets.

Alex Stokes: Yeah, I mean, I think we should get into it. One… Comment from Kini, just in terms of client updates. Yeah, Lighthouse is making good progress, no firm timing yet, but yeah, they're making good progress in terms of Lighthouse Network. So… Yeah, I mean, I think that is an important thing to get into here, is just the timing, so we can align on how we're thinking about things. you know, I think one place to start is… We had this meta of 2 forks per year. So, from where we are today, you know, I would be curious how clients think about this. a proposal I could just toss out there. you know, following that, timing of two forks a year would be Glamsterdam sometime in the summer, you know, between June…

July, possibly even August. From there, you know, especially, I think, if we can agree to a time there, then we can work backwards and figure out where everything else needs to hit. So yeah, I'd be curious for people's thoughts on that. I think sticking to this would be important, again, just to keep, Is anything predictability and regularity high? And I do think if we're targeting the summer, then, yeah, you kind of go backwards from there. It would be something like May-ish for testnets. Meaning, DevNet development is done around April. Meaning, you know, the beginning of DevNets, where we are now, needs to start progressing. And, yeah, I mean, if we want to make it concrete, we could even go ahead and say, yeah, you know, we need DevNet Zero, say, in a month, and from there, I think things will kind of fall into place.

But, yeah, I'd be curious, how people feel about this today.

Barnabas Busa: I think maybe it's not very clear what you're trying to communicate. Basically. It's suggesting that we could consider doing a EL-only fork in the summer, in case EPPS is, not able to… Shipping time. Is that correct?

Alex Stokes: Yeah, well, this is an option on the table, which is why I want to kind of have this conversation, so… Yeah, like, if we want to stick to two forks a year, and it gets to the place that, you know, really any AP is not ready, then we could just basically leave it out. It, by default, goes to the next fork. And we proceed that way. And yeah, that could be the case here in Glamsterdam. Again, I think BLALs and things have progressed quite a bit further along than EPBS, so if we stick to this mid-year fork. EPBS is not ready in time, that is an option on the table. Botus.

Potuz: Yeah, so I can't give you any timelines for DevNets, nor… not even Kurtosis runs, but people are talking about an 100% increase on the size of the block because of BLALs. I don't know how people think about that… without EPBS?

Alex Stokes: You're saying that you would have concerns without APBS, that it would be okay to make the blocks that big?

Potuz: My node today gets the median block over 2 seconds into the slot. If we double the size of the block. We still need to attest to it. Without a PBS, it sounds kind of crazy.

Alex Stokes: And this is just from the access list? this double N you're talking about?

Potuz: Yeah, as far as I know, they were talking about having a 100% increase on BLALs.

Alex Stokes: Yeah.

Toni Wahrstätter: I mean, this is true, but to be realistic, right, 100% sounds a lot here, but we are talking about 70 to 80 kilobytes with today's gas limit. So this is not even a blob. So I wouldn't… I wouldn't over-exaggerate it. Of course, on the EL side, data will be doubled. But, if you put it… Relative to the whole… to the total data throughput we already have. It's… It's negligible.

Alex Stokes: Ontario, you have your hand up.

Ansgar Dietrichs: Yeah, I mean, basically similar to what Tony just said, more just a bit general. Now, I really, really, really don't think we will ship an EL-only fork. I think that would be a really bad outcome. I'm very optimistic that, you know, we'll just hit the timelines on both sides. But yeah, because the question was raised, I think because we have this unusual flexibility this fall, because of the repricings, we can just make these things relative to each other, more or less expensive, data versus compute. And so, in this, like, just absolute, like, emergency outcome of having to ship a fork without the benefits, they're very important and very much looking forward to benefits of EPBS,

we would be able to do so, and basically just, end up having to reprice things slightly differently. We would be prepared for that, but again, to me, this is, like, a very, like, just… Only emergency outcome, I don't think that's at all in the… And main path here.

Alex Stokes: Yeah, I mean, I do think it would be preferable to keep things as scheduled. It is an option if it comes to that. Yeah, and so I think it's gonna be… Yeah, I think it'll be, like, tricky to make this a firm commitment today. At least going off, yeah, what Putin said, representing Prism. Kini here with Lighthouse, so… Yeah, at least from those comments, like… Possible, but yeah, they don't want to commit to this today. Any other clients have any input on this?

Saulius Grigaitis: I just wanted to note that I think we have a bit of illusion of two hard forks per year, because we did that last year. And we did that just because the PDAS was developed for a couple of years in parallel. And, we don't have such thing, now. We just started to, implement, EPBS, which is a really complex, thing, it touches so much in the client. So, yeah, I'm… I'm afraid that we… It's… for this year, it's just an illusion to have two hard forks per year.

Parithosh Jayanthi: I think that's fair, though, because block-level access lists also got started developing around the same time, and they are mostly ready to hit 2 forks a year. Same with repricings.

Saulius Grigaitis: Yeah, but I'm on CL side. I'm talking about CL side.

Parithosh Jayanthi: Well, that's more of a comment on scoping, right?

Alex Stokes: Yeah, I mean, I'm not proposing this, but you could imagine we change the scope of EPBS so that it's… A smaller thing to ship faster. Yeah. Barnabas has a suggestion here on a timeline, Let's see… so yeah, the BAL DevNets progress. EPBS DevNet Zero, yeah, essentially a month from today, which then, a month after that, we'd get to the next DevNet and keep things moving. This is a nice timeline. Of course, it needs to be possible. And, you know, I think what I'd suggest… I'm sure, Barnabas, you want to add some context?

Barnabas Busa: Yeah, so I would really, really, suggest that, we should do, like, a harsh deadline for the end of February, the 20th of February, for at least shipping EPBS DevNet Zero. If we are unable to ship EPPS DevNet Zero by the end of February, then I would consider pushing it to, the H-star fork, and basically doing a EL only and repricing changes, fork in… June, or maybe even sooner. And then you can still ship EPPS by the end of the year.

Alex Stokes: Yeah. It's an option. Any responses to that? So, if not, I think in terms of today, Yeah, it does sound like, based on where we're at today, I would agree here, the post is saying, like, end of February, or at least, you know, not today, but with another sequence of calls. We'll have a clearer idea. I do think we should all have in mind, roughly, the schedule, again, just to keep things moving, so… what that means for today is, like, you know, do our best to get to DevNet Zero, towards the end of February. And, yeah, I think… Given where we're at today, we'll know a lot more in the future, and we can have more clarity at that point in time.

And to be clear, there is this option that, you know, especially if we have a bunch of other things ready outside of EPBS, there would definitely be the option to explore I'm shipping, quote, an yaw-only fork. So… Let's keep that in mind.

Parithosh Jayanthi: Are there any implementation blockers or spec topics that need to be sorted out sooner rather than later that would help?

Potuz: Can you… can you specify what you're referring to, Pari?

Parithosh Jayanthi: I was asking, do we have everything required for implementation, or is there still gonna be… something changing between now and DevNet Zero.

Potuz: No, I think it's… DevNet Zero is already, spec'd out. And I think, finally, the spec is frozen in that sense, so we can actually now start… well, we are already coding, and we are already merging. So, a couple weeks ago, we actually started coding for this.

Parithosh Jayanthi: Perfect. Do you guys need anything from the testing team, or from PandaOps?

Potuz: Not for now, but in a couple of weeks, we'll be asking to have, like, our justice runs, I guess.

Parithosh Jayanthi: Okay, sounds good.

Alex Stokes: It's blocking on the kurtosis?

Potuz: Oh, there's no blocking, and we just need to code and merge.

Alex Stokes: Okay, yeah. And even, yeah, just going off of the DevNet Zero spec, there's the consistent specs here with the release, and there's one extra PR, it looks like, so… That looks pretty straightforward. So, yeah, again, that's what I would say for today. Let's all try to keep this… next milestone in mind of about a month from now, towards the end of February, and… yeah, let's see how it goes. Does that sound good with everyone? Enrico?

Enrico Del Fante: Just, just a final, comment, unrelated to APPS, so I want to just take one minute to share a little description of what happened to Teku Node's, two weeks ago, around January 13, so if we are at the end of the call, I just want to share these things, so… Is this the good time? Is there anyone.

Alex Stokes: Oh, yeah, yeah, yeah, go ahead.

Enrico Del Fante: Okay. So, just a little, summarization of what happened. So, EEF run around E13. PM, UTC, a stress test on the mainnet, causing… blob usage to spike, like, 2.5x. Consider the average of… compared to the average of the moment. So, we have been reached by, several clients… saying that Google's falling off and, producing empty slot logs. And we started a very long analysis, and we think we found some root causes of this. Essentially. This was happening only on some full custody nodes, not everyone, not every node. And the spike of blob usage caused, essentially columns to double, more than double in size, so overall data written to database doubled.

And, we… we hit some kind of threshold in the way we used LevelDB that was causing transaction rights to Slow down a lot. And this was causing a chain effect on our client, essentially. Fully… filling up some queues on the database side, causing that availability to not being declared available. Quick enough, even if we have already received the columns over the network, and so blocks 1 won't be… imported, and… all things off. So, we started… when we realized that LevelDB was the root cause, we started saying around, can you please switch to RoxDB if you're facing these problems? And we are working on several mitigations, and we're also moving to RoxDB as default database and from the next release.

But, yeah, just sharing the updates on what happened. I don't know how many was aware of this. So we are… This is a reason why also EPBS slow development was a little bit slowed down lately on our side. We were dealing with this analysis and issues. That's it.

Alex Stokes: Okay, thanks. And, yeah, any takeaways? I mean, Barnabas in particular has a question.

Enrico Del Fante: Oh, yes.

Alex Stokes: If we should wait with the blob testing?

Enrico Del Fante: We are working on some, diff… some… some approaches to mitigate everything, so… We could… Try to recreate the same problem, even if we synthetically recreated the same scenario. Locally, by simulating a very slow transaction on database that actually triggered all the chain effect that we saw on customer nodes, so I think, like, we know what the problem is now, and it's pretty internal and low level on Teku. So, we could run another spamming test again, but I would like to… maybe having also all the mitigations and improvements already in place, to also being able to compare those mitigations in parallel, and see that there was

actually solved. So, maybe we can organize later, when everything is… is done.

Alex Stokes: Okay, thanks for the update. Okay. Anything else on EPBS? Again, just to play that back, it sounds like we're in a good place with the DevNet Zero specs. Clients are busy implementing, and yeah, it would be good to see the DevNet as soon as possible. At this point, I do think we should kind of just take it call by call and see how things progress. We should have a rough deadline in our heads of about a month from now. And, yeah, we will navigate things as they go. Okay. Then, Tony had a quick shout-out on this PR, concerning the Engine API and BILs. Tony, would you like to say something?

Toni Wahrstätter: Yeah, thanks, not really much to say about it, just as a heads up for CL clients that we changed something at the engine API previously. It required, like, two calls. to the EO, When it comes to helping other peers sync, and this will now just need one call. Because the block lab access list will just be attached. To the, execution payload. to the execution payload body. So it will be… it will have an execution payload body V2, And yeah, just please, check it out. Previously, we only used Lodestar for the first Defnet, and for DEFNET2, I think the plan was to use multiple clients already, and different clients handle it differently, as far as I know, so…

Please have a look, and… those clients that are ready for DEFNET2, we can then already use them. Thank you.

Alex Stokes: Great. Yeah, and us would just have people review the PR.

Toni Wahrstätter: Yeah, review it and implement it.

Alex Stokes: Create… okay. Okay. So, yeah, everyone take a look. Add that? Anything else for Glamsterdam? I think that was all we had on the agenda. Okay. Next then would be Hegata, and again, yeah, Just to give a summary of where we're at there. So, we've started the headliner proposal process for this fork. So far, there's been a proposal on Ethan Magicians, at least a post for this, for FOCIL. And I think a new one since last time would be Encryptum Impulse, which I think, depending on how they're doing it, is probably more of an EL thing. In any case, we have until February 4th.

For proposals to be submitted, it would be nice if champions of these proposals come to some ACD by the 4th of February, and give, you know, a very quick sort of pitch or overview of the proposal. Yeah, then from there, we'll go to the headliner selection process. And I think that was all I wanted to say there. We still have some time before the next phase of that. And with that, I think this was all we needed to cover today. Is there anything else we should discuss?

Barnabas Busa: I might have something, so… could we get, maybe, every CL team to, have a… have a version for the BLAL DevNets as well. I know that we currently have Lostar and Lighthouse only. And I think it would be a good, to… to be able to basically run all the sales, pitfalls as well.

Alex Stokes: And concretely, this would be a slot non implementation, just so they can join the DevNet?

Barnabas Busa: That plus BLALs. It should be fairly small.

Alex Stokes: Oh, right, yeah, yeah, okay.

Barnabas Busa: Stuck out. Yeah, Prism… Prism is not working. There was a working branch that… might work for .NET0, but it no longer works, for sure. So it would be good to have a .NET2 spec ready for all the different CLs. It should be a one-day effort for most people, I think. It's not a huge lift. And then maybe later on, we can also iterate additional fields on top of that branch, and because we still have a few CL-only changes, If APVS is taking too long, then maybe you can include those CR changes on top of the ball.nets.

Alex Stokes: Yeah, so that was… So you said, presume there might be an issue with the branch right now? And then we'll let Techu, Grandine. Who else am I missing? I missed the first two you said that were already there. Nimbus.

Barnabas Busa: Nimbus, Taku, and Grandin. It would be great if they also took, like.

Alex Stokes: Okay. Some of them are on this call, so… Can we get an acknowledgement? Okay, we got a thumbs up, at least, from Svalius. Okay. Ari hasn't asked here about partial salt proofs, that would be worth talking about, if the relevant people are here. Is this mostly implementation and clients, Perry, or is there something else to talk about?

Parithosh Jayanthi: Yeah, an implementation status of plans, because as far as I knew, partial site pools is something we also plan on shipping this year, so… I think it makes sense to discuss where we are with that.

Alex Stokes: Is any client looking at this?

Barnabas Busa: Yeah, so I was just checking if maybe Raul was on the call, but… I cannot seem to find him, so I can maybe give an update. We are launching a DevNet for the partial stale stuff, We launched it yesterday, but the clients are still implementing the features. We have a Lighthouse implementation and a Prism implementation. They're, like, half-baked solutions at this point. We're still missing some, Metrics, but it should be, getting close to interrupting. And we are waiting for a Tecmo and Lodestar implementation, and we haven't heard from IMBUS yet. But it's work in progress.

Parithosh Jayanthi: And is the plan that as soon as someone has it merged, they ship it, or will this happen at a fork boundary, or EPO boundary, or something like that?

Barnabas Busa: This is still, under consideration, so we want to roll it out for DevNets, then, roll it out for testnets, and then after testnets, we can see if we have any other P2P issues, with clustering. If there's no issues, then, clients are free to roll it out at, Whenever they roll out their next, stable release, it's…

Alex Stokes: Alright, Potis, your comment here was about… These partial salt proofs?

Potuz: Yeah, we have… but Marco is pushing PRs, and also some internal members are just working on this, and Casey, I think, is also working on this.

Alex Stokes: Okay. Yeah, in terms of moving forward here, it might be good to, like, point to some type of champion, and I think that's… Yeah, likely either Raul or Marco, so yeah. I can follow up with them and try to get them on a future call, and we can make sure we keep moving that forward as well.

Barnabas Busa: Yeah, like, we cannot ship BPOS3 without it. That's pretty obvious, I think. So, we're gonna need to have every client implemented before we can ship PPS3.

Alex Stokes: Makes sense.

Barnabas Busa: Could we also get an update, maybe, on 7688?

Alex Stokes: This is the Stable Containers EIP.

Barnabas Busa: Yep, because most SSE libraries will have to update that. That might also be a significant effort.

Etan Kissling: Yeah, so, tests are… ready in consensus backtest. Nimbus implementation is ready. From Lighthouse, I heard that they are ready, and Lodestar results are ready. Prism, Grandine, and Teco are aware, but they are still busy with the other stuff, from what I have heard from them.

Barnabas Busa: Right, is it fair to assume that the implementation of the SSD containers is… can be done parallel to EPBS, and it's completely orthogonal?

Etan Kissling: It's a pure library change, it doesn't affect the consensus, client.

Barnabas Busa: Is this something weird.

Alex Stokes: kind of…

Barnabas Busa: Didn't the… didn't the ERP touch some of the types, though?

Etan Kissling: Yeah, I mean, the 7688 then applies the library changes to the types, but the library changes itself can be developed in parallel.

Alex Stokes: Right. Okay.

Etan Kissling: It's just like an annotation that you have to put on the type, and then it hashes the new way.

Alex Stokes: Potus, your hands up.

Potuz: Yeah, so, it's true that, we can, after we… added all the structures, we can just change the ZZ structures on the states. The only worry that I have is if it will affect our in-memory structure for the tree that we need to keep, because it changes how we hash things. I've been told that this is not so much of a change, but I still… I mean, bugs on the rehashing on the tree are always subtle to debug, so that'll be my only fear on this EIP.

Alex Stokes: Yeah, I mean, if we want to make progress here, it'd be worth someone who's not working, say, on EPBS to… de-risk that point, POTUS, so someone who can move ahead about implementation, and yeah, you know, even just a little prototype, showing that that's not an issue would be useful. I do think more generally, I'd rather us focus on EPPS and getting that to a good place, before the other CFI DIPs. It is good to get a status update on this, though.

Barnabas Busa: Anyone has looked into any other CL EIPs, by the way? Because we still have a few more? In terms of, like, implementation difficulties, or…

Alex Stokes: Yeah, I would guess not, but… I'd be keen to hear. Okay, we gotta know. From Teku, which is understandable. I imagine everyone's just very focused on EPBS at the moment. Okay. Anything else? Otherwise, we'll go ahead and wrap up for the day. Okay, and thanks, everyone. I'll see you next time. Goodbye.

Potuz: Mobile.

Enrico Del Fante: Bye.

Justin Traglia: Sorry, everyone.

Justin Florentine: Also, plus encrypted mempools is gonna be fire.

Toni Wahrstätter: Bye bye.

Ansgar Dietrichs: Very important.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.