All Core Devs - Testing #62

Transcript

Barnabas Busa: I'm gonna wait a couple of minutes so that everyone can join. Okay, I think we should be good to get started. Rakush, can you please start the stream?

Akash | ECH: Real life.

Barnabas Busa: Thank you. Welcome, everybody, on All Core Devs testing call, number 62. I'm just gonna post quickly the agenda for today. It's a pretty light agenda, so hopefully we can finish ahead of time. First of all, happy birthday to the Beacon Chain. We just celebrated the fifth year anniversary of the launch of the Beacon Chain network. I think that's a pretty big achievement for everyone, that are on this call and not on this call. So, well done. Regarding Fusaka, we did, main initiative work last week. Main Initiative work 1, was a success, we didn't find any, obvious bugs in there, so, I think we are very safe to, go into Mainnet for Fusaka Fork on coming Wednesday.

Does anyone have any mentions regarding Fusaka? In case not, I guess we will see how it will draw on Wednesday, but I think we should be all good to go. Okay, let's get into Glamsterdam, then. Maybe, Stefan, you can take over regarding the BAL DevNet Zero update?

Stefan: Yeah, sure. Can you hear me?

Barnabas Busa: Yup.

Stefan: Perfect. We're making a lot of progress. I think we have a lot of clients that are now passing most of the execution spec tests. So that's really nice to see. Regarding the DevNets, we will probably be able to start one very soon. I'm still running some Kurtosis local testnets and running into some issues there. I've already, created a PR to the execution specs with one issue I found in Nethermind with, gas, memory expansion. how it's accounted, and otherwise, I'm… we're looking into a pre-compile issue also right now with Nethermind, but after that, I think we should be able to have 3 clients that are passing most. Maybe it would be good to get some client updates as well.

Barnabas Busa: Yeah, that would be good. Is any EL teams ready to give us an update on their BAL development?

Marc Harvey-Hill: Yeah, for Nethermine, it's going well, passing 100% of the tests now. And just discussing with Stefan that issue you mentioned, which, doesn't seem to be covered by the test at the moment.

Barnabas Busa: Thank you. Anyone else?

Dragan Rakita: Don't miss out.

Dustin Brody: actually passing… -Oh.

Dragan Rakita: Go, go for it.

Dustin Brody: Nimbus is passing all, but I think last I checked, 5 or 7? I forget which number of the tests. So, yeah.

Dragan Rakita: From Reth side, we had… we… failing some withdrawals tests that I need to take a look. There should be it.

Stefan: Is Geth here?

Barnabas Busa: I guess not. Nobody from Geth or Besu? Yeah. Or is anyone from Besu around? In case not, does anyone have anything else regarding… BAL devnet? Yep, Spencer? Hey.

Spencer Taylor-Brown: Yeah, I just wanted to say that it might be worth, Us on the testing side running all the static tests, so essentially the… to tests from Ethereum tests, refilled for block access lists. I don't know, like, the fire has just been put out, essentially, for all the tests right now, and clients are passing, but… it might be worth… Also, getting all the clients to pass those tests for block access lists, too. just thought I'd say that. But also, if it's a small dev net, it could also be worthwhile getting that up in the air.

Stefan: Does this also include, like, sync testing, for example?

Spencer Taylor-Brown: No, just, essentially, extra… extra tests, like, there might be some weird niche… niche case that, it could uncover.

Stefan: Okay, but I think that's it from my side for buck-level access to this.

Barnabas Busa: Okay, thank you. Regarding APBS, is anyone around to give us an update?

Justin Traglia: From the spec side, we're still working on adding, test cases for get expected withdrawals. We have an update… sorry, we have a breakout call this Friday. I'll post the agenda right now. I might try to look into refactoring, get expected withdrawals to make it a bit cleaner this week. And I guess, just whatever updates client teams have. Regarding implementations.

Barnabas Busa: Regarding the trustless payments, I think, if anyone else has any concerns, then this would be a call to bring it up, right? Otherwise, we're gonna be proceeding with the trustless payments, for sure.

Justin Traglia: Correct. But, keep in mind, the final decision on this won't be made this Friday. We're gonna save that for… the next ACDC call, so, like a week and a half from now.

Barnabas Busa: Oh, okay. I thought that it would be made… the call would be made on Friday already.

Justin Traglia: I mean, I don't think it should. Not everyone joins the breakout call, and it's a pretty important decision to make, so… I'd feel more comfortable making it at ACDC.

Barnabas Busa: Okay. Any client teams have an update, regarding implementation? Readiness? Okay, from Lone Star, they have the… Please progress on state transition. I'm passing the 1.6.1 specs. It's great. Anyone else? You guys don't need to be so shy. Feel free to speak up. In case no other updates, we can move on to the next topic, which is gonna be further gas increases.

So, Minnet just hit 60 million guests, last week. Which is very good news. And, I think the next step forward will be targeting 80 million guests. Do we have any updates regarding how we can achieve 80 million? And, What kind of repricing games we're gonna be playing for that? Yep, listen.

Jochem Brouwer: Yeah, I wanted to mention that for the 80 million guests, we have the problem with the receipts. So this problem is that the def P2P network, it can It has this high limit of 10 megabytes of message limit. And we have a problem with the receipts, because at 83 million, we have this theoretical limit of actually going over this. Well, Kamil was saying, hey, we switched to an issue for 85, but, well, I would say 80 is getting pretty close. So, I would just mention the EIP which actually solves this thing. It's EIP-7975. well, one of us, I would still say that 83 is also very close to it. I just want to mention that we are getting very close to this limit, and this EIP solves it.

So, I think that we should start to implement this EIP, or something else which solves it. So, the EIP itself, it will just split up the def P2P messages into partial messages, and then we don't have this limit where we cannot send. The receipts which go to go over 10 megabytes. Yep.

Barnabas Busa: Right, so… but this doesn't require a hard fork, right? This can be just, rolled out.

Jochem Brouwer: Yes.

Barnabas Busa: You got pretty… okay.

Jochem Brouwer: It's a networking thing, but of course, all the clients could have to implement this. Yeah, but just wanted to mention this. Okay.

Barnabas Busa: What would happen if, one client implemented it, and we would have, like, one message exceed the 10MB limit? Like, you wouldn't be able to peer with that one client, right?

Jochem Brouwer: I am not sure about this, but maybe if Felix from Geth is here, or someone from Geth can maybe explain this, but the EPA says that this is a sync issue, so this is likely not… well, that's not very nice, so maybe someone else can fill in on this.

Łukasz Rozmej: So, the chain would progress, the problem would be that, the sync would not work for anyone. Except this one client could sync from itself, because it's a different protocol version.

Barnabas Busa: Okay, but the question is, do we want to actually, roll this out with the next hard fork in Amsterdam, or do we want to see…

Łukasz Rozmej: It can be hard for class, because it's a different protocol, DevP2P version, so we could do it without a hard fork, as soon as everyone is kind of ready.

Barnabas Busa: Okay, what do we want to target, for this release, then? Is this something we can start discussing now?

Łukasz Rozmej: So, for example, Nethermind have this scheduled to be worked on, but we are not currently working on it. I expect we will be working on it in maybe 2 weeks or something.

Barnabas Busa: Okay, so maybe some… something like end of January, could actually work for this?

Ben Adams: I mean, it's… because of where it'll happen at syncing, it's not a disaster, because it means you're… you're starting up a new node, so the fix would be to upgrade your client. Yeah, so…

Łukasz Rozmej: So, as soon as everyone has a release with it, we can… it's not a blocker.

Barnabas Busa: Okay, I will, start reaching out to different EAS and check on their implementation readiness, and then we can… Tried to schedule it in. Sometime. Probably end of June, or beginning of February. And, regarding any other blockers for 80 million, Assuming that we could go on to 70… 870, then could we just go straight to 80 million?

Ben Adams: I'd just like to point out, though it's a bit redundant, since we're doing the fork on Wednesday, and we're unlikely to increase before then, that we'd have to wait till after Fusaka to… To push it higher. But, I mean, you know, that's only a couple days.

Barnabas Busa: Push for what, exactly?

Ben Adams: To go above 80 million, so we want, like, the receipts… we want… we want the changes in Fusaka to go higher than where we are now, so that would be, like, the tap on transaction limit, etc. But As I say, that's only 2 days away, so…

Barnabas Busa: I just put a…

Ben Adams: Put it on the record.

Barnabas Busa: But we should just pipeline these things, and I don't think, like, Wednesday should be a blocker to, like, consider what else are we missing for the 18 million.

Ben Adams: Yeah, I think it… I think it's just the receipts. After a second.

Łukasz Rozmej: So, as far as I know, Carlos from EF is testing some… state scenarios? State-related scenarios, and some of them were… not great. Right now, but I think he didn't have the tests with applying the transaction cap from Fusaka, so… Yeah, so I think he wanted to redo it, so I don't know the state of it. Kamil?

Kamil Chodoła: Actually, actually do that. We need to rework all of our gas benchmarks to Fusaka right now. But this shouldn't be, like, super hard, so we'll do that this week, hopefully. And this and next week, to make it work. And yeah, this is exactly the point that some stateful tests will be, quite challenging, but it's more likely on the edge right now. And there are some new scenarios which were never analyzed by a rigon, as I believe, and Bezu, where the performance is not the greatest. But we need to sit and check if those scenarios are actually valid. So, there are quite a few points. I will try to summarize that and send a note on what needs to be done to bump it further. But for sure, we need to wait for Fusaka first to even start considering that, and EF70, as mentioned. But if 70 is required for anything above 80.

So I would even say that maybe we could go safer with 75 or something, just to leave some buffer for this issue. and then go straight to 100 after the solving that. So on protocol level. Actually, on networking level, there is only this receipt size issue. But there are some things which probably can be optimized in clients to make it smoother. the biggest issue from Netarmine's side, I see, is that those stateful scenarios are requiring some bigger changes on DB level, which we have. But yeah, we are still analyzing them.

Barnabas Busa: Okay, thank you. Yeah, the… I saw a bunch of suggestions, to just move to 75 instead of 80. I'm not sure if, that's something we want to do, even, without the, Spilling up the receipts?

Kamil Chodoła: Can you repeat, please?

Barnabas Busa: Is there a… Is there an appetite to actually just move to $75,000 instead of $80 million as the next step without the receipts? Or do we just want to, get unblocked on the receipts and just push to 80, or… Even further?

Kamil Chodoła: In my opinion, I would go 75 first, and just to leave a buffer between this issue, because maybe there is something which we didn't consider, so it will be slightly slower, but I guess this is the maxed out thing which we already have in our tests and in our knowledge. So 75 will leave us slight buffer, plus it will make us additional 15 MHz increase, and then 100 after… after solving that.

Barnabas Busa: Does it pen?

Ben Adams: I'd also add, since it takes quite a while to build momentum. But also we can't really determine the timing, then… We'd probably want to start the push before… E70 was rolled out, but also… Give us… give us that buffer in case it… it suddenly went up really fast. You know, because we, as I say, we can't control the timing when the gas limit goes up, so I'd go more for 75. So we can start. Encouraging people to move up earlier, but also have the buffer in case it happens before we Have the protocol change widely rolled out.

Barnabas Busa: I mean, 75 is something that could potentially be just bundled together with BPO3, because BPO3 would require a client release anyway. Right?

Ben Adams: Yeah.

Barnabas Busa: And we may… we may have ETH70 by then.

Ben Adams: So, we could go. We could do the AT or whatever.

Barnabas Busa: Yeah, I guess this is something we can start discussing now, and then hopefully roll it out around the end of January. Again, it will be up to the 870 implementation, and implementation readiness, whether we're gonna be targeting then 80 million or 75 million guests, whichever seems safe at that point. And Kamil. Hmm. Kamil, would you like to discuss this, testing block V1?

Kamil Chodoła: This is something which Marcin is working on. I can just give a short info on that. So, basically, we want to introduce a new namespace under our RTC, which will be fully utilized for testing capabilities. And here, this one specific example is to have a common interface between EL clients to trigger block building. from RPC perspective, so we'll be able to basically performance test block building under certain… conditions, and also it will unlock us some better possibilities for… to generate lists. So, if you have any questions to that, feel free to ask me or Martin. But yeah, this is something we are working on right now, and would be good to have some… someone from EL teams and from testing, DevOps team to take a look at.

Marcin Sobczak: Yes, and I'm working on, like, proof-of-concept implementation in Nethermind right now, so, we'll have it soon.

Barnabas Busa: Yeah, this looks great, from testing… side of you. Is any other EL Iran that has read this, PR and could comment on it? In case not, please take a look and, give us the feedback in maybe a week from now. Okay, is there any other discussion topics, regarding the higher gas limit? In case not, maybe we can peer towards the next blob flag discussion. So, there's an EIP open for the next blobs, which some…

Client teams liked, and some client teams didn't really like, because it's basically just a single flag that will limit local block builders to be able to limit how many, blobs they will include in a block. Nethermind has a implementation of this, Reth has an implementation in their, main branch, in their trunk branch. Is any other EL teams have an implementation, or opposed to having implementation for this? Poem, this is not a… a CL question, this is an EL-related question, but I can post the EIP in a second.

It's EIP7872. Anyway, I think we should try to get this included into every single EL before we hit the next BPO, because this could actually affect the local block builders if they have to build too many blobs with their limited bandwidth. They might result in pushing out an orphan block. So, this is, definitely a nice-to-have feature to… to include. I see that there's resource also tracking, That's good. And I haven't heard any, pushback from Geth either, so I think we should be good to go, and hopefully we can also start testing this feature, right after Fusaka.

Okay, and then there's one more discussion topic that's, a bit, deeper rabbit hole, so… engine API changes, I proposed an engine API change in the execution… Execution chat in R&D. I'm basically saying that we should switch to SSD for the engine API in order to decrease the times we spend on decoding the JSON payload. The topic brought up a bunch of, different, proposals, such as moving potential to, gRPC, or, moving to Protobus. But yeah, I would like to hear the opinions of different EL teams, especially, like, what they think about just migrating everything into SSD as an optional, thing.

Because, the more blobs we have, the more time it's gonna take to decode the, basics for encoded JSON. Not just in the discussion topic. I'm just curious if anyone has anything to say regarding this, on this call.

Alexey: Do you hear me?

Barnabas Busa: Yep.

Alexey: So… Just, wanted to mention, then, if we… introduce this encoding on your side. Right now, it will… will help us, to move quicker, to our end goal of having SSV. Everywhere, as a replacement for RLP. And this format is not just a data format, it's a thing that we need to… optimize for. It means that, we can, store data, and. Like, in database and, in memory, ready for, this, Encoding, and pass it quicker. We need to optimize serialization, make it parallel, maybe. And all this work can start when we have at least something to work on. And if we could have a couple of endpoints supporting this format.

It would, I would say start work on, active work on SSD, on execution on their side.

Barnabas Busa: Sorry, do you mean, like, CL that would support this?

Alexey: Sorry?

Barnabas Busa: Did you mean that, you need an example from SCL? To support the SSD?

Alexey: No, no, I mean, examples. I mean, like, applications of this format. Like, huh on Yale's side, to start experiment with optimizations and, proper, serialization, and maybe miracleization, even. We just need to, like… real application of SSD on your side, yeah.

Barnabas Busa: Wouldn't the engine API count as an application in your eyes?

Alexey: Yep, yep, exactly.

Barnabas Busa: Yeah, so, yeah, I think this could be a very good first step into assessing everything, the whole stack. And it's not a huge change. Like, you wouldn't need to touch too many databases or anything. In my eyes, this is basically as easy as it gets, to introduce SSD into the EL.

Łukasz Rozmej: So I think this is a matter of prioritization. And… While it isn't consensus in any way, so we could also do it outside of the fork. this should be considered… if we want to do it, it should be considered when scoping Amsterdam, because it is actually… especially for us, when we don't have a good SSD library out of the box, It's quite heavy on the effort.

Barnabas Busa: Yeah, so, I don't know if this would require an EIP in your eyes.

Łukasz Rozmej: It requires at least, execution spec, changes to execution spec, which are not 3D EIPs. And, and, considering, like I said, considering in… at least for the scope planning of Glamsterdam. So we just planned for it. In terms of, like, the size of the scope.

Barnabas Busa: So the journal blocker for future blob scaling could actually be limited by this, decoding. Based on the numbers that we've seen before, trouble said that, he spent about 300 milliseconds on 6 to 9 targeted max values, which is a significant amount of time. I know that, some, teams said that it was a lot quicker for them. And I'm still waiting for extra benchmarking, but, potentially we probably want to roll this out before Glamsterdam again. at least the latest by Glamsterdam, if we are actually aiming for, like, 72 blobs.

Łukasz Rozmej: Do we have numbers for SSD versus JSON?

Barnabas Busa: In any client? Well, we don't have any SSD implementation in any of the years, so I would say no.

Alexey: Yeah, let's try to compare, considering we have GitBluff's implementation.

Barnabas Busa: Okay, so I guess we should actually do proper benchmarking with the different max values. And then, if it's something that we actually have to prioritize in Amsterdam or not.

Łukasz Rozmej: Yeah, and it still should be, like, brought up in Glamsterdam planning, just because if we decide to do it, we might want to, you know, do less in Glamsterdam of other things, just to have a bit more headroom. And regardless if we even roll it out before Glamsterdam, right?

Barnabas Busa: Yep. Okay, I will, try to push for some benchmarks on this then. Mountain. Go forward. Okay, is there any other topics? I think we still have a bunch of time for open discussions, if anyone has anything else to raise. Yeah, Mario raised a good point that we can probably have benchmarks for all the different formats and figure out which one makes the most sense.

It was the engine that blops. Fun.

Simon (@siladu) | Besu: Yeah, I'm just wondering, what, what we should be aiming for, really, because I think get… what is in get blobs also applies to, get payload as well. I didn't write, get payload there. But, And that's gonna include the block as well as the blobs. Which will scale with the gas limit as well, so probably… engine get payload is the, the more important one, I think.

Barnabas Busa: Yeah, I think if we do end up going down the route of changing away from JSM to let it be SSD, or let it be literally anything else, we should actually consider, too, switching everything in the engine API, not just A specific function. Like, get maps, or get payload.

Simon (@siladu) | Besu: Sure, I would agree with that if we go with SSE. But I'm wondering what, like, what is the impact on the CL? At what point does it become a problem? Because that'll determine the priority, I guess.

Barnabas Busa: Exactly, yeah. Yeah, we're gonna need to have better benchmarks for that, I guess. And have everything written down, black and white, and then we can figure out, How big of a problem this is. Justin, I think we can, we can merge the ball and PPS breakout calls to ACDT from, like, next year onwards. At the 1st of January onwards.

Justin Traglia: Okay, that works for me. Sometime, beginning of next year.

Barnabas Busa: Yeah. I'm not sure if anyone else has a comment on that, but… Yeah, next week, basically. But, like, it will be a bit too short notice, and I think we're gonna have a bunch of cancellations also in the end of the year, with Christmas and New Year's coming up, so I think it would be cleanest to just, have the current schedule going on for the rest of December, and then, yeah, decide which calls to cancel, and then… From next year onwards, we're just gonna start with, ACP. those topics. And I think the first ACDT will be on the 5th of January. And regarding the lost ACDT, Well, I'm not quite sure yet. But we're gonna have it for the next few weeks, for sure.

You can get back to that, recording from now. Okay, thanks for everyone's time. I think we can call it a day for now. And see you all on Wednesday at the Fork. Good luck, everyone. Bye-bye.

Justin Traglia: Hi, everyone.

Łukasz Rozmej: Bye.

Pooja Ranjan: Thank you.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.