All Core Devs - Testing #60

Transcript

Barnabas Busa: Hello, everyone. Let's give it a few more minutes, and then we can get started.

I think we can get started. Akash, can you start the stream?

Akash | ECH: We are liar.

Barnabas Busa: Thank you. Welcome, everybody, to ACDT number 60. We seem to have a very light schedule for today, but let's jump straight into it. So, Fusaka-related topics, last week we had, supporting BPO2, and, Hoodi Fusaka activation, fork activation. Most things went smoothly, except, Prism, found a bulb. I will get, Manu to talk about that in a little bit. And, for the rest of this week, we are expecting BP01 to go live on Hoodi, and hopefully that will be a smooth transition as well. Manu, would you like to take the floor?

Emmanuel Nalepa: So let me pass the… So, HackMD. So, yeah, briefly, I just read the TRDR. So, we notized two things, with Prism. The first one is starting at Houdi, Fusaka and Houdi. We noticed a few attester nodes on the network, which are struggling. By struggling, I'm… Talking about… Producing attestation where target.epoch minus source.epoch is higher strictly than 2. There's not maybe Prism, or maybe not, I don't know yet. And, yes, and so when, this is, this is the first issue, and when, we have some node storing like that. in some cases, PRISM, when receiving attestation from those notes, may, in some cases, reject attestation, and then don't score to the peer…

who broadcast it, who, yeah, who gossiped the attestation and eventually disconnects the PR. And at the end, yes, we see that we mainly lose a lot of lighter spills, and… We gain a lot of, present peers. Yeah, so this is the TLDR, if you want to read in more details, there is a HackMD which is linked in the chat. And we are working on a fix. Hopefully today, or… Early this week.

Barnabas Busa: Okay, great. So we were hoping to get all the claims released out by end of today.

Emmanuel Nalepa: So, I assume that might not be possible.

Barnabas Busa: represent anymore?

Emmanuel Nalepa: No, no, no, no. Not for today, for sure. For sure, even if we, if we fix the… click the bug today, I mean, we will need some days, a few days for soaking before releasing. So, definitely not today.

Barnabas Busa: Right, okay. What about any other client teams? Anyone found any Fusaka-related books, in the recent days, or during Hudi? Okay, and regarding, any other… yep. Enrico? Enrico, you raise your hand.

Enrico Del Fante: Sorry, I was in mute. I joined late, so I don't know if it's been mentioned already, but by the end of last week, we discovered some… Bad blob sidecars. KCG proofs flow… with the wrong KCG proofs floating around in the network, and we did an analysis that has been… Like that in many… many, destinants before. So, we noticed that, because we had a bug log that was printing, that some of the website card we received would have been rejected, and then we did… we dig into that, and we found out that the proofs were wrong. So, according to my Yeah, according to my, to my latest update, it was a load star primarily sending this over the network, and apparently they're working on a fix.

And, we started the minute release, and we should do… we should go out with it in the upcoming hours.

Barnabas Busa: So was it just a partial number of blobs? Or all blood… Transactions? We're incorrect.

Enrico Del Fante: The blob sidecars were wrong, so it was… they were including wrong KCG proofs, essentially. Not passing KCG validation over the gossip rule validation.

Barnabas Busa: Did we not catch this earlier? When the block became invalid.

Enrico Del Fante: That's a good question. We've probably changed something on the logging side that become… make this more, evident. the frequency of on Houdi happening was… probably higher, but we discovered… we checked that it was also happening on Sepolia, probably on Olski too, so yeah. Don't know why.

Barnabas Busa: What about on Fusaka Devna 3?

Enrico Del Fante: I haven't checked there. I haven't checked there.

Barnabas Busa: Because that would be a platform where we could do end-to-end tests, basically, and validate that if our center picks for it, then… We would be able to validate it.

Enrico Del Fante: Yeah, I don't know how to actually catch this, maybe having, A metric that tracks how many rejection happens on every subnet. Which can tell you, even in testnets. DevNets, that's there. There is something wrong sent over the network, but otherwise, everything is normally recorded as a debug log, so it's kind of hidden, so… If… The good thing has been distributed on the network anyway, you don't see that there was something wrong sent over anyway. Yeah, maybe, maybe some, some kind of metric. could help. In the future.

Barnabas Busa: Yeah, Manu?

Emmanuel Nalepa: Yeah, I wanted to say that I also, saw, a lot of, Lodstar, peers, passing, 0.6000Z cell proof. And, yes, I said this already on the Discord, but… for example, for Prism, if, PRISM does get blobs V2 from the EL, and if, for some reason, the EL gives bad proofs, then in this case, PRISM won't verify the proof, because we trust the execution layer. And then, this proof will be, will be, you know, in, in, in, data column sidecars, and then Prism will send data column sidecars with bad proof, and will be, you know. And that messages will be rejected by other peers, and at the end, peers will reject… will, don't score and eventually, disconnect,

For example, Pierre, who, who, who did the… gets blocked V2 because of, potentially, bad… from the, from the ear.

Enrico Del Fante: Yeah, and also Teku, and probably most of other kinds will do that. So… If the source of this problem was the EL, maybe a test on the engine API, on every CL… EL client could have triggered this, detected this.

Barnabas Busa: Yeah, Nicole?

Nico Flaig: Yeah, so I think Matt is not here, so I can give a quick update what we found so far. So there is not really an evidence that it's from the EL yet. We had a bug, that we are fixing, where basically we would, call get blob32 or reconstruct cells, and then we import the block, and then… What we do internally, we clean up the block input, which basically tracks the columns and the block, and then when we would receive another column for the block over Gossip, we would restart the whole process, because we internally, the code was already pruned. So we would call get blob32 again for that data, even though the block is already…

Fully there and, imported. And we think the zero boost might be some internal buffer that gets overwritten due to that, while we are still… Flushing the data to gossip. And maybe regarding the, why we didn't see this on the DevNet, so there was a pretty major refactoring we did about a month ago, so this code wasn't really running for that long on the DevNet also.

Barnabas Busa: Yeah, mono?

Emmanuel Nalepa: Sorry, I just forget to lower my hand.

Barnabas Busa: Have you noticed any of these zero proofs, floating around that were not published by Lodestar?

Emmanuel Nalepa: Yeah, mainly by those stories. But not only. But mainly.

Barnabas Busa: What does that mean?

Emmanuel Nalepa: I mean… Which other…

Barnabas Busa: Which other client have you detected?

Emmanuel Nalepa: Yeah, I saw at least one Sprism, at least… exactly one Sprism, I saw exactly one Sprism. And that's why, because of that, I assume… assume, a bad EL proof coming from the EL may be the cause. But it's, yeah, it's very majority of time, it's lost our, and… And sometimes it's not Lustar, sometimes it is other clients, including Prism. But… Yes, let's say 0. Between 0 and 1% of the time, it's not close to.

Barnabas Busa: Okay, do we feel like this is a critical problem going forward?

Emmanuel Nalepa: I don't think so, because in this case, just the… The, the, the sidecar is just rejected. And that's all. And so the 40 peer may be, Don't scroll… is don't scroll and may be disconnected, but the sidecars, the 40 sidecars, are not gossiped over the network by the first player.

Barnabas Busa: Right. Do we have any other Fusaka-related topics? In case not, then maybe we can move on to the AI gospel testing. Anyone from NetherMind or the testing team, Ari, could you give us an update how everything is going?

Kamil Chodoła: I can give an anomaly to that. So, for now, we are focusing on stabilizing the entire environment, that's why last week, we decided to switch to 100 megas blocks only, and this is better right now, results are much more stable, and The feedback loop is faster, that's a good thing. Other than that, working on adding newest version of EST compute scenarios, but having some issues with generated Genesis files, working with Louis on that. And in the meantime, we are working together with Marius… on some repricing branches and repricing machine, where we're only testing the… and very… grasping the inputs for potential repricings for Glamster now. And from stateful scenarios point of view, here I need to refresh the machine, which is testing SAN and other stateful tests on top of LoadNet, because there was some issue out there, and it is down for now, but I need to revive it.

And in the meantime, everything, is progressing, I would say, in proper direction. We need to improve our metrics collection to have a better input for repricings moving forward.

Parithosh Jayanthi: Also, to note that clients starting this week will be releasing with 60 million as the default, so as and when the network starts updating, we should see the gas limit go up. Or at least starts being booted up.

Kamil Chodoła: I see already some blocks on 35+, again, so…

Ben Adams: We may… we may have an issue, because… I think the gas limit was voted up last time, rather than using the defaults, so… It may be more resistant to going up if everybody's manually set their value. We might need to reach out. For the community.

Kamil Chodoła: And what will happen, probably, will be that we will… I'll update at once on…

Parithosh Jayanthi: Outlook, right? Also, the… yeah, the good thing is there's a hard focus, so… worst case, we'll hit the gas limit up a bit then.

Ben Adams: Yeah, but if they've used flags to set it to 45, then… Little stat, 45.

Barnabas Busa: I think most users will be using, the default flags.

Parithosh Jayanthi: Yeah, but it is worth reaching out. We can do a round of that once the releases are out.

Barnabas Busa: Yeah. Okay, so is there anything else for estimate testing? But then, let's, get into grump serum testing. Stefan, would you have an update for us?

Stefan Starflinger: So the definite, is pretty ready. Nethermind, and Russ have signaled that they're ready with the last changes from the call on about. But Geth, and PESO, I think they haven't signaled yet, and I think we might have also a testing update. Do you have a… Testing update regarding the latest tests.

raxhvl: Yeah, I've posted it in the chat.

Stefan Starflinger: Okay, that sounds pretty promising. I think as soon as that's ready, we can start with the DevNet.

Jared Wasinger: Sorry, but what do you mean by signaling in this context? I mean… Like, when you say the clients have signaled that they're ready or not.

Stefan Starflinger: Oh, I… in the blog access list chat, I tagged all the client teams and asked to give, like, a thumbs up when they're ready.

Jared Wasinger: Oh, okay, gotcha, yeah. I'll do that, I think Geth should… Geth should be ready, yeah.

Stefan Starflinger: Yeah, so that would be nice if, when you're ready, also when Biso's ready, and then I will, start the, DevNet. I think that's it from my side.

Barnabas Busa: Yeah, Mark?

Marc Harvey-Hill: Oh yeah, I just had a quick question about these, test results. So it's from… are those, test fixtures up to date? So I know we agreed to make some, kind of, change just before the DevNet, And I'm not sure if that's included or not in those tests.

Felipe Selmo: Rahul, I saw you left and came back. Did you catch that question?

raxhvl: No, no, I just lost it. Could you repeat that?

Marc Harvey-Hill: Yeah, I was just asking if those, tests, 1.3.0, is that up to date? Because, yeah, the Coinbase change, is actually.

raxhvl: No, I think… Yeah, there is a new release coming up.

Marc Harvey-Hill: Okay.

raxhvl: That'll be likely today or tomorrow. That includes close to, I think, 25 tests. Yeah, so that should cover everything.

Marc Harvey-Hill: Okay, great. Thank you.

Felipe Selmo: Yeah, and the updates to the Coinbase are also there too, so it does update some of the older tests, according to the breakout. Session conversations.

Dragan Rakita: Are those 100% passing on that new test suit, or on old one?

raxhvl: The results are from the previous test. This does not include the Coinbase test, so we'll… we'll update the results today, once we have a new release.

Barnabas Busa: Okay, great. Is there anything else regarding… Doctor back to this testimon?

Stefan Starflinger: I think that sums it up.

Barnabas Busa: Okay, anyone would have any updates regarding EPBS?

Mario Vega: Hey, so, I don't have that many updates, but, Justin did ping me before the meeting that he won't be able to join. Basically, the only updates is that there are two new releases on the consistent specs. One is 1.6.0, and the other is 1.6.0 beta number 2. So, that's… that's basically it. Just… just please give it a look. I, I see that there's a comprehensive, readme, at least in the 1.6.0 beta 2. So yes, I think that should give you more context. But yeah, more than that, I'm not… I'm not sure.

Barnabas Busa: Yeah, I also haven't heard any update from buying teams, signaling that they would be ready for DevNet Zero, so… I don't think we can do, APB as a net0 before NetConnect, but maybe… all DevNet Zero could actually happen sometime this week, and then we would have something, running throughout DevConnect, and then we should target for, APBS DevNet late November, early December. Any other, questions or updates regarding Glamster then? Maybe then we can talk about, the mainnet client releases. We heard from Prism already that they would not be ready, for today.

But, I would like to hear from every single client team if they could make it.

Kamil Chodoła: So, for NaturMind, we are working on release right now, but we are facing some unexpected regression around sync, So… Worst case scenario will release today, minimize the version. And we'll have hotfix probably in the coming days.

Andrew Ashikhmin: For Aragon, we've released 3.2.2 today, which has Fusaka on mainnet.

Barnabas Busa: Can you guys also confirm everybody, set the gas limit to 60 million by default?

Andrew Ashikhmin: Yup. It was said in the previous patch release. So, yeah, it's sad.

Barnabas Busa: Okay, great. And, any CL teams? If you're reading the chat, I see that, the expiration will not release, and… Taku is in progress, I'm gonna start his working RNAs as well. Oh, I think it looks like, we're not gonna have a complete list of releases by the end of today. We should really try to do a deadline by…

I would say Wednesday this week. And if we cannot meet that deadline, then, we might actually need to consider rescheduling main network. Just in… in case to have that, for today in between. releases, but… Yeah, I really don't want to have that. Yeah, is the 5th of, 5th of November as a final deadline would make sense, and then I think we could have the blog post out by Thursday, maybe by ECD. I haven't talked with Alex, when we wanted to check out the blog post, but I think, this week would be very, very good.

Enrico Del Fante: Point is that we sh… we don't want to release if, for… by any chance, there is some… there is still a… As a condition in which we could postpone. So, it could be tricky, then, to… Whatever colder it is. We could, but there might be a noise.

Barnabas Busa: Yeah, I think if we can have all the clients raised by the 5th, we really shouldn't have a problem, still forcing, maybe not on the 3rd of December. I'm just a bit worried that if we do slide it, for, like, next week, there are gonna be further delays, yeah, due to DevConnect. And that could actually lead to potentially, causing some issues down the line. Because we did want to leave at least one month between now. Reason, Minnet. I think the main, the main question here is, If PRISM would be, if PRISM thinks that they would be ready by Wednesday.

Emmanuel Nalepa: Yeah, so, in the best case, we… In the best case, yes, we… we find the… we fix the bug today. Which is a third. And we need a few days for, for soaking. So yes, fifths will be the really best, best, best case, in my opinion, for a release date. It's more, reasonable to expect a release this week, but maybe not on Wednesday. Yep. Yeah, yeah, yeah, so, I mean, yes, so in the best case, release candidate on 5th. If we fix a bug today, we can have a release candidate on 5th. Yes, but maybe not a release on this day. Yep.

Barnabas Busa: Yeah, maybe Lighthouse can also comment? Not sure if anyone from that house is here. I'm quite sure you find anyone. And the call from Nathan. Yep, met you.

Matthew Keil: So, we identified a bug as well, that didn't show up until we got to the larger network. We've figured out what the issue is. And we're in the process of also soaking a fix. And talking about it actively. So we should be able to get the release out, but that is also kind of a… by the 5th is our goal, but just wanted to put it out in the world that it's a best-case scenario.

Barnabas Busa: Thank you. Yeah, I think we should, stick to the original timeline of the 3rd of December, let's push out the releases, and we need to re-evaluate by Occo Devs on Thursday if we really don't have all the all the major releases by, all core devs, then, we might need to discuss an alternative deadline. Or, main net. But I think we should stick to the, urgent schedule for now, because, yeah, the 30-day is, It's almost there, with, having already settled by the 5th. But if, by Thursday, all coordinates, we cannot, we cannot have the releases, then… then I think an alternative proposal will have to come up. Does that sound reasonable? For, everyone?

Okay, great. And I just got a confirmation from Alex that we're gonna be targeting the blog post also by Thursday. So, yeah, just please keep working on your releases. Okay, there's one more topic that Mario wanted to bring up, the testing complexity assessment checklist. Could you please, bring it up, Mario? If you're here.

Mario Vega: Yeah, of course. Yeah, it's very, it's very quick. So, we are gathering information on, the… proposed for inclusion EIPs for Amsterdam, and we've seen that there's a ton of them, and so, we have a tracker for this, I just showed you the link. We're gonna use a checklist, basically, to follow the, to try to determine how the testing complexity for a given EIP is, like, either high or low. This is only for the execution layer at the moment. But, in any case, if you want to jump in and chime in into this issue, please do so in regards of any EIPs that you think we should prioritize. Because you see more impact of this EVM EIP, just please go into the issue and comment on there, and we'll try to prioritize,

to assess the complexity of the testing of this EVM EIP, so we can move it forward, and maybe even start with the specification implementation, and also the testing implementation. But yeah, that's basically it.

Barnabas Busa: This is great, Mario, thank you. Would you be open to also include the CL EIPs in here? I know that they would be targeting different kind of tests, but… We only have a, very few, I think maybe 3 CL-specific EIPs in Amsterdam.

Mario Vega: Yeah, absolutely. I mean, if you have comments on the… on the CL EIPs that you want, that you want, that you think we should prioritize, just… you just feel free to leave them here. We'll try to redirect this to the proper people to… to… for them to… to start implementing or, assessing the complexity of an EIP, yeah.

Barnabas Busa: Yep. So, would you expect, that each team goes through all the IPs, and assigns someone to it, or…

Mario Vega: So, yeah, that's it, that's the question, right? So there's too many EIPs, in my opinion. So what I'm asking here is that if we can, like. get a sentiment from the rest of the core devs for Glamstagram EIPs, just to see which ones we should prioritize on… on assessing. based on their feeling on which one will have the better impact in the EVM, I think, I think that's… that's what we should, like, lead with, with those EIPs, yeah.

Lightclient (Matt Garnett): I feel like the problem is that we have 10 EIPs that are just focused on pricing-related concerns.

Ansgar Dietrichs: Matt, can you clarify how is this a problem in this regard specifically?

Lightclient (Matt Garnett): I mean, we want someone to, like, deeply look at all these EIPs, and we also want to try and find the things that are impactful, but if we're just going… you know, if we're going to do the pricing, which is seemingly, like, highest priority, I don't even know if we have the bandwidth to also be doing that many impactful things to the EVM on top of that, that are not related to pricing.

Ansgar Dietrichs: Right, but that seems more like a general folk scoping kind of Argument and discussion, whereas for, like, specifically the testing impact assessment. I mean, yeah, I would focus on basically figuring out what are the first batch of, like, most important, highest impact ones that we should try to get a feeling for first. And that probably includes some repricing, maybe includes some non-repricing ones.

Lightclient (Matt Garnett): It just feels like a lot to me. I feel like we should have… some clear prioritization of the EIPs before we just… Analyze all of them. But if people feel like we have the bandwidth to do it, then we can do it.

Mario Vega: You know, I completely agree with you, Lagland. Yeah, that's what I… I feel like we're overwhelmed at the moment with the amount of EIPs that we have to analyze.

Łukasz Rozmej: So, what I wanted to get from.

Mario Vega: the rest of the core labs, is there's… if there's, like, a clear sign of which ones we should prioritize. Yeah, I completely agree, this is too much.

Łukasz Rozmej: Nevermind team went through, I think, all the APs, maybe there were some newly… We went through all this, we find, and we can share some of our views. We… I think it's not yet, like, published so far, but we can at least, if we want, give you, like.

Barnabas Busa: It's been pretty hard to hear you in the past.

Łukasz Rozmej: What? Okay.

Barnabas Busa: Nope. Maybe you can give a… just leave some, text in the chat? Because your audio is cutting out.

Łukasz Rozmej: I'm driving.

Barnabas Busa: Okay.

Marc Harvey-Hill: Yeah, he was just saying the team's gone through all of the different EIPs, and, we're in the process of writing something up that we can share soon.

Barnabas Busa: Yeah, I think it would be good if some bigger teams would be able to, like, give the other teams a heads up of, like, which are the EIPs that they would consider kicking out right away, decorate as fast as possible, so that other teams wouldn't need to basically chew through those EIPs, and I think that would be very, very good. Could we have the Netherland list open, or, like, publicly available for everyone, and then maybe we can already start kicking AIPs out on this week's, Occo Devs call.

Marc Harvey-Hill: Yeah, we can post it somewhere soon.

Barnabas Busa: That would be good. Thank you. Anyone else have any other topic that they would like to discuss?

Lightclient (Matt Garnett): I think just last thing on this… For the testing team? I think the highest priority is the repricing EAPs. So, if you're still waiting for more signal, I think that's… The clearest signal to me is that we need to look at these and figure out What makes sense, where the complexities are gonna be, what the interplay between them is going to be, and then when clients start releasing their preferences, we can start looking at the non… Free pricing related stuff.

Barnabas Busa: Yep. Charger?

Fredrik Svantes: Yeah, this is unrelated to… to that, but, Yeah, as we're moving towards the mainnetwork, we are… we also have the mainnet upgrade and incident response team plan, which Alex has been working on. You can find it here. Basically, it will be the same as we did for the idea of this is that we do it the same way as we do with Pectra, where each of the client teams sound Shared roles. Teams have, basically coordinators assigned to, As points of contacts, so, It's probably a good idea for client teams to at least start thinking about Who should be the primary coordinator? Who should be the backup? Coordinator, etc, and yeah, it's something that we're gonna have to be… Doing in the coming weeks.

Barnabas Busa: Yeah, could you guys please make a PR to this, more than fine if you feel like you could be your client team's primary or backup coordinator, and you can start merging in those PRs. Or, yeah, possibly just reach out to Alex. I'm not sure if there's an open PR about this already.

Fredrik Svantes: I don't think there is… he's… I don't think he's opened a PR yet.

Barnabas Busa: Yeah, because I just saw that this was at work.

Fredrik Svantes: Yep.

Barnabas Busa: Okay, so I think we should merge this, into the upstream repo as soon as possible, and then, client teams can make a PR to that. Thanks for bringing it up. Okay, is there anything else? Okay, Slat, good luck with all the releases, and well, hopefully everyone We'll be ready by, Wednesday or Thursday this week. And see you all on Thursday.

Emmanuel Nalepa: Video, thank you.

Stefan Starflinger: Thank you.

Andrew Ashikhmin: My buddy.

Mario Vega: Thank you, goodbye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.