Interop Testing #35

Transcript

Hello, welcome to the interrupt testing call, the last one before Spectra. And I think from next week we'd be using the um official ACDT moniker. Um let's start off with um with the Petra updates. So from the East Panda Ops team, we've uh so all the client teams have made their releases and the blog posts have been updated to indicate that there were two uh bug fix releases or three bug fix releases over the last week. Um we've already run the bug fix releases through assertter and all of our boot nodes, test net nodes, validators, everything has been updated with the latest client releases. As of now, um, nothing was

immediately noticed. Uh, we're still going to continue looking at the logs, making sure that everything is fine. Um, there was one thing that, uh, Barnabas pointed out with the Flash Bots. Um, MEV boost. No. Uh, there's a timeout that's being returned over there. Uh, this is on mainet, but it's likely just Flash Bots needing to scale up their infrastructure. Uh we've already pinged them about this and haven't heard back. Um but it doesn't seem to be a bug in Me Boost or anywhere else and it also works fine with uh with the other release. Uh do we have an update with Petra testing from the EEL team or East?

So, we've been checking the hive uh there. There's a couple of hive uh tests uh not passing, but these all seem like nice to have passing because they are based on scenarios that are not technically possible. Um so, we're going to prepare we have a document that Spencer prepared and we're going to share that with the teams just to uh double check with them uh that they are um they are okay with this with these errors and then we can go ahead. But they seem uh they seem to be just uh either test issues or uh scenarios that are not possible to happen. Um so yeah, we're going to reach out to each uh each client team to to to

discuss for discussion. Yep. Apart from that, there's not that much else to to discuss from our side. Awesome. And you also did a manual review, right, Mario? You just want to briefly mention that too. The manual review. Uh which part? Oh yes yes yes sorry. Um so yeah it's basically we just um we went over I can share the the the link but I don't have it at hand. I am going to share it a little bit later. Um so basically I did a manual review of all the client teams. I went over the repositories and manually checked um the contract addresses in each one. Uh this is a very superficial check because even if you do this uh there's still possibility that the values that I checked are not used. But

it's just a way to for us to send it to check that everyone hardcoded the correct addresses, hardcoded the correct time stamp for the fork and epoch. So I'm going to share the link. I'll I'll I'll I'll do it in a bit. Yep. But yeah, everything seemed fine from from the from this review. Awesome. Thank you. And I think anything that we had uh the pandops team had noticed over the last week also. Yeah. Uh I think anything we had uh opened up as a discussion point with client team has also been completely resolved. So we should be good. Um we did have a mainet shadow fork uh last week as well. And the mainet shadow fork um we investigated the prism deposit uh

tree topic. Uh there was some sync issues on Aragon that were completely fixed slash um fixed by adding some flags on our side and then there was a get uh override Prague um minor bug that wasn't the flag wasn't being respected and that's also been fixed. Um however we weren't able to conclude the deposit testing purely because of a weirdness with um how the tree was built and what was valid on the shadow fork and what wasn't. Uh but we did test some other Petra related features and that went well on the shadow fork as well. Um the clan teams all had a look and then we decided that there was nothing else to

really gain from the shadow fork. So we tore it down around Wednesday last week. Um yeah and besides that um anything else from Pekra that people want to bring up? anything we should still look out for this week or test before mainet. Okay. Um and I do want to mention that the Nosis chain guys uh went through Petra last week and it seems to be successful. Uh we're in touch with their testing team. They ran through a couple

of their own tests and they're also happy with the outcome. They didn't notice anything weird, which is great news. Uh they did have a minor blip in participation rate, but I think that was just people who hadn't updated and the participation rate is basically back to what it was before Petra and that is that tends to happen with folks very often. Um okay. Uh anything else on Petra testing? I think just a manual manual mention of the latest east release would be nice.

So, uh let me share the link. Um basically this this is the first release that we make that contains all the Ethereum tests um Ethereum tests uh uh original tests into into into an release. And basically the plan is for us to migrate everything from Ethereum test into into execution spec tests. And ideally all the all the all of the new releases from now on will contain the what was originally found in Ethereum tests. Um, and yeah, just just a heads up that probably this is going to affect the CI from all the client teams because if you point right now to um to to Ethereum tests, you're going to

have to update to point to the east releases and from and from there it will be the single point of consumption for all the tests that that you ever need. Uh it's still a work in progress and uh but just a heads up for now. We're going to repeat this once we complete this process of migration in the future. But yeah, no nothing no not no uh action uh points at at the moment. Awesome. Thank you. Um and yeah, Camille gave us an update on the nosis drop in participation.

Uh, thank you for that. Yeah. Um, anything else that we should be testing in the last days? Thank you. Yeah, there seems to be some update to 7702, but this is Yeah. Okay. This is a wording update. Uh, perfect. Um, Mario, could you guys maybe add a east testo for this for this case as well, please?

We It's same. It seems from the reading from the first reading, it seems like we should already have something. Um, but yeah, we we're going to check. Okay. Um, then moving on to the next topic, uh, communication channels for Petra. So I'm not sure if everyone has already seen this but the PM repo has a Petra mainet plan document uh that outlines what is to happen in case there were there are any incidents and so on and where we'll be communicating. Uh so the primary communication channel will be a pectra upgrade channel that'll be created uh some point roughly tomorrow.

So definitely before the fork and uh the idea is that this channel gets archived after the fork is done. Um the status updates will be on the ETH R&D discord announcements channel. Uh if you notice spectra PCRA upgrade related discussions happening somewhere else or Telegram groups, please try to move the discussions back into the ER&D discord so that there's one canonical place where we can keep up to date with everything that's going on. Um same for discussions themselves. try to keep them in threads so that it's um easier to follow what's going on in the in the channel itself. Uh there is a primary and backup person for every uh client team who would be involved in

the fork and we have a bunch of um east tests that we plan on running on mainet as well as um few topics that we will be looking out for before we declare this fork is successful or not. Um, we do have a checklist on the ETH Panda Ops blog. Uh, so in case someone's just intrigued and wants to poke around yourselves, then uh there's a description on what what would be interesting to look at? Um, yeah. And is there anything not mentioned in this list or something particular that client teams want us to check first or ASAP? uh if not I think we'd be basically looking at blocks at

stations arrival times finality um and then running through some tests of all the Petra features and then that would be mostly it. Then we go into monitoring mode if there's some an anomalies. Okay. Um, in case your name is primary, your name is listed under primary or backup, please try to be available and monitoring the ETH R&D discord channel at the time of the upgrade.

Uh, I think there are quite a few watch parties and live streaming on as well. Um, so it would be nice if you if you are primary or backup if you at least have the Discord channel um, on the side panel so you know what's going on in case someone needs you. Okay. Anything else for communication on Petra? Yeah, Lucas. for the monitoring people. I think we should have a scheduled uh like meeting uh that we could send invite to and that people who are free could could be

like observing there and people uh and especially if there's something happening uh we can already uh have a place to join instead of like oh now looking into the discord where's the link etc etc. H yeah, I think we can uh set one up and share it beforehand. Yeah, that's a good idea. Thank you. Okay, anything else on Peterra itself?

Great. Um, let's move on to Pas then. Um, Barnabas, do you want to give us an update on Pas? Yeah. Last week, uh, we discussed that we're going to be launching the seven sometime this week once we have enough uh, client teams ready to go. So the specific change that is required is the the step by route and none of the ELS would need to update for that only the CS. So far we have two uh theoretically working CLS the Nimbus and Grand.

Yes. Can I talk to you a little bit? I'm sorry. Yeah. And uh so we're waiting for other field teams to finish their implementation and then we can possibly roll out uh pre7 maybe even tomorrow or maybe on Wednesday. Sounds good. Um and this means we would no longer be using the DevNet 6 spec at all. Or is the idea that the DevNet 6 one will continue for sync testing and DevNet 7 is for everything else?

Well, DENTE 6 is shut down. So, yeah, but we just spec on the Sunnyside Labs network, right? Right. Uh so I think the inside lab network would just slowly die off eventually because uh we we want to migrate everything to dev. There's also going to be a new spec release made uh 1.5.0 zero right after mainet fork. We should possibly target that for uh yep, sounds good. Um then it would be

great if clan teams can have a look at um getting their images up as uh are there any open PRs worth discussing right now or would that happen tomorrow at the pas testing call? I don't think so. Uh other than possibly discussing BPO implementation and ask client teams how they're doing with that and whether they have looked into it. Okay, perfect. Um, and I guess uh we can discuss it on the podas testing call as well, but um once Spectra is on mainet, we would predominantly be using this call to be discussing Pas. So it might

be unnecessary to have another pure testing call tomorrow. Uh but I guess we can discuss that also on the call tomorrow and decide to cancel it for the next week or like merge it in basically. So there will be no more perus calls. Yeah. Yeah. Because that's the main fault to be tested after Petra shipped. Okay. Um, anything else on pure?

Okay. Um, I think the only other active work stream right now is um gas limit testing. I know that earlier the blocker was um this OOM bug with 60 million gas but um Jared from gets posted that it's no longer a blocker so we can move ahead with um um increasing the gas limits on test nets. So, we just proposed that we would wait until Spectra shipped on mainet and everyone's feeling like they're on dry land again. And um I would suggest next week on the call we discuss which test net to bump the gas limit for. I would assume um Sapoleia

would be the first one and then once we roll that out, check how stuff goes, we can move on. Okay. Um, and yeah, is there anyone from the history expiry side of things to give an update? Like cuz I think we're past the May 1st deadline. Are there any ELS that have like a specific flag where we can like start playing around like I want to drop history till block XY Z or till fork whatever.

Yeah, I think last time we looked we were lacking a lot of documentation on what needs to happen. So does any client support history expiry right now? And if so, could you point us to some documentation so we can test it? We would like to set up some longunning syncing test where ideally we wouldn't want to sync history. So it would be good if it would have some flags. Okay, I guess we can ask in the history expiry channel what the status update

is, but it sounds like no one has shipped it yet. Okay. Um, yeah, come in and I mean uh in mind we have a support for dropping the history and preserving it via v via era files. So our validators are capable of dropping the history any time. Could you sorry could you point us to some documentation on this please? Yes sure I think it's only on a sync there. That correct Kevin?

Uh again please. It's only on a fresh sync I think isn't it? Yeah, the freshly synced nodes will already not have the premerge uh data on on Sapalia. Okay. Uh that's good to know. But I think last time when we looked at the release notes, we didn't find. But yeah, we can have a look again. Uh yeah, it was included. I I will point it out to you. Would it also be possible to drop arbitrary block heights?

Uh yeah, you can set it higher. So if if by any reason you want to drop less or more on initial sync, you can set it up directly by a flag with a block number. And this is only done during initial sync. On initial sync, yeah. So I think we have live pruning uh but in PR it's not finalized yet like dropping on the existing nodes is yeah as Bua stated in progress but if you want to drop your history you need to reync for now. Yeah, I would definitely look at updating some release notes on this because I just looked

through like searched for history on all of the releases and the last mention is I think 1.31 and it does mention history is like eran import export but there's no real like there's no context given and I think that was the feedback we got from node operators as well like everyone would like to drop history we communicated that there was a deadline of May 1st And then the May 1 deadline has passed and no one knows how to drop history or like how to recover or what is happening like the coms on the topic seems a bit lacking.

Okay. Yeah, you're right. We mentioned that it is enabled by default but we didn't state it in release notes how to do that uh direct. I will fix that. It is the latest Nethermind. Yeah. And I think you guys have a a Nethermind docs page as well, right? So you could have a page there about history expiry as a topic. Yeah, we can separate that to to make a clear and clear distinction for how to drop that. But yeah, we have the these parameters described in the CLI uh information, but it's not that easy to find out. So yeah.

Yeah, thank you. Um, and any other clients who have an update on history drop. Okay. Um, Roman says is still work in progress. Um, last I remember Gat was also still work in progress. Um I'm not sure about Aragon and Bzel though for Bess. We are also working on it. I'm just not sure about our current status

but uh I think we can import the error files and drop some history but I I need to check up with the with the other devs that are working on it. Yeah, that's in testing right now. Okay. Uh, thank you. Yeah, I think we'd also like to do a like an EF blog post at some point about history expiry, but um I think we would first wait for all the clients to have the releases in and documentation so that we can point people to and it would be very client specific

anyway because I think some clients like every client has a different way of how they expect to fetch history after it's dropped. So yeah. Um, cool. I guess the discussions are happening on the history expiry channel on ETH R&D. Okay. Um, anything else to discuss today? Any topic people want to bring up? Should we come up with a new deadline for history expiry?

The last one that was communicated was that we wanted to drop history on Seolia for 1st of May. Um, I don't know what the new plan is, but I guess we should discuss that on the history expiry channel until we need some flags. That that's the thing. We would be happy to help, but if we have no flags or no documentation from any clients, then it's very hard for us to test anything.

Yeah. Yeah. I don't know if other clients are in this boat, but I think with Petra coming, people are a little apprehensive to put many more changes the last few weeks on master. Otherwise, if there's any kind of issue, we would we don't want to be pushing a bunch of stuff that is not as rigorously tested to all the nodes. So, I'm guessing after Petra, pretty quickly people will start merging stuff again. Yeah, that makes sense. Um, I guess we should first discuss a plan on the history expiry channel. Um, I'm guessing most of the people involved in history expiry are not on this call. Um, but yeah, we should at least have a plan as

to when people are dropping history, when draw docs are out and so on. Because otherwise it's just incomprehensible for any node operator to know what's going on. Uh, yeah. Should we also make sure to invite Piper to these calls because he seemed to be the the champion for coordinating his story at least until what he until this gets shipped. Yeah. Uh that makes sense. I think then yeah once Spectra is done like like Han said people will have a bit more bandwidth. Um I guess we would shift focus to pure das limit testing and history expiry. Uh, so it would make sense to have Piper there as well.

As far as I know, Piper is the champ, you know, 44 force and not specifically for history expire. History expiring zone could be just removed and yeah, especially for premerg data. Yeah, still makes sense to have Piper here. And I think I mean more importantly the point of history expire is to deliver uh benefit to stakers and users and it feels like right now we're not delivering that. So we should figure out our

Um, anything else that we should discuss today? Okay, then see you guys on the other side of Petra. Bye. See you. Thank you. Bye. [Music]