All Core Devs - Testing #61

Transcript

Mario Vega: Hi everyone, I think we can give one more minute before starting. Give people time to join. Thanks.

I think we can… we can start. I don't know, people.

Pooja Ranjan: We are live.

Mario Vega: Thank you. Hi everyone, welcome to ACDT number 61. Today's November 10th. we have, the same, basically the same items of the agenda as every week, and I think we should just get started. Yeah, starting with Fusaka, we had BP01 activation last week. So, the first topic I wanted to discuss is whether there were any events that were significant enough that we need to discuss this week. Barnabas, do you know if we have, If it was a good transition?

Barnabas Busa: We had no problems on the BPO2, but we have discovered a bug in NetherMind. Maybe someone from the NetherMind team can reach out and discuss what has happened.

Parithosh Jayanthi: Do you mean a bug relating with BPO2, or the release bug?

Barnabas Busa: the release bug. Sorry, regarding BPO2, I think we were good. BP01, BPO2 is the trick.

Marcin Sobczak: Yeah, so…

Barnabas Busa: So…

Marcin Sobczak: Yeah, we had, we added some bug fixes on top of, the version 1.35.0. We released 0.1 with bug fixes, but these bug fixes introduced even worse bugs, so we then reverted to… we released 0.2 version, which is on top of 0.0. And we already, solved this… the bugs, which were in version 0.1, but it was, like, under-tested. We just released it, It was under-tested, and, we are, we are, going to, to do a release after mainnet fork with, with, some known optimization and fixes. And right now, we go with version, .2, which, we consider as, good enough, and,

And, more stable, and at this point, the stability is more important than having fixes of very rare bugs, etc. And, yeah, we introduced, we had a broken snap sync, because we introduced, regression in downloading block receipts. And, other, code, like a consensus bug, but, yeah, it's, We are going with version 2.2 now.

Mario Vega: Alright, thank you. Also, Barry mentions that there was also a red, bug. Is someone from Red in the call that would like to chime in? Alright, I think there's no one from Red. Do you guys know, Barry or Venomas, do you know the context on this?

Parithosh Jayanthi: Yeah, I think the TLDR is they merged some changes that were supposed to be meant for block-level access lists, but, the changes ended up affecting the mainnet release before the fork. But I don't have any more details on the topic.

Mario Vega: Alright, thank you. Yeah, I think, I think it's basically the same thing, right? From Nevermind. It would be nice if I… if we can reach out to Reth at some point to see if they're planning on making another release, just to make sure that the blog post is updated, I can… I can… I can follow up with them. Cool. Anything else from… the HUDI VPO activation, or the mainnet, client release.

Parithosh Jayanthi: Bill.

Mario Vega: Not too sure.

Parithosh Jayanthi: There's one more that Sam, found out, which was, Lighthouse, it's a non-consensus critical issue, but, Lighthouse is downloading the blobs from the checkpoint sync provider, rather than fetching from peer-to-peer. So, the team has mentioned that they'll fix it in a subsequent release, but it's non-consensus critical, so it's fine.

Mario Vega: Alright, alright, thanks. Enrico?

Enrico Del Fante: Just one thing that I noticed, 15… kind of 15 minutes ago, Our nodes are detecting some columns by route request via PC on mainnet at the moment. It's kind of curious to see. I don't know if anyone else noticed those, RPC calls at the moment. Yeah, nothing important, but, just, if anyone adds… It's just a curious thing, maybe some… there is some client, buggy client, doing this RPC request. Beforehand.

Mario Vega: Do you have more info on where these requests are coming from, or…

Enrico Del Fante: It's really… 10, 15 minutes ago, I haven't dig into that, at the moment, but I don't have the peer, peer ID at the moment. I was just seeing these requests coming, coming… Yeah, just a curiosity.

Mario Vega: Alright, definitely needs to be investigated. Anyone else has… Seen this? Okay, no, I think it's not recorded by anybody else, But I think we should follow up on this.

Enrico Del Fante: I will, write a message somewhere in Telegram or Discord, and maybe run… A thread on it.

Mario Vega: Alright, thank you. Yeah, and moving on, the next, There's a comment in the chat that there was a halt. From Nimbus on Fusaka 113.

Barnabas Busa: Yeah. I noticed that on, pretty much all Nimbus super nodes, the CL just stopped. It is not crashing, it just stops sending anything, stops receiving anything, stops logging anything, and this is with the latest minute release. So it's definitely something that they need to investigate.

Mario Vega: Alright, anyone from Nimbus in the call that, Just to make sure if you are aware of this issue. I cannot see anyone from anyone's in the call. Can… can we follow up with this on… with them on… with the Inimus team? Via message or some… or something?

Barnabas Busa: Oh, we are already talking with them.

Mario Vega: Alright, okay, thanks. And just to say on that topic, is there any… anything else that's happening on… of DevNet, on the 3? Currently.

Barnabas Busa: Yeah, a bunch of nodes, ran out of disk. I did some maintenance today, I cleaned up some databases, I reduced the number of blobs, so hopefully, we're not gonna run into the same issue, because the blob, database size was over 50030GB. So we were just spamming too many blobs. I reduced it now, significantly, so hopefully… We're not gonna have machines running out of this for the next month. And we… we should consider shutting it down, once, mainnet is worked.

Mario Vega: Okay. Okay, thanks. Does it mean that… So we're spamming a lot of blobs in this network. Does it mean that we are not cleaning up correctly after the… The time period where they should be cleaning.

Barnabas Busa: Not really, because, at 20 blobs per block, a 10-day period is… It's supposed to be around 640 gigabytes, I think. And because we were spamming 20 blobs a block, it's… yeah, all the nodes slowly started to run out of disk. All the super nodes. Pull nodes had no issues. Of course.

Mario Vega: Alright. Thanks, Okay, anything else regarding, Alright, we have someone from Reth. Yand, do you mind, Sharing a little bit about the, the bug that you guys hunt. Last week?

Jen: Yeah, I just added the… sorry, can you guys hear me? I had also issues with my mic. I just shared… I just shared also the release notes, so the V191. That should be the release for Pazaka, which includes a patch. This came in with, like, an EVM regression, as mentioned before, that was already introduced for preparation for polls. And yeah, we're taking measures to make sure we don't have any further search regressions. I've been talking to the testing team. I'm on tests to include.

Mario Vega: Perfect, thank you. And I can see that the blog is already, updated for this, so we should be good there.

Jen: Thank you, Feng.

Mario Vega: Excellent, thank you. Alright. Anything else Fusaka-related? I think, the only remaining things are we have to investigate the… strange requests in mainnet. But anything else apart from that?

Barnabas Busa: It would be good to hear from Prism team, whether they would have a stable release this week. Because, right now they have a Treasury lease only for Milo.

Mario Vega: Anyone from prison that would care to shame in right now? Alright, I guess not. Could we follow up with them on AC… I mean, it's… is it too late to follow up with them on ACDC, or… What other avenue do we have on there?

Barnabas Busa: Yeah, I already reached out to them, but I got no response, so… I'm not sure where everyone is from, personally.

Mario Vega: Alright, yeah, I think we have to maybe keep on trying to reach them. Otherwise, ACVC, I think, is the next, A spot where we can find them. Hi, Casey, yes, exactly. We're, we're, we were wanting to hear about the Prism release. Here, you can share.

Kasey Kirkham: Sure, yeah, Preston right now is, working on putting a release together, so we've been soaking over the weekend, and things are looking good, so we'll make sure to give an update in the, the Steel server. But yeah, we're looking at, Building that out into a release as soon as possible. Hopefully… hopefully today. So, yeah, we'll give an update in the… in Discord.

Mario Vega: Great, thank you, thank you. Okay. Alright, anything else related to Fusaka? Alright, if not, I think we can move to the next topic. So 60 million gas, I think, We already discussed in the previous weeks that all clients are ready for $60 million a minute. So, we can basically just jump into benchmark testing. Do we have anyone from NetherMind that can update on this?

Marcin Sobczak: Yeah, so, we have, Brantis from Nethermine and GAF with EIP7904. And, we did some, like, initial testing, and And, yes, simply intestine benchmarks are, working as expected. And at this point, we need to generate all the new payloads to match the consensus rules after this EIP. So it's our next step, and after finishing this, we will be able to run a dashboard with, with, like, instances of, standardomite, standard GAF, reprised Netheramide, and reprice GAF, and compare the results. So, like, we are at the step… at the step where, we need to generate, new…

Like, new payloads for these benchmarks.

Mario Vega: Alright, thank you. And there was… and there's one comment, .

Ameziane Hamlat: Yeah, I can, I can give the details. So… Testing7904, so if you keep the same dashboards, you may see, maybe a drop in the M gas per second, so I guess what could be interesting If, if you want to check the difference between, like, current schedule and… 7904, you may need to, to include TPS, which could be, an interesting… Yeah, and at least two metrics.

Mario Vega: Thank you. Okay, anybody else, that needs to shameen? Either under repricing or benchmarking testing? Alright, we also have, a call, I think, this week, or is it next week, about gastropricing. So please also… Let me see… When is the next, repricing call? I think it's next week? Do we have?

Maria Silva: Sorry, are you… are you asking about the breakouts?

Mario Vega: Yeah, I mean, just generally about gastrophe.

Maria Silva: seem very good.

Mario Vega: Yeah.

Maria Silva: Yeah, so we had two. We had one last week and one two weeks before, and now we don't have any else planned. because it was an overview of all the EIPs, I mean, if people find that the specific breakout or repricings would be helpful, I'm happy to schedule a few more, but with DevConnect coming, I think Maybe it wouldn't make sense to do it right, right away? But yeah, we don't have any breakout schedule at the moment.

Mario Vega: Excellent, okay. Yeah, yeah, I was just wondering. Thank you, thank you for chamining. Is there recordings of this, general pricing, breakout?

Maria Silva: There are. I'm happy to share after.

Mario Vega: Excellent. Thank you. Alright, yeah. If nothing else on gas or pricing, or estate testing, or benchmark testing, I think we can move on to Glamsterdam. So the first topic is, lock-level access lists. Anyone care to chime in on… Updating the current status. I think the main question is whether we are planning on launching, DevNet at the end of the month, still.

Parithosh Jayanthi: Yeah, so that is still the plan, and Stefan is on his way to DevConnect right now, so he can't make it, but he asked us to get two questions resolved. The first one is if the block-level access list tests are fixed, and I think Spencer is looking into it, and we might have been using the wrong release, so that's probably the cause, is… Yeah, Spencer, do you have any other information for that?

Spencer Taylor-Brown: Yeah, so they're… they're, Basically, the most recent block access list release, the version 1.4.0, is different from the previous releases. Basically. Previous releases were block access list specific tests. But this new, I guess, latest, the one that's, that needs fixed, contains the block access list specific tests, and then also… All of our existing tests refilled full block access lists. So I think, there was a… 7702-related issue, with that. release, there will be a patch for that in the next release. Also, we're running these tests right now in Hive. And all the blood tests are failing because we need to fix the… We need to fix the, BPO-related configs. On how we're starting up the clients with these tests.

Anyway, I guess, just a TLDR is that… Hopefully we'll have a next… the next release either tomorrow or the very latest on Wednesday, and we'll update… we'll update Hive as well. And yeah, these should be fixed. So I think Felipe from Steel has been working on this the most, and he… is out of office today, but he'll be back tomorrow, so, there'll be some action, for sure, either tomorrow or Wednesday.

Parithosh Jayanthi: Perfect, thank you. And, the second thing he mentioned was that, so Spammer has a basic EVM fuzzing scenario in there now. And even just running the basic spammer with, with this config in a local block-level access list network kind of makes it immediately. There's an example of a failure on Nethermind. Stefan said he'll also repost the entire Kurtosis config so that clients can look into it, but it sounds like it's very brittle right now, so it would be great if we can get some fixes in and we're ready to launch immediately, the entire infrastructure set up. As soon as it's not as brittle, then we'll spin up the network. We were hoping to still do it on Wednesday, but I'm not sure if clients get time to fix it that soon.

Mario Vega: Thank you. So what I can see from the results at the moment is that… so we… basically, the previous the previous releases, they did not contain other… other things that… than targeted tests, right? So what happened now is that we… we are filling every single test that we had in the repository with block-level access turned on, and now we have, basically. 1,000 tests at the minimum failing for every client. Something I noticed is that, I… from debugging in the logs, in the clients, I don't see that many of them are printing the full block-level access list that they are expecting. Right? So, I think, I don't know if this was discussed on the, on the local accesses breakout, but I think we need to make sure that all clients do print, if they find…

the block level access does not match what they were expecting. It would be nice if every single client, at least on debugging, they would bring the entire block level access, so we can easily debug and compare What the test expects versus what the client expects. So yeah, that was, that was something that, I, I, wanted, I want to share. Cool. Yeah, but yeah, come in, bye.

Parithosh Jayanthi: It's kind of the same ask we had with the debug, bad blocks endpoint. I think there, also, the request was to have not just the block-level access list that you think is bad, but also what you generated. And I think Bezos already done this, and Jared looked into it and said he will do it for Geth. I'm not sure about NetherMind and the other clients, though.

Toni Wahrstätter: Yeah, everyone will edit. This was part of the block of access list breaker call result result. At the very end, all the clients agreed to adding that.

Mario Vega: Amazing. Thank you. And coming back to the point of the DevNets, I don't think we should launch until we get this, this number of failures on Hive to a very… to a very low number, more manageable, because we're certainly gonna find more things. And it's gonna be, like, breaking all the time, at this point.

Parithosh Jayanthi: Yeah, for sure. We also don't want to launch it when, like, a basic fuzzler can take it down in half a minute or something.

Mario Vega: Alright. Yeah, thanks, thanks for the input. Anything else on block-level access lists that we want to share in this meeting? Okay, yeah, okay, so, and TLDR, basically just go to the Hive, Hive results page, look at your client, look at the logs, and please chime in. Also, of course, if there's anything wrong with the tests. So if you think you are in the right, in the correct, regarding rejecting a block load access list, please also chime in, and with the, with the testing team. either with, Felipe, Tony, or Rahul, we can, we can take a look at the test and also, also try to fix it if, if you guys, think that the test is incorrect.

Alright. Good. Moving on, to ePBS, Is there anything, anything worth of discussion today?

Justin Traglia: I can chat for a second. Hello? We had the ePBS breakout call on Friday last week. It was pretty quick, but we discussed adding an off-protocol value field to the execution payload bid structure. I think the name for this is Execution Payment. We also discussed including a previous Randall field in the bid, which is necessary. And calling, process builder payments earlier and process epoch, that's still undecided. We plan to… Include these changes in the next spec release, maybe within, like, a week or two, and then DevNet Zero will be based on that. I personally don't think that DevNet Zero for EPBS will happen probably until mid-January, at the earliest, due to DevConnect and the holidays, so just keep that in mind.

I know clients are still making good progress on implementations, And the next, breakout call, which was scheduled for, I think, November 21st, has been canceled due to DevConnect, so… I think the next one… was, it is December 5th. I don't… that's pretty much all I have, unless clients wants to give… wants to give. More detailed updates.

Mario Vega: Thank you. Okay, any clients? For any other updates? Yeah, I think that was… that was basically it. Thank you, thank you, Justin, for the update. Alright, I think that's basically the entire list of topics. The only other item in the agenda today is whether we are going to proceed or cancel with the next two weeks of calls. The next week is a little bit easier. Basically, is that kind of week. We are… a lot of us are gonna be over there. and discussing in person, so I don't think we should hold that meeting. The only open question is the week after that, because mainly… mainly because most people are gonna be traveling. Some, part of the IPANDOps team are also gonna be offline the week after that. So, I'm talking about the… specifically, the…

24 of November, the call on that. Is there any opposition on canceling that meeting? Should we… should we… or… What's the… what's the sentiment?

Saulius Grigaitis: maybe not directly answer to this, but I think most of the… or at least a significant amount of the teams are actually experiencing some, let's say, bugs or not polished releases, and And my general question is, how do you feel… how do you feel about that? I mean, there isn't that much time left to the main it, and… well, the bugs are still there, you know? So… Any thoughts on this?

Parithosh Jayanthi: Yeah, maybe just reiterating what Barnabas is mentioning, the question is how many of these are really Fusaka-related versus could have happened in any major release. I do think… For at least recent few months, we've kind of kept off on making major releases so that we can have everything merged into trunk branches, so… kind of makes sense that the first release would have some issues or the other. The only thing we can hope for is we're working on a mainnet shadow fork, so that we should be able to catch a few more of them before they go live, and I think it's also expected that clients would have a hotfix release if they do find anything. That's also one of the reasons why we

Have releases about a month before the fork, so that releases can soak, we can have extensive rounds of testing and catch everything.

Saulius Grigaitis: I'm… I'm just wondering, maybe there is some statistic on, how… Much of, like, these large, stakers or, or staking entities, have a redundant, setups, because just for example, data that was a bit nether mind on Hudi. At least on my instance, I needed to resync, the EL, so… So, I mean, if we… if the setups are highly redundant, in terms of, you know, multiple clients are used by the same staker, redundant setups, then I think these, these releases that, has, some issues, like we… have now. Maybe it's not a big problem, but if the setups are not that redundant and are not that diversified, then

There could be some turbulence after the hard work, at least based on the issues that we have now.

Parithosh Jayanthi: Yeah, for sure. We don't have exact numbers from everyone running every validator on mainnet, but at least for large staking pools, they tend to have a backup beaker node that they can fail over to. A lot of them use a testint vouch slash similar style of middleware proxies, where if there is a failure in one node, you can define a strategy to failover to a different node. And there's also SSV and OBOL-style clients on mainnet, so I don't think the answer is 100% redundant, but it's not 0%.

Saulius Grigaitis: Okay, sounds good then. Likely these issues will be covered by these redundant setups.

Mario Vega: Alright, and one comment from Barnum is that this is… One of the smoothest test networks since the March.

Barnabas Busa: Yeah, I think so. Like, we had many forks since the merge, and every single, or had some kind of bug here or there. I think Flipica has been one of the most stable since, since we started forking the testimonets. Yeah, smoother than actual merger as well.

Mario Vega: Alright.

Barnabas Busa: Yeah, I don't think there's much to worry about, to be honest.

Mario Vega: Any other concerns that, or opinions from other clients? Never mind, I mean, or Reth, did you guys, experience the, the issues? If you want to share anything, or…

Ben Adams: Yeah, we… We released a fix for it, like, 12 hours after the release. We were a little bit ambitious in the What we included in the, in that initial release. But yeah, it's fixed now, and we're… we're being a bit more cautious.

Parithosh Jayanthi: Maybe a follow-up question is, we do still have some testing bandwidth now that the releases are shipped. We are focusing on Shadow Fork, and then making sure that there are no high failures, but are there any other angles or… Specific things that we should be looking at that client teams are worried about, because we can then try to prioritize that over the next weeks.

Saulius Grigaitis: Do you guys do a non-finality testing? On a larger network.

Parithosh Jayanthi: Do you mean the, like we had in September? We did an opportunity test round then, but we haven't repeated that right now.

Saulius Grigaitis: Yeah, I think it would be valuable to have a… Last time, just before the… Before the hard work on the mainnet.

Parithosh Jayanthi: Okay, I think we can plan that in for after DevConnect in that case. Yeah, we will discuss that and get back to you.

Saulius Grigaitis: Great, thanks.

Łukasz Rozmej: Yeah, I have a question. We probably do have it somewhere, but, some stats for when hammering blobs for post-peer DAS in terms of, like, blob pull. Do we have that… confirm that everything is fine there?

Parithosh Jayanthi: Yeah, Sam's been looking into that, and hasn't found anything worth surfacing right now, but part of the problem is the blobs kind of originate from us, because we're doing the spamming, so it's hard to tell if there's a bug that is hidden, because we're basically reusing testnet infrastructure.

Mario Vega: Oh, I think, I think, going to Lucas' question, do you, do you guys have, like, any dashboard where we can… I mean, is there, is there anything worth, like… Or is this internal, and you only surface if you find any bugs, or do you have, like, public metrics, or…

Parithosh Jayanthi: It… mostly just ZATU dataset data. We can share something, but… Yeah, the question is, like, what we're looking for.

Łukasz Rozmej: So my, my question was mostly about performance degradation, or, like, Starvation, or… bandwidth starvation, or latency spiking, things like that. Because of just the high memory… high bandwidth usage of blobs per post-Fusaka, and mempool.

Parithosh Jayanthi: Okay, so, so, okay, got it. I think the Sunnyside Labs has some dashboards for that. We can, share that, yeah.

Mario Vega: Awesome. Alright, any other concerns from Clintense? Regarding current status? And there's a question from Casey about, private blobs and rear-heavy situations.

Barnabas Busa: We also didn't notice any issues with private blobs. We did many tests, when… The relay was the only one seeing the blobs, and no other nodes were able to use, get blob V2. So… We didn't see any issues with that. Rear communications might cause issues, with some specific ER clients. I'm not sure if… we have fixed all of that. I remember Netherland had the pre-org issues, they push out fixes. We haven't had a non-finity test since I would say maybe a month and a half now, so… yeah, we can definitely redo that on Devon Street.

Kasey Kirkham: Yeah, and if I can jump in, one concern I have on the CL is that when we reorg, you know, we exercise some pending block queue code paths that are not usually really, exercised in tests that don't have these reorgs, so, you know, we have a missed, block, not only do we need to get the block, now we need to go get, like, a super node might need to get 128, column sidecars, you know, and that can just put a lot more strain on the node. So I think that would be… if we have a way to test that in. some of the testnets, that would be good. And, and for the private blobs, I'm wondering, are we… are we continuing to test that, like, as we go through, like, Hoodie and so forth?

Barnabas Busa: We are not. We're doing only on DevNet 3. We don't… Basically, participate on testnet, manipulating testnet data. We don't run any relays, so this would be up to the relay operators to submit private transactions, and Yeah, I mean, they… they might have already done it, We do not have control over those.

Kasey Kirkham: Gotcha.

Mario Vega: So if I… sorry. Yeah, go ahead, Praszh, sorry.

Parithosh Jayanthi: There was, one more, bug that we were talking about, relays and, map boosts in itself. I've posted it here, Bharat, along with the new construct folks, dug into this. The issue was that if you have multiple, relays configured, then the… there's a race condition. It's now been fixed, and there should be a release soon. But yeah, just keep an eye out for that one.

Mario Vega: Okay, thanks. Just going back to the question of non-finality, are we doing that on DevNetri? At some point, it was at the consensus?

Barnabas Busa: Oh, we can do that, yeah. We can redo another one.

Mario Vega: Alright.

Barnabas Busa: I would just ideally want to have the numbers bug, Figured out before we trigger an infinity. Because as far as I know, that's the only place where Nimbus has detected issues with this.

Mario Vega: Alright, for sure, yeah, yeah, okay, so we wait until it's… that one's fixed. If we already reach out to Nimbus, I think we should be fine, and then we try to trigger the non-finality. Dimensory. Perfect. Alright. Any other comments from client teams? Alright, if not, going back to the original question, so do we keep… The column 24… For a CDT, or do we cancel? Is that… Or should we leave it open? Maybe you can leave it open and come back to it, and…

Alright, there's, boarding right now, do we cancel or keep it? It seems like… Alright. Yeah, it seems like we have to… we… we are leaning towards cancel, so… Yeah, let's just cancel. So, just to summarize, the week of the… sorry, the call of the 17th is going to be canceled, and the call of the 24 is also going to be canceled. We are back on, December 1st, for ACDT. Alright. Good. Any other thoughts? On, on… that need to be discussed today, or… because that's everything in the agenda for today. Alright, if not, I think we can, we can, we can end it here. Thanks everyone for joining. See you on… One… one raised hand. Casey, do you have last comments?

Kasey Kirkham: Yeah, I just… since I said before that I would follow up on the Prism release, I did hear from Preston that, he is planning on getting the release out today, so I think we can't expect that to be done today. That's it.

Mario Vega: Excellent, thank you so much. Alright, thanks everyone, and thanks for joining, and see you in DevConite, or hear you in, on December 1st. Goodbye. Thank you.

Antoine James: Thanks, everyone. Bye-bye.

Pooja Ranjan: Thank you.

Marius van der Wijden: What?

Louis Tsai: Thank you.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.