All Core Devs - Testing #50

Transcript

Parithosh Jayanthi: Hey everyone, let's give it another minute for people to join before we start. Okay, let's get started. So, the first item on the agenda today, is Fusaka updates. So, do we have Barnabas on the call for DevNet updates?

Barnabas Busa: Yes, we did a non-finality event on WN3 throughout the weekend, and this morning I have restarted all the machines that were producing bell blocks, and they're slowly syncing up back to head. Currently, we see a lower participation than before, which is quite interesting. … Right now, the major nodes that are offline still are, I guess, another mined ones, because those were pretty, in the images before. I have, here their database, and they should be able to sync back to HUD. Currently, we see about 36% participation on the network, though. Which is rather low, and it's a bit surprising that, about 20% of extra nodes have decided to Also, stop thinking.

Parithosh Jayanthi: So, does this mean that during the non-finality period, we had about 50-ish percent participation, and then once you started all of the nodes back on, we reduced participation again?

Barnabas Busa: Yes.

Parithosh Jayanthi: Okay, that sounds like they're not able to keep up with the sync load, right? Or is there some other prevailing theory?

Barnabas Busa: Mmm, yeah, that's exactly what I'm thinking. We have rate limiting enabled on all the machines, and I think, some of them are hitting that. Especially for notes.

Parithosh Jayanthi: What's very interesting is, none of the R machines are out of sync.

Barnabas Busa: So, basically, right now, the network is, held up by mostly our machines, so it might also be some cloud-related issue.

Parithosh Jayanthi: And at least according to logs, how does it look like it's going? Are the nodes progressing, or are they just struggling to find healthy peels?

Barnabas Busa: So, there's a bunch of nodes that are progressing still. Most of them have caught up, especially in the ES side. there's a few nodes, like Lighthouse GAF1 that have not able to get started thinking, and I think it's still the same syncing bug that Lighthouse previously reported. I'm not sure if anyone from Lighthouse is around.

Pawan Dhananjay: Yeah, I'm here. Yeah, I think it's the… I'm not sure if you're using the… other brands that I gave you on Friday, but, like, that… we found, like, a couple of issues over the weekend on that one as well, which was related to… like, a few bugs on our unstable branch, so we have a few fixes on unstable that we are going to merge into that branch today, and again, test if … stuff should work as expected, and yeah, I'll keep you in the loop. To… as to which branch to use.

Barnabas Busa: Yeah, okay. So, right now we have the blocks and columns on Lighthouse Gas 1, which is… which has stopped syncing, and we have Unstable on everything else. But you can update it to whatever you want. Yeah, so slowly the CLs are, forward syncing, but because we had a quite long non-finity, and because the forward sync is very, very slow on every single CL, it might take another few hours before we see any improvement. I… I would actually like to keep it as is right now, not really touch anything, unless we see something definitely wrong, and see if we can recover back to finally in, like, a day or two.

Parithosh Jayanthi: Yeah, that sounds like a good plan, and yeah, if there's any fixed images, then we should roll those out, but otherwise, I agree, I think we should… monitor how the nodes are going and take that data into account. What blob count are we at right now?

Barnabas Busa: We are on 20, I think? whatever BPO5 was, it's max blobs per block 20. But Spammer had a bug, and we are not spamming block transactions right now, but because debugging.

Parithosh Jayanthi: Yeah, but during the non-finality event, were we able to hit 20 blobs?

Barnabas Busa: I mean, we are still at the non-finity event. Yeah.

Parithosh Jayanthi: I meant the core non-finalty went over the weekend, at least.

Barnabas Busa: Yeah, we can check that, but I think so.

Parithosh Jayanthi: Okay, cool. Because those are the blobs that everyone would need… would sort of be struggling to fetch columns for.

Barnabas Busa: Yeah, so, like, even an hour ago, we had, consistent 20 blocks in the blocks that were proposed.

Parithosh Jayanthi: Okay, that's a really nice test. Thank you. Anything else you wanted to add on that, or anything the client teams wanna talk about? Yeah, Enrico?

Enrico Del Fante: Yeah, I just want to mention that, Couple of hours ago, we've discovered a bug in our, state upgrade from Elektra to Fulu. And, I share the link of the PR that fixes this. It contains a couple of things, but this is the main thing. So, essentially, we missed, the… The, porting of the pending consolidation from the state of, from the later state to the Fulu State. So… I think that we never tested on any testnet that across the FARC, there was actually a consolidation pending. So we always started with an empty consolidation, set. So maybe there is something on the spec test that needs to be addressed, or…

But seems, like, serious, so we fixed it there.

Parithosh Jayanthi: Yeah, this definitely sounds like a spec test, right?

Justin Traglia: Yeah, I can… I can look into this.

Enrico Del Fante: Yeah, let me know if you need more context on that, but the link should be enough.

Parithosh Jayanthi: And I guess….

Barnabas Busa: So we have….

Parithosh Jayanthi: make sure that we have at least one spec test for every one of these special requests.

Enrico Del Fante: I was thinking, like, we should constantly… whenever there is something that moves from the status, from a state to another state, we have to always constantly testing it. Even previous introduced stuff. So not just testing what the new status is introduced in by the fork, but also all the regression possible from the previous forks.

Justin Traglia: So, we do have fork, like, at fork testing. I'm thinking of, like, deposit requests and consolidation requests, but I'm not sure if we have, like, pending consolidation. At the fork, so that would be good for me to look into.

Enrico Del Fante: Probably the missing thing. Yep.

Parithosh Jayanthi: Awesome. Thank you. Any other updates for DevNet 3 or Spec? And I'm assuming there'd be another spec release, right, Justin?

Justin Traglia: Yeah, those. There'll be another spec release, probably. I don't know, a week or so.

Barnabas Busa: Is that gonna be an offer fix?

Justin Traglia: I was actually thinking of doing Beta Zero. If this would be the one that lines up with the first testnet, … Release.

Barnabas Busa: Is the idea that the EPBS and net zero is gonna be based on that, too?

Justin Traglia: Say it again?

Barnabas Busa: Is… is, APB as then at zero also going to be based on top of that?

Justin Traglia: Oh, I don't know. I mean, could be.

Parithosh Jayanthi: I guess it doesn't matter too much, because this is just adding tests, right? So, from a specification perspective, alpha 5 and beta 0 would be the same.

Justin Traglia: Yeah, it doesn't really matter much.

Parithosh Jayanthi: Yep. Any other updates relating to DevNets? We do have the MEV workflow working? Is there anything to report over there? Anything we're investigating?

Bharath Vedartham: Yeah, so, in terms of the workflow, yeah, last week, we were able to get the Private order flow thing, we, just kind of… we have… we're able to get a resolution with that, like… Basically on a high level, it looks like our builder was… only retaining, like, transactions for, like, 5 blocks by default. We somehow bumped that up to, like, 10,000 blocks, and things seem to generally be fine. I think the RBuilder folks also made that a configuration parameter, so… Future, we don't have to modify code for that. And yeah, the other good thing is, like, the RBuilder folks have launched their own builder, too, like, so we have, like, two builders submitting to the MethBoost Relay, which is nice, and generally, that's good testing.

Yeah, outside that, there's one thing which, Fahil from the PBS Foundation is looking into, is that, So there's one issue, again, you know, it's probably worth… it's not a very serious issue, it's just that when we transition from Elektra to Fusaka, it… we are… I think there are cases where a block simulation fails with the relay. That's because I think our builder is including some… pre-EIP, pre, Fusaka transactions without cell proofs, This is basically an issue with the logic, like, it's filtering out blobs for certain transactions, but including the transaction… the IP4844 transactions. So, that is something, like, I think Fayel is looking into. It's not a really critical issue, because that The mempool, kind of slowly, like, drains itself out of, like.

the, pre-Fusaka transactions without cell proofs. But yeah, I think File is looking to that. Maybe I'll also spend some time later on that. But outside of that, there's not much, with the MV workflow we release, like, builder specs. We merge, like, MevBoost to main. We're looking to merge MevBoost Relay to main, but we're waiting on, like, the attestant IOPRs, like. For GoBuilder Client. To get merged. I think somebody from Attestant.io has started looking into it and has raised some review comments, which I've addressed. So, I think… Yeah, we're looking to, like, merge everything to main, and there is that one issue which we'll… we'll see, like, how we can resolve it, or if that's even a problem at all, like, that's also not clear.

Parithosh Jayanthi: Okay, perfect. Thank you. Besides this, what is the DevNet 5 plan? Do you… maybe you want to tell us what the blockers are there, Barnabas, and we can get an update from clients as to where they are?

Barnabas Busa: Yeah, so the main blocker would be, trunk branches and, all major features merging today trunk branches. Ideally, we wanted to do that this week, but I think, giving an extra week would make the most sense for all, design teams. And in this case, we would aim for, all major features merged into the trunk branches by this week, Friday, and then maybe we can do Devnet launch on next week, Wednesday.

Parithosh Jayanthi: Do client teams want to chime in? Does this timeline work for you guys? Are we… do we have to change around dates? What is the… what's the general consensus?

Pawan Dhananjay: Sorry, did you say next week, Wednesday?

Barnabas Busa: Yeah, of course.

Pawan Dhananjay: Yeah, that's… that seems good for us.

Parithosh Jayanthi: Yes, this is for DevNet 5. Okay, it seems like most, if not all, clients have agreed to that. If for any reason you guys feel like we're missing something, then… Please reach out to us, and then we can still modify the timelines, but it takes a few days' worth of preparation to get these networks up, especially if it's over a couple thousand nodes. So it would be nice if we know well in advance of time, If we have to change our plans. Okay, the next agenda item is, one from Alex. Do you want to talk about the temp check?

Alex Stokes: Yeah, sure. … This ties in a bit with timing, for Sokka, and yeah, I guess one thing we could do here is couple the hoodie release with the mainnet release, and I just wanted to make sure that was okay with everyone. Rather than have, like, a distinct mainnet release. It would just help us pipeline things a bit, and compress. Otherwise, what would just be some downtime? Anyone have any issues with that? like, we have precedent for having test next coupled, and I feel like we've done this before, where it's… say a testnet and mainnet, but I just wanted to make sure.

Pawan Dhananjay: what would be the gap between the actual hoodie folk time and mainnet folk time? Is that the same, or like.

Alex Stokes: Yeah, that's a good question. So… I think the exact dates are gonna depend a bit on how, like, the next few weeks go with the DevNets and… say, Sepulia and Holushki? But yeah, I think I was thinking something… Okay, so we have this invariant of having a month between the release and mainnet. I think we could have hoodie somewhere in between that month. So then, to answer your question, yeah, something like 2-3 weeks.

Pawan Dhananjay: I think in that case, like… like you said, it would depend on how the other testnet folks go, like, if… Yeah, if the other testnet folks go pretty smoothly, then I think we'd be a little more confident to have, like, a smaller gap, otherwise… Yeah, we would maybe want to have a bigger gap.

Alex Stokes: Yeah. Okay.

Parithosh Jayanthi: Okay, yeah, I guess we're going to be discussing this more and more over the next couple of weeks and months, so… yeah, this was a September check. Any other Fusaka topics? Yeah, I think Roman's asking, do we have a tentative target date on when we decide testnet timestamps? Do we do this after DevNet 5, or do we want to do so, earlier than DevNet 5? What is our blocker here?

Alex Stokes: To me, it's mainly DevNet 5, so… If DevNet5 goes well, we can also do this, like, BPO analysis. We wanted to do it with DevNet4. That should give us numbers we feel comfortable with for mainnet, which we would then also use for all the testnets. So, assuming DevNet 5 goes well, then we can look to schedule Holushki and Sepolia. I think pretty soon after that.

Parithosh Jayanthi: Okay, that sounds reasonable. I think we'll try and have the DevNet 5 analysis up as soon as possible after. I think Sam's already working on a lot of the… the Jupyter notebooks, etc, so that once the data is there, it should be quite straightforward to plug in the data and then get an analysis out. Yeah, and we were also talking about reducing the amount of time between BPOs for DevNet 5, so it won't take a week-long testnet, but it might be a bit shorter. But yeah, I'm… I think we'll release a full spec for DevNet 5 and so on over the course of the week.

Justin Traglia: Do you have an idea of what the time between BPO work should be? Like, 12 hours?

Parithosh Jayanthi: 8 hours. I think about 12-ish hours, sounds reasonable, and then we do want to collect some data with MEV as well, so maybe 12 hours plus another few hours of MEV worth of data, MEV-style data, and then so on.

Justin Traglia: Sure. Yep, that sounds good.

Parithosh Jayanthi: Yeah, I think Casey's asking for potentially a written, DevNet and release timeline, but maybe that can be, for all core devs on ETH R&D, or somewhere where we can discuss it a bit more async?

Alex Stokes: Yeah, I can put something in the next few days somewhere, and definitely, yeah, we'll discuss by this week's ACD. Well, yeah, let's discuss there. But I'll get something async before that.

Parithosh Jayanthi: Okay, thank you. Any other Fusaka-related topics? Were there any other bugs on the DevNet last spec clarifications we needed to get checked out?

Justin Traglia: There was a typo in the most recent release, one of the new configuration variable names. I posted a comment about it on Discord, but just know it'll be fixed in the next release.

Parithosh Jayanthi: Okay, perfect. Thank you. Next topic on the agenda is, gas limit and bloatnet slash Performance DevNet updates. So we have an update from the sync test. We're able to get sync tests running well on Geth as well as Bezu. Both of these clients also offer a flag similar to a required block, which has been very helpful on these smaller shadow folks in terms of finding peers. The others are struggling to either find peers or maintain their peers, so it would be great if Nethermine, Wrath, and Aragon could provide us with a similar flag. We flagged this in the state floor channel as well, but if for some reason you're not on the channel, or you want to discuss this, then, …

yeah, please DM someone from the PandaOps team, and we'll put you in the right channels. And is someone on the Bezu team here who wants to talk about 60 million gas update? I see that they had a comment on the PM repo about it.

Luis Pinto: Yeah, I can share some updates. So, we've noticed that we are highly impacted by the divisions, so Maude, Steve, and we are working on that. We've tried, … couple things, to avoid, a big overhaul, but I think we are gonna have to, do the NUTS implementation from scratch in Java, avoiding all the libraries, so, yeah, we're working on it, right now. And… I had some more updates from Simon.

Parithosh Jayanthi: Could you clarify which, library you're talking about?

Luis Pinto: The big integers, Java library.

Parithosh Jayanthi: Got it, okay.

Luis Pinto: Yep. We… we have some kind of… we have a bit of, … memory churn, object churn, because of the big integer, which we would like to avoid to… to make things faster. So we… we need to do, arithmetics directly on the… on the, on the bytes, or, or primitive arrays, basically. N… Just give me a second, I have a… I had an update from Simon. which I'm trying to… look for… Yeah, so he says also that there's no EC recovery issues. … Yeah, and so we're working on this, … on these benchmarks that now run EAST opcodes, or, test cases. Which we hadn't had a chance to run on our benchmarking.

… And next, unblocking 6 million gas, by improving EC ads. We also need to work on that, but it's now less urgent given the other problems that we have with more basic opcodes, like division, and so on. Yeah, that's pretty much it, what he passed along.

Parithosh Jayanthi: Okay, perfect. Thank you. And do we have someone from the Geth team who wants to talk about maybe their bottlenecks, or would they prefer to do that async later?

Marius van der Wijden: I can quickly talk about it. We haven't… like, the big one, the only big one is Modex. And we haven't, resolved our strategy, how to move forward with it. We have 3 different ideas. Either we go with, with, GMP, Or we, … We make a proposal to update the standard library. Or we pick some of the edge cases in our own code. All of these three different ideas have different drawbacks, and we are still debating, how to go forward with it. But it should… We should at least… resolve which way to go forward, this week, and then we're… depending on the strategy chosen. It would take, us a bit to… to get there. But it's also…

I think the general consensus is we don't feel like… ModeX is something that will keep us from reaching 60 million. At this point.

Parithosh Jayanthi: Yeah, thank you for the update. And next on the list is, Carlos, do you have any updates or anything you want to talk about, relating to state growth? Or would you prefer to do that next week?

CPerezz: Yeah, not much to that from my… I just came back from out of office today, so yeah, not much to say now. Working on all of the test cases we already defined, and waiting on… Ref to… to get back to us on the questions we have for client teams, such that we can complete all of the test cases, and we can move forward with implementing as much as possible.

Parithosh Jayanthi: Perfect. Thank you. Okay, the next topic was, … I wanted to get a temp check on clients' opinions on tools like OpenTelemetry. So, one of the things we've been noticing is it's getting extremely hard to, sort of, deduce data just from pure Prometheus metrics, or from logs. Both of those are not very easy to do any data-driven analysis. Especially if you're going down more on the performance optimization route, it might make sense for us to have a standard system, something similar to OpenTelemetry that clients implement, and we can ingest and use that as a comparison point. It's completely opt-in. I would assume that it's even disabled by default, and we just run them on the performance images, as and when required, but it could also be something that's never merged into master, but lies in the performance branches of clients.

Yeah, we just wanted to get a temp check on how this is, how plants feel about this. It could also be very useful for the future when we're talking about gas repricing. In terms of, … Yeah, figuring out if we've actually priced the gas correctly according to all the clients. Like, benchmarks are sort of treating it like a black box, but OpenTelemetry would give us actual data from each client as to how the performance is. Yeah, I know that Prism has OpenTelemetry already, and wanted to know if, yeah, other clients have opinions on this. To Alexi's point, we can maybe compile some examples, or do you have something? Yeah. Okay, I don't think we have an example directly right now, but we can share some on OrcoDevs and maybe…

Yeah, see how clients feel about it there. Okay. Yeah, we'll talk about that async then. The next topic on the agenda was, … EPBS update? Do we have someone from the EPBS working group who can give us an update on how things are going?

Terence Tsao: POTUS is not here. I guess I can give a quick update. So, … Let's see, so this Friday, first of all, this Friday, there's a breakout call, so highly encourage you people to join the breakout call if you're working on EPBS, or looking at Glemson Day in general. Other than that, … I have been mostly focused on, consensus spec tests and consensus spec update on EPBS side. And, the… currently, the SPAT tests have pretty good not pretty good coverage, but struggling to have coverage on just basic, like, SSC structures, so you can run SSE spat tests. There are a few, like, open questions where I'm still resolving that it's just, like, some of the renaming things.

Making sure the fields are in the right position and stuff like that. I don't want to, like, go over them so much in detail, because, like, it's still relatively early-ish, but then I do think, like, this will be brought up in the EPBS breakout code, if you want to pay attention to… if you… if you want to follow those. But yeah, but currently, I think most of the focus on just getting this back in the right stage that, like, a client can start implementing it, and, which I do believe client can start experimenting it already and give feedback, because, like, Tagu and Prism already had a finalized DevNet earlier this year, and then, other than that, yeah, it's basically just fact changes, so yeah.

Happy to take any questions, feedback.

Parithosh Jayanthi: Cool. Is there anything you guys feel like you need from a testing or tooling point of view, or is everything okay there?

Terence Tsao: In terms of tuting, I think once we have some prototy stuff then running, we definitely need tooling just making sure DORA can show, like, the right page, making sure, like, for example, like, 4K, the four choice, also can consider, like, the notions of, like, an empty slot, for example. And I think there will be a lot of tuning, like, a lot of tuning, like, needs to be updated. I think it may be good to start thinking about this, like, what type of tooling may be affected in the EPBS land. But yeah, definitely a lot. And, yeah, I think it's probably good to start sooner or later.

Parithosh Jayanthi: Okay, cool. I think we can put that on our plate for once. Yeah, probably in the next month, we should be able to start looking into it. Do we have Tony or someone else for an update for block-level access lists?

Toni Wahrstätter: Yes, I'm here. Yeah, quick update, we have a breakout call on Wednesday, 2PM UTC, so… For everyone interested, please show up. The main topics will be some final discussions about the scope of the EAP, and also then getting into testing, specs, and yeah, how to get to a definite.

Parithosh Jayanthi: Cool. I think we merged in support slash have support for Glamsterdam fork activation on our Genesis tools, so it should work on Kurtosis as well, in case you guys are ready for local testing.

Toni Wahrstätter: Perfect, yakra.

Parithosh Jayanthi: Okay, Sunnyside Labs, do you guys have an update? Do you want to share anything?

Minhyuk Kim: Now, we don't have any reports this week, but we, our current work is, analyzing some issues with Perfect PureDesk Network, so we have a few issues that we have identified in a few clients and are communicating directly with the client teams. And another thing that we are working on is we are building a chaos testing tool to try to stress the network in a more aggressive way, so… Like, in addition to applying stress to, like, RAM or CPU, you're also adding features like, splitting the network into half. And Trying to see if the network progresses as usual after merging the network back in. So if you guys have any specific chaos testing scenarios or things you guys would like to test, you can

try to simulate it in our DevNets.

Parithosh Jayanthi: Awesome. Thank you, really looking forward to those PRs and awesome job with the chaos testing stuff. We have an open floor now in some time, so are there still any open topics that people want to bring out? Okay, last call… And… if not, then that's it for today. Thank you all for joining.

Justin Traglia: Bye, everyone.

Andrew Ashikhmin: Thank you.

Fredrik Svantes: Thanks, bud.

Mario Vega: Thank you so much.

Marius van der Wijden: Gosh, bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.