All Core Devs - Testing #88 (CL Breakout)

Transcript

Justin Traglia: Hello, everyone. Just one minute, and then we'll get started. So there is nothing on the CL breakout agenda, so I think we could just use this time to… discuss how things are going, or any… just anything that comes to mind. So… How are things going?

Saulius Grigaitis: I have a question. When are we going to have a bit… if you have a testing on the… on the… Devin, that's, like, you know… some more chaos. Previously, I think we had some blobs spamming and so on. And maybe it's a good time to try to have more efforts on that. I'm trying to break the… To them not.

Parithosh Jayanthi: Yeah, a lot of that is planned for this week. I just need the Aussies to wake up to merge in a few PRs for me, and once that's done, I can release our malicious CL on the network. But yeah, until then, we can just do, like, basic spamming and stuff, that should be easy, we can start after the call.

Saulius Grigaitis: Yeah, I think, you know, as the network is getting stable, more stable than it was before, I think it's a good start, a good time to start breaking it again.

Justin Traglia: Have we been testing the new 300 million cast on it?

Parithosh Jayanthi: Not acutely yet, but we can also do that.

Justin Traglia: Okay. Do you know if we've submitted a, I should say a payload with more than 8,192 deposits. To test that.

Parithosh Jayanthi: I actually don't think so. I can do that now, too.

Justin Traglia: And what is the blob limit on a StepNet? Is it 21, just like Mainnet?

Parithosh Jayanthi: Yep, we… I don't think we touched the block limit at all.

Justin Traglia: Understood And while you're talking, what does the malicious prism node do differently than Death Star?

Parithosh Jayanthi: It works along with the builder, so we have a different architecture now. Earlier, you just had one node that kind of had its local view of the network, and instead, we now have a malicious Prism node, but also a malicious orchestrator. And the orchestrator has a more global view of what the builder's doing, what the Prism node's doing, and then it can control what each of these parties do to create more edge cases.

Justin Traglia: Okay, yeah.

Parithosh Jayanthi: Mmhm The edge cases could be that the builder releases the payload later, but at the same time, the proposer is withholding some blobs, some weird situations like this, I guess. But besides this, I think a lot of edge cases are already covered by Death Star. Just malformed payloads and stuff like that.

Nico Flaig: Yeah I was just thinking because I have some more features planned for Devstar but yeah maybe we can coordinate who implements what I mean all the late payload stuff block withholding egg relocation and stuff like this For some, you need a proposal, I guess. In theory, the proposal can do everything, right? You don't really need a builder, but yeah.

Parithosh Jayanthi: Yeah, the main reason we wanted to add some of these topics in the builder is it's just new paths, so even if we don't find anything, we still want to assume that the builder can always be malicious.

Justin Traglia: Random sort of unrelated question. If we're planning on having a DevNet 8 and a DevNet 9, that would probably be, like, another month of DevNets. When do you anticipate we'll start upgrading testnets?

Parithosh Jayanthi: I think my ideal would be September. Okay. But it really depends on if we're finding anything in the next month. It's also trying to align with the schedule that the ELs have for more final repricing numbers. So we need at least one large-ish, public-ish DevNet. So I'm hoping that can this be DevNet 8, and let's say we do that two weeks from now, so that's first-ish week of August. We ask a bunch of people to deploy stuff on the DevNet, try and break stuff. I'm sure there's a bunch of contract developers who are going to hate it. It'll take, let's say, roughly 2 weeks for us to process that information, and then collect a bunch of data on what the actual repricings would look like. Parallelly, we can have a non-finality DevNet. I think there's, like, 3 or 4 non-finality things we want to do, along with…

One potential decoupled consensus test. And at that point, I think we've done all the testing that we normally do. We would have done malicious proposals, we would have done basic buzzing, we would have done benchmarking, we would have done non-finality.

Justin Traglia: Okay.

Potuz: Christine, apologies, I can't raise my hand. Can you call me a little bit?

Justin Traglia: I'm sorry. Say it again. Can I call on you.

Potuz: Yeah, I wanted to chime in on this.

Justin Traglia: Oh, yeah, I mean, you can chime in now.

Potuz: Okay, great. So… I would focus less on consensus. I mean, everything that Pari said is critical. That's fine. But the thing that we need and we need to test well before testnets is the builder API. We are not testing with external builders being called with signed authorizations, builders that care about authorizations, builders that don't, and so forth. those paths of the validator client and the key manager API, And all of these things about sidecars are the things that need to be tested.

Parithosh Jayanthi: Would, as far as I remember, on DevNet 7, the clients don't have that code paths implemented, right? Is that something you guys can have for DevNet 8? Okay, Prism has it cool.

Potuz: It's not merged yet, so it's not really standardized. That's why we need to have at least the post version of ProduceBlock And the builder API finally merged, I guess.

Parithosh Jayanthi: Okay, so there's, from the chat, I take that there's a Key Manager API PR that's probably not merged yet, or is there no candidate to begin with? And, the Builder API PR, that has to be merged, right?

Terence Tsao: I'm gonna beat today for.

Parithosh Jayanthi: I'm guessing that's a beaten API here?

Justin Traglia: Yeah, so I think.

Potuz: Yeah, there's a Beacon API.

Justin Traglia: so product… Notice related to testing with builders more. Maybe a bit of an unpopular opinion from my side, but I'm not sure that we should be testing — or the EF should be testing the — other builder flow. Like, are all clients gonna implement this, and… Also, I mean, our… We just need to ensure that the in-protocol P2P bidding works. Mike Float works. Yeah.

Dustin Brody: Yeah, wasn't there all this effort gone through to say, we'll get the peer-to-peer builders on, you know. epoch zero, so to say, of Amsterdam. I mean, I thought that was part of the point, is… To allow for this kind of thing.

Justin Traglia: I mean, I think that's separate. I mean, yes, we wanted to.

Dustin Brody: No, I understand they're separate protocols, of course, but, I mean, to reduce the dependence in this kind of decision-making on saying. There's some builder API being there.

Potuz: Like it or not. 99% of our blocks come through builders. I don't want to go into the circuit breaker just because we haven't tested this. It's fine if the P2P stack works, we're not going to die, but we don't want to be failing and then falling back to the circuit breaker.

Justin Traglia: I mean, yeah, I understand that 99% of blocks come from builders. But, I'm talking about we should let builders test out the direct connections between proposers and builders and the whole builder API flow. Let me do that.

Terence Tsao: It's not so much about like testing in a way it's that we need the API to merge so we can start implementing it. It's because like if we just like keep delaying API merging, then like I feel like we're going to merge it pretty late. Even when the testnet is happening. So the earlier people review the API, the better it is.

Justin Traglia: Yep, that makes sense.

Parithosh Jayanthi: Does someone have a list that we can track? Does the Key Manager API post it there? and a Beacon API here posted. Is there any other?

james: Wait, which Beacon API post? The one that Luca presented?

Parithosh Jayanthi: No, James posted one, BikiniPay address. Oh, I guess it's just deprecate slash validator config. Okay, sorry.

Potuz: There's a PR by me about adding a post or replacing the get to a post version of ProduceBlock before. I don't know if that's the PR that we're going with or some other PR that does the same thing, but we need that PR.

james: Yeah, so there is a key manager, PR that's based on top of your PR, photos. And then there's also another change on builder-specs, too, that's also related to… your your Pr.

Parithosh Jayanthi: Who are the people that need to review these PRs now?

Terence Tsao: I think there were a bunch of comments from last week on POTUS's PR, so I think, like, yeah, I think now it's, like, waiting for POTUS to review those comments.

Justin Traglia: Terence, do you know if PBS Foundation is going to support Any of this like.

Terence Tsao: I am not sure, but I know there is weekly… I don't know what the correct name is, they usually call MEV Boost Community Code every two weeks by the MEV Boost implementers. I haven't attended those in a while now, but those are hosted by PBS Foundation, and they're looking for, things to work on, so this would be nice to collab with them. I believe you're also in the MVBoost community on Telegram channel, so that would be good. I know CommitBoost, their team is very, willing to help testing as well, but they are all waiting for the API to, I will say, finalize and merge. So, yeah, so the… yeah, so I will say, the sooner we get more feedback, the better.

Justin Traglia: Okay. Yeah, I also haven't attended those calls in a very long time. But if you propose, if you bring this up eventually, that'd be great. Thank you. On this topic, are all clients going to add support for direct connections with builders? I'm just curious.

Enrico Del Fante: I think we're not yet, we are… Just planning the implementations of those.

Justin Traglia: But you do intend to have that.

Enrico Del Fante: Yeah, yeah, I think Stefan is… He's on it.

Justin Traglia: Okay. Okay.

Parithosh Jayanthi: Could we target all of these changes for the next DevNet? let's say ETA… Two weeks from now, latest.

Enrico Del Fante: I don't know.

Justin Traglia: Yeah, it's not a question for me. It's for clients, but.

Nico Flaig: Also, there's still some uncertainty, right? Like the builder preferences, whether or not we send them ahead of time or just in time, and… I don't know, there were, like, few builders saying they needed ahead of time, but then on Discord, I couldn't really explain why they needed, or there was not even… I think Alex just left a comment there, I haven't read that, but… I would be in favor of, like, not adding too much complexity of things that we are not really certain are any useful, so… If we can try to simplify the spec itself and avoid complexity, that would be good.

Dustin Brody: I will say, as far as Alex's comment, which I know to what you're referring, I found that, Sort of un… convincing, I suppose. So, it's essentially a timing edge case of a proposer saying they want to, that they're willing to accept a certain sort of category of bid very late in the time process. And, like, I'm not sure why this is… And his comment didn't say why this is so… critical, I guess. Because it's like, the premise was before the builders had the idea that this proposer was not going to accept a certain kind of bid, and then they changed their mind very late. And now they have hundreds of milliseconds. I'm like, OK. But that seems to be on the proposer.

At that point, more than anything else, in any case. I don't have hugely strong opinions on that, but if that's the strongest case for the ahead of time, I'm not sure I see it very well.

Justin Traglia: Is there anything else you'd like to discuss? Otherwise we can end the breakout session.

Potuz: So regarding testing, can you guys tell me what is Death Star doing now? Because I'm starting to run Kurtosis scenarios and I'm mostly focusing on how clients attest. And I see that the stations are going everywhere. So it's not clear to me that that clients are testing correctly. At least it's hard to pinpoint now. But but I think we should be focusing on 4 choice on those instead of like trying to get the chain to break, which would be like a chaos scenario, just trying to understand why clients are testing for the wrong block.

Nico Flaig: So I think one thing that makes it harder to understand on the DevNet right now is that we actually have late payloads, I think. So we started some honest reorgs, right? Payload reorgs. So that's a bit weird, because DevStar basically always reorgs payloads, and then if the payload was actually late, it's, honest behavior. So I guess that thing, like, I would need to handle somehow. But yeah, basically it just builds on empty, right? And then it also withholds the PTC votes in the block. Which I guess make it even harder to analyze what what's happening. But yeah, we should be able to have fork drill stumps as well, so…

Potuz: Yeah, I believe that Cortosis is better for this kind of scenarios, because then I can actually track the attestations, and I know what is the status of each payload, and the peering is controlled. So I can isolate the actual 4 choice issue instead of like trying to guess what what the hell is going on on the chain.

Nico Flaig: But you are testing late payloads, right? And honest re-orgs.

Potuz: Yeah, I'm testing like all types of withholding. And even long… and even long reorgs, like collusions between the builder and the proposer, and these kind of things. But what I wanted to stress is that the… I mean, the results so far, it seems to be fine, because if I put, say. An even distribution of clients, we seem to be working collectively correctly. But I still see some random clients at the stations that I just don't understand.

Justin Traglia: Okay, anything else we want to discuss. Okay, I propose we end the breakout session.

Potuz: Bye-bye, guys.

Justin Traglia: Hi, everyone.

Nico Flaig: See you. Bye.

cayman: See ya.

Parithosh Jayanthi: Right.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.