All Core Devs - Execution #248

Transcript

Ansgar Dietrichs: Can someone say something so that I know my audio works?

Justin Florentine: Good morning, Ethereum!

Ansgar Dietrichs: Perfect, that's exactly what I was hoping for. Alright, this is real life immediately, no… no actual waiting period. Okay, well, welcome, everyone, then, to… Lukasz… Number 248. I mean, I guess just because we're live, we might still want to wait for a couple seconds just for people to arrive. Just everyone please be at their best behavior.

Okay, then I think we can get started, We have a pretty full agenda today, so I'm not sure if we'll quite get through all of it, let's see. First, regarding Glamsterdam, there was one item that came up quite short notice yesterday, regarding the chain security, report, I think the report was actually published already last week, if I'm not mistaken, but… point being, basically, like, the… impact of some of the repricing EIPs, I saw that PC had some, had a comment on the agenda, but I'm not sure if he's on the call, or otherwise, if anyone else with context on this is on the call and could… Speak about this issue.

Hubert Ritzdorf: This is, working, just with me.

Ansgar Dietrichs: Unfortunately, I'm at least hearing… having some issue hearing your audio properly. Hubert…

Hubert Ritzdorf: Is it elected store?

Ansgar Dietrichs: Still quite hard to understand, I mean, maybe if it's easier for others, but I saw in the comment also some… some… in the chat some comments that there was a bit of a problem.

Hubert Ritzdorf: Sweet. 31 seconds.

Ansgar Dietrichs: I mean, we could also otherwise, I know that some people look… Now it sometimes sounded much more clear.

Hubert Ritzdorf: Sorry, is it better like this? Yes.

Ansgar Dietrichs: Much better, much better. We can hear you very clear now.

Hubert Ritzdorf: Can you hear me, Dawn? Okay, thanks. Yeah, I hope it's better, otherwise, please.

Ansgar Dietrichs: We can hear you really well.

Hubert Ritzdorf: Yeah, sorry for the… for the technical issues. Yes, so we were, or I was asked to say a few words here. In order to hopefully make this more understandable, I have prepared a few slides. I don't want to keep too much time, but I thought it might be more understandable with the slides. Feel free to interrupt me at any time if something is not clear. So… This is, joint work by the whole chain security team, especially Toni, who unfortunately cannot be here today. So, what we want to talk about is potential security issues with smart contracts that are introduced due to the changes in Glamsterdam. So, we are aware that Ethereum Foundation and others have quite extensively looked for transactions, good, I call them honest transactions, that worked before Glamsterdam.

and are no longer working. After Glamsterdam, a lot of work has gone into that. That's not what I want to talk about at all. What I want to talk about is transactions that… evil transactions, that did not work before Glamsterdam, but are working now after Glamsterdam. So that's what I want to talk about, and that's what we are concerned about. And to… to explain, I will present a somewhat contrived example, so you can afterwards all say, okay, but no one writes that type of code. But I didn't want to present an example that is too close to reality, because we're currently going through a responsible disclosure with multiple DeFi projects, and I kind of didn't want to.

violate the rules of discon… Responsible… responsible disclosure, but… I hope the example explains what we are concerned about. So, let's imagine we have a smart contract, and that smart contract has a withdraw ETH function, so it lets someone withdraw ETH, and the withdraw ETH function Receives a signature by the withdrawer, and then has permissionless execution. So, then afterwards, the function sends… attempts to send the ETH to the withdrawer, and if that fails, it uses the well-known self-destruct trick to send the ETH. Why? Because the self-destruct trick always works. And because that trick always works, it also does not really check the return value, of the self-destruct trick. So you could now say, this is not great coding practice, and, yeah, that can, of course be argued, but that's kind of not what I want to discuss, because,

I want to discuss potential impact of such code. Afterwards, the withdraw ETH function adjusts the user balance, so it does like an SSTORE. So, let's see what that looks like in a chart. So, here we have the permissionless caller. So the permissionless caller calls the withdraw function, and that's our… on our victim contract. The victim contract, I left out all the other parts to simplify it, it just does this A self-destruct trick here now, which works with the create. and the self-destruct, and everything, works fine, the create returns, it does this SSTORE to update the balance of the withdrawal to zero, and the transaction succeeds. All is good. So, this is a contract that is safe with Osaka rules. Why is it safe? Because if there's sufficient gas, the safety structures works, and all is fine. If the self-destruct runs out of gas, the victim will only have 1 over 64 gas left.

And, because we didn't do any kind of special cause, and the victim needs at least 2,300 gas for this SSTORE, and hence, the… we can do some math and say this is safe, because if this were to be unsafe, the self-destruct would have consumed at least 144,000 gas, but it consumes, at most, 32,000, so turns out it's safe. So this is safe when running out of gas, this is safe when with sufficient gas, so this contract is safe. Here we have it, just briefly recap. Visually, if the self-destruct runs out of gas, the create, returns, but now the victim only has 520 gas to continue, 520 is nowhere near enough for the SSTORE, the transaction reverts, all is good.

Now, what can happen with Glamsterdam? With Glamsterdam, this self-destruct can now cost 200,000 gas, in case this does create, this does create a new account, because then 180,000, roughly, are spent as the new account state, gas. So… For the… What happens now, if the self-destruct only has 200,599 gas available and runs out of gas, then this force send, this intermediate sender we were using, will revert. But now, due to the New Glamsterdam rule, the victim will actually recover this state gas that we spent here, right? It will be recovered immediately, the hundred… 83,600, and hence the victim can now execute the following SSTORE and update the balance. So, let's look at this, kind of individual representation.

So, now we're in Glamsterdam, hence the gas amounts are bigger. So, here we are forwarding 200,000 gas, roughly. However, even with the 200,000 gas, the… self-destruct runs just out of gas, right? And here, now, in the revert. these 180,000 will be refunded, hence the victim is now executed with more than 180,000, hence it can perform the SSTORE fine, hence the transaction succeeds. So what happened here overall. the ETH was not transferred, but the balance was updated, right? Like, so, as I said, this is a contrived example, I realize this. However, I chose this example because, of course, I want to take a limited amount of time, and hence want to take something that's as simple as possible.

as a consequence, anyone, because in our example, this has permissionless submission, right, anyone can decrease the user balance without the real receiver receiving the ETH, by just front-running the real transaction with a transaction that has too little gas. And then, now you might say, okay, but they have no advantage, but in many cases, the attacker might be able to steal the stuck funds afterwards. Okay, and this is… this was just one example, one tiny example, that I wanted to give to… give you a concrete example, because I personally feel like these concrete examples helped. But Glamsterdam also breaks other developer assumptions, such as they are hard-coding certain gas amounts, either as reserves to keep inside the contracts before they're calling something, or as past amounts.

Of course, we all know hard-coding gas amounts is not great, but unfortunately, that's, that is there in a bunch of contracts. Developers sometimes have the assumption that gas left will only decrease, which is no longer correct. Within a single frame, the… that certain contracts cannot fail, because they… yeah, in the example that we saw before, where we basically said, okay, kind of this self-destruct contract, it's a simple, it cannot fail, As we know. the calling contract retains 1 over 64 gas left, and sometimes the developers argue this 1 over 64 will not be sufficient to continue. But now, as the gas consumption increases. the 1 over 64 also increases. So, the gas consumption might now be sufficient to continue. And, as we also saw, there's the special refunds.

Lastly, there are a bunch of developers that are aware of all of these problems and did out-of-gas detection in their contracts. However, the existing out-of-gas detection that they developed into their contracts can now also be bypassed in certain cases, and we have a blog article on how this can be bypassed. Yeah, so we, of course, invite everyone to read this blog article, hence I don't want to go into the details here. So, what have we done? We have… kind of looked at a few manual transactions, we have tried to search through source code with Sourcify. And we have written this blog post, to make people aware, and through these searches, we have, found 13 major DeFi projects that are affected by this.

And these, we have notified the 13 major… I mean, not… to be fair, not all of them are major DeFi projects. I want to say some of them are major DeFi projects. So… We have notified them that we are currently in responsible disclosure with them. However, what we have not done, and I want to make this very clear, is this search was in no way exhaustive, so we are not claiming we have seen everything, we are not claiming we're done. And we have… we even know of kind of very minor impact that we haven't even notified yet. For example, there's a bunch of projects where, you can invalidate signature nonces Which might be annoying to the project, but we can't… it seems like a minor inconvenience, but it's also not fully… we are not fully able to judge, we just didn't have time to notify those yet, and we have not at all looked at off-chain tools or front-ends.

or anything like that, where the simulations might be now incorrect, or they might still hard-code some gas amounts, or something like that. So, we don't know anything there. So… in a slightly provocative summary, I want to say, should Glamsterdam make contracts with incorrect development assumptions unsafe? And I guess that's what we're trying to what we're trying to ask, that's the question we're trying to… to get to, because we're realizing that a lot of the contracts that… that are becoming unsafe due to Glamsterdam, they have some, something which maybe some people consider bad coding practices, but they generally have, or they… the developer had a slightly incorrect assumption or something, but I guess still the question is,

how, how this should be dealt with, and, we feel like there's relatively little awareness, of this problem coming up. Yeah, thanks, that's, that's it from my side. Of course, I'm very happy to answer questions, yeah.

Ansgar Dietrichs: Sounds good, thank you very much, and I'm sure we have several There are people that would want to speak up on this, so…

Hubert Ritzdorf: Yeah, so the, so Ansgar wrote in the chat that there, Some things are… there's going to be complications when… when scaling. I… Agree that the, that, this is the case. So, as some might know, there was a similar previous case with EIP1884, where, like, also in the hard fork, there were some, contract implications. I just feel like they're… oh. Oh, sorry. I committed to you, sorry, sorry. Sorry, the… I quickly finished, sorry, and then I'll try to fix my speaker, sorry for all the problems today. It's, as usual, everything works, and then in the moment, you need it, nothing works. The, Yeah, so what I wanted to say is that, the, from my perspective, the… back then, there was a relatively good analysis done, I mean, which achieved the coverage that we haven't achieved yet, so that's all I wanted to say, sorry.

Ansgar Dietrichs: Sounds good. Well, I don't know exactly if I can speak to you right now, let me actually, one second, let me write in chat. In the meantime, maybe, if anyone else already wants to speak up. I see a lot of activity in chat, so if anyone else… Maybe once, too. Speak up on the question of how to handle. M… the situation? And… Hubert can… can hear us now, so that's… that's a good… that's good as well. So, so any questions, or otherwise? I mean, I also otherwise would have a comment. So then I guess I'll go first, because no one else is raising their hand yet. Yeah, I mean, to me, it seems clear, at least from this… at a minimum, before we go into the specifics of what even should be done here, it seems… it seems like it's

clear, assuming not all of these results are false, and I don't think… I've not seen anyone indicating that they will be, so assuming that these results are right… that would indicate that the impact analysis, kind of, like, done so far, has not… has just not been sufficient in its coverage. I think, Hubert, you already, at the beginning of the presentation, did this nice, very simple, like. extensive, extensive work towards just keeping valid transactions valid, basically, but maybe we… there was a blind spot on, keeping invalid transactions invalid, so that seems like… Clearly, kind of in the coming days and presumably weeks, it might not be so quick. One thing that will definitely have to happen is just, like, a very extensive, extensive work on this, making sure that we actually fully understand all the implications, all the affected contracts.

And then there's the second question of. actionable, what are the consequences? I think, I also, I think I saw this in chat as well, but basically, I think there's three different tiers of response here from our side. Like, one would be, of course, the most benign one, if If all of this can be addressed and mitigated in time for a on-schedule mainnet rollout, as planned in December. That would, of course, be the ideal outcome. Option two would be a delayed mainnet rollout, but with the same specs as we have done today. So that would mean we would allow for more time for mitigations, for analysis, making sure that we actually have a full understanding of the full list of affected projects and contracts, and then time for mitigation. And then number three, the most severe potential response would be a change to the Glamsterdam specs.

And so then, of course, we could go into, I think you… towards the end of your presentation, you had this question of. should… not having followed best practices result in contracts breaking, so basically, like, is it worth basically breaking anything at all, or not? I think that is an interesting conversation to be had, but I feel like it's almost like it's the third step, so only if we were to get there. I think the first question would just be, how can we As fast as possible, make progress towards understanding the full scope here, and working with the affected teams. So that we can then even inform the decision of, do we A need a delay of the mainnet rollout, and B potentially even need adjustment to specs?

Yeah, I'd be curious if that makes sense. So basically, like, first focus more on, rapid analysis, before we can then make… assessments of consequences. Is that, in general, approach-wise, reasonable for people? Toni?

Toni Wahrstätter: Yeah, I think it's reasonable, and my feeling is it's just very difficult to understand the exact impact right now. And I wouldn't even want to make, like, timeline decisions based on that. It would be better to first, like, get a clear understanding of the exactly… of those that are affected. And based on that, we can still, discuss delaying mainnet. If, for example, affected parties need more time or have difficulties upgrading.

Ansgar Dietrichs: Yes, this makes sense. And so, like, just because that would already tie in, we're not going there yet, but that would already tie into the later, agenda points, so we were planning to talk about, well, for one, the Hoodi testnet fork, but then also the mainnet rollout candidates, and so, just for clarity here, I think we all agree on this, but just to state it explicitly, of course. no decision on mainnet dates, mainnet rollout, or even potentially the potential for changes to the Glamsterdam specs would be made until there's a clear, thorough understanding of the impacts here, the affected teams, so until that… until we have that understanding, basically, like, the mainnet rollout would be on ice, basically.

Justin?

Justin Florentine: I think we should probably talk about Hoodi, since that's coming real fast. It's potentially gonna be something we're gonna need if we need a control to compare tests of these types of, issues versus what's already been deployed to Sepolia.

Ansgar Dietrichs: Can you expand on this? When you say control, do you mean in case we would want to have spec changes?

Justin Florentine: How many places do we have test networks that we can, hold on, I'm being attacked by a robot vacuum. How many places do we have test networks that we can actually craft these types of transactions and send them to DeFi projects? that have not been updated to Glamsterdam to compare to. That's what I mean by a control. Like, if we have this issue, we're gonna try and reproduce it on Sepolia. Great. Where's the control that's not mainnet?

Ansgar Dietrichs: I see, so you're saying, basically, you would… in your mind, that would justify keeping Hoodi on the current specs, so that we basically have one testnet on old specs and one testnet on new specs to compare.

Justin Florentine: That's right.

Toni Wahrstätter: Of course, this assumes now that we actually changed the specs, right? Because I think it's not so clear that we actually need to change the specs at all. Maybe we can just notify the affected parties and they upgrade their contracts.

Ansgar Dietrichs: I think I understood Justin's comment as actually not assuming that spec changes would be necessary, just saying that while we are basically trying to understand the impacts better, it's good to have one testnet on each side of the fork, basically.

Justin Florentine: Because the only other alternative would be get, like, heavy DeFi project participation on a synthetic devnet.

Ansgar Dietrichs: Right. Daniil, maybe go to you first?

Daniel Lehrner: Yeah, yeah, just, just, I, I just wanted to say, if we have to change the specs. We have to throw away Sepolia for sure, because it's on specs that are not on mainnet anymore. And if we do the Hoodi hard fork on Monday as well, we have no testnet, because where would we test the hard fork? Because both would be incompatible. So it's not… Do we need spec changes or not? It's if we need them, where would we even test them? If both of… if both Hoodi and Sepolia are on the wrong specs?

Ansgar Dietrichs: Justin, you still have your hand up? Was it again, or… Potuz?

Potuz: Yeah, I want to second what Daniel said, and I want to stress that if we… we are going to be able… we're going to be forced to ditch testnets that are on the wrong fork if we actually change the spec. And ditching Sepolia is not so bad, but ditching Hoodi is bad. It affects a lot of parties that it's much more decentralized, we have, like, a lot of validators that are going to have to be deployed again, so losing Hoodi would be a problem. Sepolia wouldn't be so much of a problem. So I would lean towards delaying Hoodi, just in case that, we might… Changes back because of this. Otherwise, if we change the spec.

Lightclient (Matt Garnett): I mean, sounds like we should delay Hoodi. I don't really hear a good argument to go forward with the fork when there are questions that we need to answer, and losing Hoodi is not good.

Parithosh Jayanthi: I… my main counter-argument is that now that releases are out, we would need everyone to have newer releases, and make sure that all of the community has updated the newer releases, which is a non-zero risk of just a failing fork. And if we have a failed fork, then we have to spend a bunch of time actually recovering that, versus,

Potuz: We can just say that…

Parithosh Jayanthi: Hoodi as you…

Potuz: No releases party. We can just say to drop back to the previous… to the previous clients.

Parithosh Jayanthi: True. Yeah.

Ansgar Dietrichs: Lukasz?

Łukasz Rozmej: So I think we can have Glamsterdam, let's say, V1, that is right now on Sepolia, and if we have to make a change to the specs, we have Glamsterdam V2, and then it goes to mainnet, so basically Sepolia has this weird timeline when it's, you know, not all, everything is activated the same way as the mainnet, but I don't think we have to throw away anything by that. We just need to support that in the clients, which is a nuisance, but I think a small nuisance.

Ansgar Dietrichs: Sounds good. Yeah, it seems to me like this is the main decision that we have to take on this issue in general right now, right? Like, whether to go ahead with Hoodi or to, for now, halt the fork, or try to hold the fork. everything else, then we can take async in terms of next steps and analysis of all of this. So on this issue in particular, it does seem like, so far, there's a few reasons for and a few reasons against. I don't know, I saw also in chat, for example, Barnabas as well, more skeptical of trying to halt the Hoodi fork. And Barnabas, if you want to speak up?

Barnabas Busa: Yeah, I'm just, I'm just worried that we might lose Hoodi over this.

Ansgar Dietrichs: Right, so it seems like we seem to agree that if it was easy and guaranteed to work to cancel the Hoodi fork for now, we would prefer to do this, or at least maybe for now, prefer to do it, and then we can basically continue… choose to do at a later date. There might just be concerns around how… How likely it is that we can do this without risking the health of the network? Do we have opinions on this? I don't know, Lukasz, you still have your hand up? I'm not sure if… maybe… Otherwise, 21st.

Toni Wahrstätter: Yeah, I just wanted to say that When we say, we wait for Hoodi, and we want to find out if spec changes are needed, the sounds… more trivial than it is, because if we actually want to do any spec changes, it's not like that we can restore those invariants that we are breaking with, Glamsterdam. For example, gas left will never go up. It's not so easy to restore them. So, when we say spec changes here, this is, like, we're talking about quite deep changes, I would say.

Ansgar Dietrichs: Right, I guess the argument would be more precautionary. We could always then still say next week, decide to go ahead with Hoodi, like, next ACDC or something, we could decide to go ahead with Hoodi after all. But basically, it's easier to delay a bit and still do it, rather than to do it now and then regret it, so…

Toni Wahrstätter: Yeah, I agree with that. It's more like, if we say we won't go ahead with Hoodi, then this could mean, like. Hoodi and Sepolia won't happen that far… that soon, because it's not so trivial, but I very much agree.

Ansgar Dietrichs: Maria?

Maria Silva: Yeah, I think, to me, we really need to figure out this question on the spec changes. Yeah, as Toni was saying, I think in some cases, it's not even… Like, I don't even know how we would maintain the old, invariant, so I think it's really important for us to just clarify If we are even in a world where we want to do this. And for that, I feel like, To me, it seems like the analysis would be if we find, like, a non-upgradable contract that is large enough That would warrant, this change. And so, I don't know, Hubert, if you can talk a bit to that, Have… you said that your analysis was not comprehensive, but was it comprehensive at least on the… be contracts.

Or… and if not, like, how much time Would we need to have a more comprehensive analysis, at least on those high-risk ones, so that we know that we won't need to have any spec changes, and it's just a matter of delaying the fork if the apps need more time?

Hubert Ritzdorf: Yeah, thanks. So the… so we tried to, I guess, search through… we didn't search specifically through the high-value, contracts. We searched through… we kind of… used regular expressions through Sourcify, and we sampled random mainnet transactions. The, we think that, raising awareness will, of course, also help, because these high-value projects can realize by themselves that they are vulnerable. The… Sorry, just one quick thing before I answer your question, Di. And what I also wrote into chat, we have the problem that we let some of them, or we reached out to some of them, but they are not responding to us because, for example, we are submitting through… we're letting them know through some bug bounty project or some security email list where

simply, they're getting too many, reports, and they have kind of apparently stopped responding. Yeah, so… for some of them, we have reached out, but we haven't gotten a response, so we… we have a hard time estimating how long this kind of fixing cycle would take. in terms of, like, how much time do we need to cover, or would someone need to cover the big projects? I think, at least 2-3 weeks. But so far, we have focused on mainnet alone, and I guess the question is, do we care about, like, big projects deployed on L2s? Do we care about, like, what exactly do we call a big project? And then, that would, of course, change the answer.

Maria Silva: Essentially… essentially, this tells me that we are still not sure if there could be… Issues on large contracts that cannot be upgraded. So that could mean that, we may still need spec changes.

Hubert Ritzdorf: Yeah, so among the ones that we have found, many of them are upgradable, and the… Others, they might be able to migrate, but this is… Honestly, this is a bit hard for us to estimate, because we don't know all of the big DeFi projects in full detail, and we don't know how all their off-chain components tie into it and everything. So, I think, honestly, this will be for the projects themselves to judge how well they can, How well they can deal with this.

Ansgar Dietrichs: Right, so then, I do want to come back for now, because, I mean, I would hope, by the way, like, Hubert, like, I assume you all will be very actively involved in this analysis work then offline, but I don't think… I just think we don't have enough… Of an… of a clear understanding to… today, to really talk right now on the call about also, for example, already, like, specific mitigation steps and everything, so I think all of this will have to wait for a long… for more proper analysis. So I think for now, really, the main question that we can answer today, or need to answer today, is the fate of Hoodi. Everything else, then, is to be decided at a later date.

So I really… I'm trying to understand how can we… how can we just… it seems like if we can stop Hoodi safely, without any major impact on the health of the network, we would want to. Do we have any kind of updated opinions on how likely that is, or is there anything we could do to de-risk that? Can we maybe first reach out to node runners, understand whether they would still be able to roll back? Is there anything, basically, we can do to de-risk stopping the Hoodi fork without blowing up the network? Oh, basically, let me ask it this way. Is there any stoppers from making the decision now to… to halt, to basically, to abort the Hoodi fork for now? I would say this is the default decision. It seems like everyone would prefer to do this. So basically, is there any stoppers, anyone that still has major concerns that this would blow up the network?

Any reason not to stop the Hoodi fork? Toni, I think, I mean, actually, this is a good question in chat, how long do we want to delay? My default suggestion would have been to just… Stop the fork for now and not set a new date until we understand better. what the considerations here are. But of course, if someone has a reason why we should already set a specific new date, then feel free to propose that.

Parithosh Jayanthi: Maybe to Oisin's comment on, on chat, is there, any reason anyone would not be able to revert an update?

Oisín Kyne: I say not, I was more wondering if they had… time locks, but, like, I assume no one sets a particularly long time lock on testnets, so they're okay. But I don't have that for confirmed.

Parithosh Jayanthi: Okay, perfect.

Ansgar Dietrichs: And yes, also just to, because I see, from ethrex in chat, the, the comment. that this will most likely delay the main network date into 2027. I think this is correct, but I think the overall sentiment is, and I think correctly is, that at this point at least, it's not at all clear that this is… That the impact here is small enough or contained enough that it would not be worth delaying, and so the responsible thing to do is to… to take that cost, accept that cost, and so, yes, by… by delaying Hoodi, I think chances… this makes the chance of a… a mainnet fork before the end of the year, very unlikely, but that is a cost that, is necessary in this case.

Okay, so then, going once, going twice, if anyone still has any specific blockers, any reason for why we should not make the decision to delay Hoodi speak now, otherwise we would just lock that decision in. Of course, can always reconsider async if something comes up, but…

Barnabas Busa: I think it would be wise for us to make a follow-up blog post to your blog post. I should say… Decided to reschedule, basically, the fork. I would actually ask, client teams to… Push out a new release as well, just so if… Someone is running, like, latest. Then they would get the unscheduled Hoodi releases.

Dustin Brody: One point there, as far as Nimbus, as usual, we vendor Hoodi directly. Must, must, must go into Hoodi. The Ethias one.

Ansgar Dietrichs: So, can… can you say this again?

Dustin Brody: Nimbus, I think, possibly uniquely among clients, I'm not certain, but regardless, if some color pie does it. applies to them, too. We do not have a separate configuration thing where we set when Hoodi goes, etc, or any of the other test customers. We directly vendor each client's Hoodi Right, so… we can do a release, but for that release to happen, must, must, must go into eth-clients/hoodi.

Barnabas Busa: Yeah, we can get that done today.

Parithosh Jayanthi: Yeah, we'll… we'll have the PRID.

Dustin Brody: Oh, okay, great, that's all I wanted to.

Ansgar Dietrichs: Okay, so then let's… it sounds like let's lock in the decision.

Oisín Kyne: Harry, one second, Dima Enco I've got in touch with, and I think would like to hop on and say something.

Ansgar Dietrichs: Yep, sounds good. We could also, by the way, if that is valuable, we could also, if anyone else basically is trying to get hold of anyone, we could also check back in on this decision towards the end of the call or something, just in case anyone else joined, but for now, we can also briefly wait. In the meantime, if we… assuming for now at least, assuming for now we do go with the decision, is there anything else logistically? We already said the blog post, the client releases, is there anything else logistically we should discuss right now, in terms of how to then go ahead with it with the aborted hard fork?

Saulius Grigaitis: Just one idea, maybe if we set the date, the new date, for the Hoodi right now, and include in those releases, then… We… We will save some time, because if we just now postpone for an unknown period. We will have releases now for that, and then we will need another release, for the date, for the new hard fork date. I mean, we may just guess that, Hoodi on, the glass, or on Hoodi… the new date will be, like, in a month or something, and if we can… Decide on that quickly. We will save some… Sometime.

Ansgar Dietrichs: Yeah, I see some skepticism about this in chat from Barnabas, and some thumbs up. Yeah, Dima, yeah, so background is that there's some, security concerns that, that, recently, came up. There is a… a chain security, report about, some impacts of the, of the, of the, mostly the repricings, but, but, but, but the general, the, the repricing EIPs. some contracts that are impacted, and so it seems like the current… the previous analysis, has just not been full coverage enough, so that there… at a minimum now, needs to be a bit more time to understand which projects on the mainnet are impacted, to, to give them time to upgrade, and potentially, if the impacts are very severe, and if there's anything we can do about it, potentially make some small spec adjustments to Glamsterdam.

In either case, given that, it's not at all even clear whether the Kant PlumSum specs will be the final specs, but also specifically because it could be valuable to have testnets, both with the pre-Glamsterdam and the post-Glamsterdam specifications. There was the desire to delay the upcoming, the Hoodi hard fork next week. And so then the question was… one question was whether, from the Lido side. there might be concerns around, like, obviously now very short notice, cancelling the Hoodi hard fork, or whether that would be possible for you all. I'm not sure if you can speak up on this.

Dima Gusakov: Yeah, I definitely can. So, it depends on what things you are about to change in the specs. So, first of all, we have already upgraded the protocol, like, 2 hours… not 2 hours ago, but yesterday. If the expected change is only within repricings. Except for the fact that you will make prices even higher, and then it will blow up something on our side, we are good. If we are talking other changes. Then I need to know what… what changes we're talking about, and then we will need to figure out, like, how to roll back some stuff. In general, the upgrade for Lido was mostly off-chain. Like, I believe most of the on-chain parts can be rolled back if needed.

So, in general, I don't see, like, at the moment, a significant concern regarding delaying hard fork, other than the fact that we already have the date set within the contracts, like, for all of the contracts that are interacting with Consensus layer, we have this… Jesus Christ, pivot slot, for the hard fork, but again, it's doable, we can roll those contracts back, no big issue. So, it heavily depends on what you guys are planning to change in the spec.

Ansgar Dietrichs: Right, so just for clarification, there would be no alternative Glamsterdam spec anytime soon, so it's just about basically just staying on the Fusaka spec, for now. Basically, just not doing the Glamsterdam hard fork on Hoodi at all.

Dima Gusakov: For how long? For an unknown period of time.

Ansgar Dietrichs: Yes, unknown period for now.

Dima Gusakov: Okay, well, yeah, I guess we can live with it. We will need to do a rollback of all of the changes we made, but not the end of the world. Definitely doable tomorrow.

Ansgar Dietrichs: Potuz?

Potuz: Yeah, the question for Dima, more than how is Lido affected, is how much stake Lido has. Can we count on you to, like, revert to your clients so that we achieve finality? In case a bunch of people show up with the clients already forking into…

Dima Gusakov: Yeah, that's a good question. Well, I wish we were a single operator, and I could tell you that, okay, I'll ask DevOps to roll out clients. The thing is that the setup for Lido is the same as on mainnet, so it's 30-something, big operators and then small operators in CSM that I can only ask not to update. or to roll back. So, in general, we are open to help and collaborate here, and we will definitely reach out to our operators, asking them to, Updates to whatever client version, or rollback to whatever. versions you guys will provide. I cannot guarantee that, like, 100% of Lido stake will… will be up-to-date by the time you decide… by the time the hard fork should have happened, but we will do our best on our side, definitely, to support the network and not let it fall apart.

Ansgar Dietrichs: Kev.

Kevaundray Wedderburn: Yeah, just a quick follow-up question to Dima. Do you know what the total amount of stake is? For, for Lido?

Dima Gusakov: 2.3 million, as far as I see now. So, not, like, a huge portion. But again, all of the stake that Lido operators are running outside Lido and Hoodi, I have no idea how much, but again, we have, we can reach out to, like, the majority of Big professional operators and, home stakers. But again, we can only reach out and ask them to do that. We cannot guarantee that, unfortunately. This is how protocol works.

Ansgar Dietrichs: Sounds good. So, doesn't sound like any specific, blockers, then, from your side either? Like, sounds like, you would be okay with going ahead with aborting the hard fork, then? Mmm… Is there anything else in terms of the logistics, out… around how to… how to basically handle this… this aborted hard fork that we should still discuss? Kev?

Kevaundray Wedderburn: Yeah, just to confirm, so the two next steps are for a blog post to notify the ecosystem, and for ELs to push a new release that just reverts the Glamsterdam changes. Is that correct?

Ansgar Dietrichs: That would be my understanding. But the blog post should not wait for the releases, right? The blog post should just recommend the… going back to the earlier releases, pre the Glamsterdam fork. It's just that for those, just… listening to the latest releases. They also have the latest releases. Okay, sounds good. Then, of course, these things are never great if they happen, but it seems like we then have clear understanding of the next steps here. Is there anything else we should discuss in terms of, over the next days, how the… analysis work should happen, in terms of, obviously, ideally, we get to a place where we can start to make some assessments as early as possible, including, say.

should we consider spec changes? And so on. Again, there's no… in the sense of, like, we will take whatever time it needs, but I'm wondering if… is there anything we should discuss now? We should decide now in terms of how to go about the analysis? Fredrik?

Fredrik Svantes: Yeah, so… there's been a group created where this is being discussed. There's also been an We reached out to a few different parties, some of them were mentioned in the chat already. To see their ability to do the analysis, and in what timeframe they will do it. So hopefully we would have an answer on that. By end of day… And then we have, few people… I guess, who are here, that are also looking into this from various perspectives. I believe, Francesco, Toni and Perry, and others are looking into this, from different types of angles. So I do think in the next couple of days, we will…

Have more information on when we can expect the analysis to be done, and… Yeah, that would probably be the next step here, I think. I guess by ACDT, we should be able to present, more details on this.

Ansgar Dietrichs: Sounds good. Then my understanding would be that there's no use in basically using any of the standard ACD toolkit tools, like… scheduling a breakout call or something, I feel like it's… that's not really what's called for here, it's more… this, async work that will happen anyway, and then on ACDT, we can see about next steps. Is that right? I mean, seems right to me. Yeah, Matt, so Matt asks in chat, should we discuss a bit what findings would cause us to change the spec? I'm a little skeptical about this, but I'm open to it, though. Maybe, Matt, if you want to speak up as well, maybe just to briefly say my skepticism. I think it's a somewhat nuanced topic, because, say, some of… some of the changes, for example, or some of the problems, they… they…

obviously contrived, simple, simplistic example that was given in the presentation. I think that one was a specific instance of something that is just a general consequence of any repricings, or can at least be a consequence of any repricing, so basically. that is very hard to change, but there's also some specific Glamsterdam behavior around… there was, I think, in the blog post, around the refunds, and also around the reservoir construction, the state gas reservoir. it feels like it's a very nuanced question that depends a lot on, like, what we actually see in terms of real world, which types of patterns actually exist. I'm not sure that it's… it's going to be that easy to have a high signal discussion about this right now, but Matt, if you maybe want to briefly at least talk about it?

Seems reasonable.

Lightclient (Matt Garnett): Well, thanks for the intro to the discussion. I mean, I sent a message in the chat a little bit earlier. That sort of expresses some of my views, but… I feel like I'm getting the sense from the way the conversation's unfolding a bit, is that… We are trying to almost prove that some of these, like, you know, initial findings that we're having, we're trying to, like, prove that this is somehow an invariant that we told developers could break. And it's now breaking, and so therefore, we're, like, trying to maybe assign fault towards those developers. And, to me, that feels a bit of a wrong way to approach this. Like, I think that we have done a fair bit of education around

you know, gas prices will change, or gas costs will change, don't hard code your gas costs. But I think, like, the examples we're seeing are… A lot more complicated than someone who's hard-coded the gas cost into the call opcode. And I think that, at the end of the day, if a sufficient number of users, or a sufficient amount of funds on mainnet are making use of a functionality. That we can't just… Say we have to, you know, Break that functionality. Obviously, there's… nuance to these arguments always, but if we find that there's a lot of contracts who are using this, and we find there's a lot of funds that are locked into this. I mean, I personally don't think we should break the user space, for those reasons. If we do decide to break the user space, you know, some of these things are upgradable, then we have the question of.

When can we do this main… this upgrade to mainnet? Would we give protocols a period of time where they need to go and upgrade their contracts and go and audit those upgraded contracts? what's the forcing function there? Would we be willing to do mainnet even if some protocols haven't upgraded because, you know, we've decided they're dragging their feet? Like, I think we start getting into, like, extremely messy situations there. So I'm curious if there's arguments against this, what other people are thinking in terms of breaking user contracts.

Ansgar Dietrichs: Yeah, do we have anyone else with, thoughts on this? Again, today will only be the start of that conversation, but… Reasonable to have that start now.

Hubert Ritzdorf: Yeah, maybe really quickly. So, as I said, we somewhat had a discussion before with a previous hard fork when I was involved in it, but, like, the… I think, on the one hand, it's kind of what Matt said, and he said of, you have potentially immutable contracts on a somewhat mutable execution layer, and, how are people supposed to do this? On the other hand, I also agree that we kind of… Certain changes will be necessary, and, the… kind of… they might not be able to wait forever. So, yeah, I think… Hence, it will be very hard to… kind of find a balance here, find a balance here up front. I think we will have to see what the cases…

Ansgar Dietrichs: I see several contributions on chat, anyone else that wants to speak up on this? Toni?

Toni Wahrstätter: Yeah, I said the same in the chat, I think… Of course, like, if very fundamental functionalities or, big contracts, big user… big contracts with, big, a lot of users would break or anything similar to that. We would, of course, change the spec and avoid that, and delay the hard fork. But at this point, it's just too early, because we don't really know what is the actual impact on those users. We already know that Some invariants will be broken, and we have to check, like, how fundamental are those invariants, which is something we don't yet understand very well. And… Then we can discuss, how to… how to approach it in case we actually find out we want to change the specs.

Because it's not so clear that We can even just add those invariants into the protocol again without undoing, like, big parts of what made those EIPs worth it? So, yeah, I think it's too early to speculate about potential things that we know better next week or in two weeks.

Ansgar Dietrichs: Anyone else with, thoughts on this? But yeah, otherwise, I'm… maybe also then, kind of, if there's no clear, other, hands here raised, then maybe good stopping point on this for now. But I think I agree with both Matt and Toni, right? Decision one is, will there be spec changes? I didn't want to steer the conversation towards the answer no, this might very well be a yes. So A, will there be spec changes? And then B, Of course, dependent on the first answer, when will mainnet roll out. One thing that is, I think, good to keep in mind is that, in terms of work on, Hegota, which will be the next topic here on the call as well, for the remaining part of the call.

As long as there are no major spec changes, which, of course, again, they might still come down the road here, but as long as there are no major spec changes, then, work on Hegota in principle can happen in parallel. So even if we end up delaying the hard fork for quite a bit, it's not obvious that that would necessarily have so much follow-on effects in terms of… Delays of later hard forks. Okay, do we have any… anyone with any closing comments still on this general topic, the general topic here of, anything really related to Glamsterdam? Both the security situation more specifically, or Hoodi, or anything else really related to Glamsterdam?

Dima Gusakov: Like, just to confirm, so the decision is that you guys are canceling Hard Fork on Hoodi, as well as on mainnet for an indefinite period until the further notice?

Ansgar Dietrichs: Yes, so mainnet was never scheduled yet, that was supposed to happen next week.

Dima Gusakov: That's true.

Ansgar Dietrichs: So for now, mainnet is just implicitly halted, but yes, Hoodi specifically is canceled, hereby. Okay.

Dima Gusakov: Okay, cool. I'll take actions on Lido's side, thank you.

Ansgar Dietrichs: Perfect. Yes, sorry, that's… yeah, so… very explicitly, Hoodi, for now, is canceled. And there's no… no new date for Hoodi set yet, this will be decided at a later time. So for now, it's just… Hoodi is canceled, and… Yep. Csaba?

Csaba Kiraly: Yeah, I was joining late, and I heard that the… there's supposed to be a recommendation to the clients? Is that really needed?

Ansgar Dietrichs: I think the intention… oh, sorry, go ahead.

Csaba Kiraly: Yeah, no, I just wanted to say, to me, it seems, at least for those clients who can just continue, it's much… There's confusion if there's no such recommendation going out.

Ansgar Dietrichs: Yeah, do we have thoughts on this? I mean, I think we just have to make a decision. I think the original intention was just for timeliness reasons, so we can basically have the blog post out ASAP, instead of having to wait and coordinate. Potuz?

Potuz: We've already triggered the release, we have a bunch of security patches, and I believe other clients also have security patches that they would want to have users, at least on mainnet, using. On the other hand, I believe that we can act right now and have the blog post as soon as now, rolling back the Hoodi fork and instructing users that have already updated to either roll back or wait for their clients to have a new release, but just to have this in mind. I mean, this… the notice of this should be going out right now.

Ansgar Dietrichs: Right, so would it make sense in the blog post to basically have a list of, here is the clients that already have a new release, we keep adding this and adding them as they come, but basically say, if your client does not yet have a new release, basically your recommendation is to roll back. Is that a good pragmatic way of handling this? Okay, sounds good. So then, that's how we will do it. We'll go ahead, blog post, and we will keep the blog post up to date with any of the new client releases. Okay, sounds good. Then, any, I think, ongoing discussion can still happen in chat on this. I do think it still makes sense to use the remaining 25 minutes for the Hegota section of the call.

I know it's a pretty hard cut now, context-wise, but I think it's still valuable to keep having that work also go on in parallel. So, on that note, just going down the agenda there, the first point that we had, Well, actually, let me briefly pull forward the miscellaneous, the one-point miscellaneous point, because I think, actually, we might need this next week. Barnabas, you wanted to propose extending ACDT time to 90 minutes. I feel like, especially now, it might actually be quite valuable, so let's do that first. Do you want to briefly speak up on this?

Barnabas Busa: Yeah, I mean, you described the question. Whether we want 90 minutes or 60 minutes. I think, we've been running out of time in ACDT quite a lot in the 60 minutes, so I think it would be time to officially Preserve 90 minutes for all future ACD calls.

Ansgar Dietrichs: Seems reasonable to me. Anyone strongly objecting? Speak now. Okay, so 90 minutes it is. Sounds, sounds good. Thank you on that. And then, Barnabas, you're also up right again. The first item on Hegota was, I think you wanted to just briefly mention the dynamic base fee calculation adjustment for shorter slots, Just for awareness, do you want to briefly say a few words on this?

Barnabas Busa: Yeah, most of the clients already took a look at this, PR, so it's basically using, Derived base fee calculations of a hard-coded version. So this way, we can test, different kind of, slot, seconds per slot. values without needing to always change the ER code for, A fixed, fraction. as most of the clients already approved this, I'm gonna… work on trying to get this merged into the EIP. Yeah, that's about it. Just want to raise some more awareness for clients that have not taken a look at it yet.

Ansgar Dietrichs: Sounds good. So, anyone who has not yet taken a look, or not enough of a look, yeah, make sure you're up to date. Okay, then, next… Item, that one is specifically… there was… there's always the… there's been a desire, to have… a rough, agreed-upon priority ranking of EIPs for the, kind of, upcoming hard forks, so that the testing team can prioritize which ones to assess in more detail first, and then also that there's… Ahead of time, a rough understanding, a rough shared understanding of in which order they will go into devnets. The idea is not for this to be literally binding, so that, like, you know, you can never include, say, something a bit further down the list before you include the topmost one, but to at least basically, like, have a shared basis, or expectation basis to operate under.

M… Nixo took the time to put together a, kind of, a first proposed version. I wanted to go over this as well, but then the Glamsterdam topic took up a lot of time today in preparation, so just was wondering, whether We want to… Yeah, I mean, A, in general, if people disagree, that having some sort of… again, this can be a living document where we can, over time. roughly reshuffle the ordering a bit, as, for example, some EIPs end up looking a bit heavier than they initially were, or something like this, but, just, just to basically, like, use maybe just this, the linked list as a draft, as a V1, and then basically, like, async, iterate on it, and, and figure out, for example, what is a good format to keep this list in, like, where should it live?

all of this, I think, again, given that it's not binding in any direct sense, I don't think it needs super strict governance here, so I just wanted to basically. have some sort of shared agreement that for now, it's also linked in the agenda that we are for now just using this list, and then if we want to adjust the ordering, if people have any… quibbles with it, we can iterate on this, but yeah. So… Just going once, going twice, basically, anyone who wants to protest this approach, or… again, nothing locked in forever, but… And I agree with Spencer's comment as well in chat, that I think this is one of those areas where, like, going into the next, next scoping process for ISTAR, I think, ideally, we can make this even more, kind of like.

kind of directly, natively part of the scoping process to have EIPs in an ordered way.

Lightclient (Matt Garnett): I think it would be good to talk more about how we would want to approach the next scoping. Because I feel like a similar related issue with this is that I think Right now, when we go… when we have an EIP, which is PFI, the default is kind of to reject. But once the EIP is CFI, It feels like some people believe the default is to ship it in the fork. And I think that if the default is to ship in the fork and that's something that we want to agree on as ACD, then I would say we should be approaching the PFI to CFI discussion a bit differently. Because I think sometimes people will make the argument in the PFI discussion when it's contentious and say, well, it's contentious, let's just continue considering it by moving it to CFI.

But then there's this implicit assumption that CFI is kind of something we're just gonna ship. So, we should really, like, discuss, like, what the difference between those, you know, two assumptions are. Like, I'm okay with saying that CFI is default ship. But if it is, then let's… let's, you know, explicitly say it, and then in the discussions for going from PFI to CFI, let's discuss it with that assumption that this is something we want Ethereum to do, and we're going to have to prove that it's not really feasible in that fork once we get to CFI, if we're not… if we end up not wanting to do it.

Ansgar Dietrichs: Yeah, I mean, given that this directly affects Hegota still, because we obviously still are at the CFI to SFI stage for many, many EIPs, I think it's reasonable to discuss this briefly now. Justin, do you have thoughts?

Justin Florentine: Yeah, so I'm gonna agree with lightclient here, possibly violently, and I don't think that we should actually make any of the changes, to, to default CFI into shipping it, because the whole point of CFI is saying that, yes, we are considering it, but moving it from CFI to SFI is where we do our labor assessment. Like, that's where we say, like, yes, we want to ship it, but we can't this fork, sorry. So, like, we can't lose that step in the process. So, I disagree with the premise that CFI is a default ship, because we're not doing any actual, like. labor assessment, we're just saying that we like it. So, wow, that's a lot of hands. I'm gonna step down. Thank you for, letting me come to your TED Talk.

Ansgar Dietrichs: Lukasz?

Łukasz Rozmej: Yeah, I agree that CFI is not, cheap, definitely, because we are… we, even during this scoping, had some EIPs that we CFI'd, but a lot of people had some kind of, asterisks there that we can consider this shipping if, and there were some, some, conditionals. So, if those conditionals can't be met, for example, because they are impractical, or just can't be done in some way, we default to DFI there. So, yeah, CFI, this is definitely that we will put an effort to into exploring this. And we'll see where it gets us. I don't think it should be considered the default ship.

Ansgar Dietrichs: Mario?

Mario Vega: Yeah, I think it's… it's a little bit naive that to think that CFI in something does not affect our perception of it. Because, I mean, we are putting work on it. The specs didn't put work on it, the testing, all that involves just putting something into CFI. It's already, like, the… we're getting into the fallacy of, like, we are… we are sinking time into it, we are going to try to ship it, so I think it's naive to think that it doesn't affect our perception. So, I would… I would… I'm not suggesting that CFI should be SFI at, like, immediately, but we should be more cautious about CFI for the future. I think CFI is a big decision still.

So we should always consider that, before, before just accepting anything into the fork, in my opinion.

Ansgar Dietrichs: Lukasz, you still have your hand up? Otherwise…

Łukasz Rozmej: No, sorry, I will have it.

Potuz: Yeah, so, probably I brought this discussion a little bit before on R&D. I agree with everything that, that, Justin and Matt said, but I want to stress something, that even if we were… in a world in which CFI meant, by default, SFI, or by default inclusion, which I consider silly. The reason why we had CFI was to be able to test something that we aren't sure if it's going to be there, so it's not by default to include. But anyways, even if we move to such a system, we would be forced to have an extra layer to attend for those EIPs that we want to make a decision, that we want to test. We will always be in a situation in which we want to test things before making any kind of decision, not only for inclusion, but even to just have a prototype. So we always will have

EIPs that we want to be… some clients working, because they want those EIPs in, or because they know someone that is involved with that EIP, but clients are not forced to implement. So there are EIPs that are in that situation, most EIPs are in that situation, in which I am not forced to code it. So my complaint is not about the governance process itself, my complaint is really about the spec. Which is, we should not be including things that Clients are not forced to be coding that EIP writers are not forced to be adjusting to in the main specs, either in the consensus or the execution specs.

Ansgar Dietrichs: Yeah, I think, to me, this makes a lot of sense, just overall. We should always try to have… I mean, obviously, this is never a black and white situation, but I think the more we have a shared understanding of what these labels in practice imply, and mean, I think that's healthy for the process overall. I'm wondering, is there any… anything concretely actionable, specific to the Hegota scope that anyone would want to propose here, in terms of adjustment to the process? adjustment to how we treat the CFI, I think on the EL side, we have currently 19. Now, one of them is informational only, one of them is, I think, mempool only, so… so it's maybe not quite 19, but it's close to… close to that level. That's a lot. I would assume not all of them end up in the hard fork. Is there any desire to make any adjustments to how we go from…

from CFI to SFI, again, we will have this priority list, we can iterate on this over time, so maybe that's a good helper tool, but yeah, anything else in terms of Hegota specifically that someone would want to propose? Matt.

Lightclient (Matt Garnett): I guess, maybe not specifically Hegota, but again, like, kind of related. I… I feel like if… we're not having this default ship assumption with CFI to SFI. I think it's quite difficult to have a… real argument on ACD about whether or not we should you know, be doing something to Ethereum. And I feel like this stems from… at least the way that I believe that we should approach changes to Ethereum, which is that we should be doing the absolute minimal changes to the protocol to achieve what we want, and to make Ethereum achieve what we want. And I think that, in general, you know, if we're gonna, say, default ship the CFI EIPs, then it becomes challenging to have that conversation, because the sunk cost is already there. It's pretty difficult to say, well, we already tested, we already put it in the devnet, it's already kind of working. Does it hurt to ship it?

But I think that if we, like, change the way we think about changes to Ethereum, I think every change to Ethereum is risk. which is basically what we're seeing with, like, you know, 8037 and 8038, and, like, many other things. Like, anytime we change it, like, we're risking breaking user contracts, we're risking having security incidents. And so we should do the absolute minimal number of changes. I don't think that we should just change the protocol to change the protocol. Like, every change we do should… be in pursuit of what we believe Ethereum should be doing, or, like, should be providing to its users. So that's kind of why I think, you know, for one, it sounds like we need to increase the bar from PFI to CFI, which is something we really can't do for Hegota.

So I think… you know, what can we do now that we're in this situation in the middle of scoping for Hegota? Maybe what we can do is, like. think more seriously about the changes that were going to SFI? Are these things that Ethereum should be doing? Not, have we implemented them, have they tested them, are they… can we just ship it? Like, does it affect people? But, like, is this something that's actually good?

Ansgar Dietrichs: Yeah, and I guess, again, just to bring it back to Hegota, I think… I think, ultimately, any of this works process-wise, because these labels, as long as there's shared understanding, whether we have that discussion, that mindset, that question of what should Ethereum even ship, whether we have this at the CFI stage, or the SFI stage, or both, or anything in between. any of this works, it's just important that we have shared understanding, because otherwise things can just slip through, and we end up shipping something, and we realize we never actually had that discussion, because these labels were ambiguous, and we never knew when the right moment was to have that discussion, so I agree with you there, and so I think given that we are in the middle of a hard fork scoping process right now, just something we should keep in mind regarding the already CFI EIPs, if there's any where we feel like, actually, do we even want to ship them? Of course, that's something we should make sure we

Discussed before… before it is too late, basically. Mmm… If it's okay with you, Matt, I would move on for now. We won't get through all of the agenda anyway, but still a few items. In particular, I think one decision, at least, that I would like to still do today is, that's next up on the agenda is, and it's related, it's the next devnet for Hegota, and in particular, how we want to approach this. My understanding is that there's two alternative approaches that have been discussed. One of them would be to keep the CL and EL devnets separate, for now, and that would mean having a… a Bogota devnet 2, so an EL-side-only devnet 2, to follow the devnet 1,

and basically, like, have a bit more time for EL side iteration, to add some more EIPs into it, and so on. Or, the other approach would be to immediately now go to Hegota devnet 0, so the first ELCL cross-layer. devnet. I saw some comments in both directions, well, there was one comment in both directions on the, in either direction. on the agenda, I also think from the testing team, there was some desire towards, at least reasonably soon, going towards a shared cross-layer devnet. Do we have anyone from the testing team that could briefly speak on this? Mario?

Mario Vega: Yeah, what I can say, I mean, I'm not opposed to, like, merging right now. The only reason why I see merging as valuable is that FOCIL and frames do have interaction, so that's… the reason that I would suggest, like, getting them together as soon as possible, and they are both SFI, so I guess it makes sense. just from a testing perspective. If there's a cost in having them joined from the beginning, maybe that's worth discussing, but the value is real in terms of testing, yeah.

Ansgar Dietrichs: Jihoon?

Jihoon Song: Yeah, I just wanna share that. The interaction between FOCIL and frames are not ready yet. So if you want to target devnet 0 sometime in next week, I think it's too early to test FOCIL and Frames, but if you want to launch it without the interaction between FOCIL and Frames, then just FOCIL I think it's doable. So… Yeah.

Ansgar Dietrichs: Right, I think, Marc, I think you said, that on the frames breakout, there was some desire to stay… to basically have another EL-only devnet first, is that… is that right?

Marc Garreau: Yes, but there was some discussion since then. I think there was maybe some more sentiment expressed about a combined devnet since that conversation took place. There was… during the frames breakout call, there were a few spec changes discussed, and the possibility of adding multiple new EIPs to an EL-only devnet? But yeah, that conversation proceeded. The, the combined one?

Ansgar Dietrichs: Sounds good. Do we have people? I mean, I see in chat also discussion of, like, how realistic it would be to, say, have a combined devnet next week already. But just so I understand, Matt, maybe from, also from the Geth side, would a separate next EL only, like, if we had just a Bogota devnet 2, would that be ready? Could we launch that next week, basically?

Lightclient (Matt Garnett): Yeah, that was kind of my working assumption. I feel like the changes… there's… There's about 3 main changes I'm imagining for the Bogota devnet 2. One of which we haven't merged yet and are kind of discussing a little, but I would like to merge, and we generally felt on the breakout room that, optimistically, we should merge it unless… On this call, people felt it was… Not a good idea. But I think, yeah, I feel pretty confident we'll have 2 ready, which we could ship next week. And I'm just thinking people will start traveling for Devcon, so… If we are not shipping a devnet this week, it feels like we might slip a couple weeks. Due to Devcon-related trouble.

So… But, like, I agree with Mario, like, it would be nice in Devcon to start talking more about the interactions, and so maybe that's a better outcome, even if we don't have the… Even if we miss a devnet deadline later next week. people have spent time thinking about the interaction. Like, honestly, I can see the benefit of both.

Barnabas Busa: Would you consider then doing Bogota spec? But with the combined? Or would you then just fall back to the previous EF scope?

Lightclient (Matt Garnett): I would rather do the previous devnets go, personally. But… We could do it all, I guess. I don't know, I'm pretty ambivalent on this. I think others should weigh in. I can go either way.

Ansgar Dietrichs: So, maybe a question would be for teams specifically, like, how much… because I know not everyone, of course, will be at Devcon, but there will be some kind of core dev prisons at Devcon as well, some in-person discussions and everything. How much would spending the next week… looking into the interactions specifically help prepare for this. Basically, if… because I feel like otherwise, if we basically couldn't choose between no devnet at all, or another EL-specific devnet, I think I would, in principle, pick the EL-specific devnet, but if the time spent on The interactions, is this valuable in preparation? that… that would override it. So, would anyone… is there any EL team that would be like, hey, yeah, actually, we would… we would prefer to focus on…

Working towards a merge devnet, even if we can't ship it pre-devnet, pre-Devcon? Sorry, I feel like there was a… too complicated a question. Anyway, yeah, I don't… I don't quite know which way to go here, because, just because, like, it feels like there's significant, like, a lot of people speaking up in either direction. Can we maybe… I would say the simple… the simple thing to do would just be to say, like, if we can ship a merged devnet before Devcon, we should do it, and if not, we should just go with one more round of separate devnets, just because… So we have something shipped pre-Devcon. M… Does that make sense? And so if so, like, how realistic is it to ship a merged devnet next week? Because I think then, to me at least, that would be the decision factor.

Lightclient (Matt Garnett): Yeah, I guess maybe, Mario, from your perspective? But what would it look like to have them merged?

Mario Vega: Yeah, sorry.

Lightclient (Matt Garnett): release.

Mario Vega: Yeah, so I think if we already have Bogota ready, I think that's… that's test… that's testing in itself, so I think it's valuable. We should… we should go ahead with it. But I think we should start looking into the interaction. maybe, as you said in Dev… or somebody said in Devcon, we should, like, start really thinking about these interactions, and then we start then planning the… the… the joint devnet.

Ansgar Dietrichs: Okay, so I think then that's just also because we are about at time, so I think unless anyone would protest, I would say then that decision would be to say, next week we will launch Bogota devnet 2, so, like, another separate devnet. If for some reason this is delayed, until after Devcon as well, we can reconsider whether it's worth even having it, or immediately going to the Merge devnet, but point being is the understanding would be At Devcon, for anyone in person, we would already discuss interactions, and then the first devnet after Devcon will be a merged devnet. Does that work for everyone.

Lightclient (Matt Garnett): Yep.

Ansgar Dietrichs: Okay. Sounds good. Sounds good. Then, everything else on the agenda, we will basically postpone until the first ACD after Devcon,

Lightclient (Matt Garnett): One point that's kind of related to the Bogota devnet 2. We talked on the frames breakout, people said they would review this PR a bit more. we were kind of optimistically looking at including in Bogota devnet 2. I addressed… One of the comments… from Daniel regarding an issue we found, but overall, I think that this PR is ready to go. Onto the devnet, so… if anyone who's had a chance to look at it in the past few days. wants to push back on including it, I will probably merge it after this call. Last chance?

Okay.

Ansgar Dietrichs: And then there is the question chat, I mean, we are at time, but there's the question chat whether we should, at least for those that can stay for a few more minutes, discuss briefly, like, the scope for that devnet, the EL side, like the Bogota devnet 2. Does that make sense? And if so, does anyone have thoughts on… On this? I'm unfortunately a bit out of the loop, we obviously have this, again, this protein… list, if that is help… if they can be helpful. M… Yeah, anyone with more context, Spencer?

Spencer Taylor-Brown: Hey, yeah, I think, at least this is my feeling on kind of what we'd want to add. I think, there was discussion about adding the two floor EIPs as well, so the Unified Transaction Context Floor, the Block Access byte floor. But then I think adding 6 EIPs is gonna be way too much. so yeah, the… I guess the… there are two of the, I guess, PQ-related EIPs Here, they deactivate self-destruct so that we can remove tests, and then the bump nonce is… Something at least we'd liken earlier, just so we can, test the transition. Of the fork for this EIP as many times as possible. But yeah, maybe… maybe there's preference to add… add less or add other… other EIPs here as well.

Ben Adams: Hey, how's it going?

Ansgar Dietrichs: Hello, Ben, we can hear you. Maybe we should not hear you. Justin?

Justin Florentine: I just want to bring up that I think if the intent here is to, you know, study interaction between FOCIL and frames, we probably shouldn't add 3 more EIPs to it.

Ansgar Dietrichs: Right. I mean… Yeah, that's the trade-off, of course, with going with another devnet. Do we… is this a decision, by the way? Should we… should we finalize the scope today? Like, is this valuable?

Lightclient (Matt Garnett): Yeah.

Ansgar Dietrichs: What can we take?

Barnabas Busa: Yeah. Okay. Because if we don't finalize it today, it's gonna be way too late to launch anything next week. So we need to either, figure out whether we want to… you know, keep increasing the scope of Bogota devnet. Or we want to focus on the FOCIL merge.

Ansgar Dietrichs: Sounds good. So then, with the list that Spencer put in chat, like, these four EIPs. what is… like, what's people's feeling? Is this too la- too large, or too, like, basically too long, too short, or just right?

Justin Florentine: I mean, basically, we'd consider that too much, because we're ahead on FOCIL and behind on frames, so this makes our job a lot harder.

Ansgar Dietrichs: Which one would you kick out, if you could kick out one?

Justin Florentine: I'd kick them all out. Thank you. I would bring in SETCODEFROM, though.

Ansgar Dietrichs: Okay, so you'd only kick out the other 3.

Spencer Taylor-Brown: I guess I'd add that, like, personally, I feel like combining the FOCIL frames has more value, and it actually feels like it might be easier than adding more EIPs, but if we're not going to combine then it makes sense to do… to add EIPs to the devnet, too. M.

Barnabas Busa: Yeah, otherwise it's just status quo, I guess, which we already have.

Ansgar Dietrichs: Right, there's also the small spec iteration on frames, for example, that Matt just now talked about. But yeah, I mean, it seems like SETCODEFROM, it was mentioned a lot, so that one seems to be clearly, so then I think maybe let's just briefly discuss the other three from Spencer's list, Lukasz? Do you have a take?

Łukasz Rozmej: So, Nethermind is, quite ahead on everything, so, according to my, kind of. Call to… to speed up, we would like to include all of them.

Ansgar Dietrichs: Right, but it does sound like Besu might not be able to make that in time for a pre-Devcon, devnet launch? Is that, is that right?

Justin Florentine: Yeah, that's correct, yeah.

Barnabas Busa: We just need clients for the launch, though. So if you can get 3 clients, then we should be able to do… network.

Ansgar Dietrichs: Yeah, I mean, it works for me. Do we… would we prefer to basically go with the full 4 EIP list, and and basically launch with a smaller set of clients? Is that preferable for people, or… Just in how… what's… from the Besu side, would that be okay, or would you strongly prefer To have a smaller scope that, you all would be able to meet?

Justin Florentine: I mean, if the low watermark is 3 clients, and you've got 3 clients, then I wouldn't try to make a case that you should wait for us.

Ansgar Dietrichs: Yeah, it seems like because you're ahead of time, you said, on FOCIL, so, like, you basically make that time up there. So then I think… I think it is reasonable. Do we have 3 clients? Like, Nethermind just now said that they would, target this? Like, do we have, I guess, Geth and then maybe one more client that would be willing to commit to this set of 4 EIPs new in the devnet? I mean, first of all, Geth, Matt, I mean, obviously, I don't know, would you… do you think, from the Geth side, these four EIPs are realistic by next week?

Lightclient (Matt Garnett): I would prefer just SETCODEFROM and 8151, but I think the other two are probably simpler than those two, so… we could do it.

Ansgar Dietrichs: Right. Yeah. Oh, sorry, Ivan?

Iván | ethrex: Sorry, sorry, I didn't raise my hand. We already have them, not pushed yet, but we should be good.

Ansgar Dietrichs: Okay, sounds good. In the interest of time, yeah, I mean, I would just go with all four then. I… I don't… I mean, I think either way, of course, if Matt… Matt, if you want to make one, I don't know. Is it okay, would you disagree now, or would you rather ask the others to reduce the scope?

Lightclient (Matt Garnett): Ugh… I guess we can do it, guys.

Ansgar Dietrichs: Okay, see, that's… that's the spirit. One fork's still half broken, and we're already pushing on the other. Awesome. Okay, then let's lock in the full for EIP, set for… For, for Bogota devnet 2? That's it, we had time. Oh, question, do we… have we made a decision? In two weeks, the ACD, obviously, will be right during Devcon. My kind of implicit assumption was it would just be Canceled. Does this match other people's expectations? I guess that's something I can… principals will just decide, but… Okay, so then let's say, no ACD in 2 weeks, so ACDE will return in 4 weeks. Okay. Thank you everyone, and see some of you all in person in two weeks. Bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.