MEV: the Gas Killer - Greg Markou - [Toronto Ethereum Developers Meetup]

Transcript

um yeah hey everybody my name is greg cto at chainsafe um and i'm gonna be talking about not necessarily talking about mev specifically but talking about how mev how you need to think about mev when building applications and specifically in the bot realm and that's because we got burned two weeks ago not hard but we had some software that had some hiccups and it was an interesting learning experience going down so i'm going to talk about the gas killer um so yeah uh welcome i kind of gave my intro already cool so thanks to mark i don't have to do as big of an explanation as expecting

but really simply put a minor acceptable value you know it's really just an understanding of the measure of how much profit a miner is taking by basically rearranging what transactions they prioritize in a block uh versus the traditional way nodes typically work which is you know they take the most expensive gas transactions and order them in that way and with mev they're basically reordering them however they choose typically because they get paid out by somebody else um cool and here's a really nice graph from james prestwich shout out to him this is like four years old but he was ahead of the game uh you know if you want like a visual understanding the idea is you know

typically you know in the block a certain transaction would have come before um and like with the sandwiching that mark was talking about you can kind of see here before you know the technically you know unlock five should have happened beforehand and because you swapped the transactions the user is only unable to unlock uh the user is able to you know get ahead and do whatever they need to do this is really out of context there's a paragraph with it but it's a good blog post um hey craig can you just unshare and then reshare there's like a green square around here in the middle of your screen yeah thanks much better thanks dude yeah cool um

[Music] yeah so it's just basically the reordering of transactions and it becomes it's not really a big problem in some cases you know if you don't really care about the order of your transaction this really doesn't matter if you're just trying to get into a block um in some cases like in bridging for instance where which is our software that kind of like had this interesting situation you know we just care that it gets included in a block relatively quickly we don't care about the order and whatnot so for a lot of people that doesn't matter if you're doing you know high volume trading on like unisop river and you're going for arbitrages then yeah this is a big deal for you but what i'm going to talk about is like for the people that think it doesn't matter to them um

and why it's important that we need to kind of keep an eye on on mev in a different angle so you know gas is right right like you always as the majority of users you probably know these two screens or something similar where you get to pick the gas price right you get to pick a you get to choose the gas price and you get to basically decide how fast you want to get it included around the left there's metamask and you can kind of just set it however you please and on the right is some i don't know which website i used but one of the websites that has all has the prices that how they assume how fast a transaction will get it and this is normal you know as a user interfacing with a dapp or whatever

um with your wallet mobile wallet whatever it is you really have a lot of freedom because you get to see what it is today at that time but where it gets a little funny and where we get into a bit of a pickle so to speak is when we start looking to automated transactions and that's when we have bots running so whether that's you know the keeper bots in the maker ecosystem that are in charge of like basically bidding on the liquidations or even bots that are just looking for arbitrage opportunities between different dexes flashbots you name it they have a bit of a different problem that we have to think about that i don't think a lot of people really understand uh the issue with it especially when it involves mev and i'll give a really good example here

so basically um automate automated trends when you do an automated bot you have to actually figure out what the gas price is there's a method called eth underscore uh estimate gas um and it exists on every node it's a gath open ethereum you know turbo gas another mine they all have it and it basically takes a sample of blocks and tries to make the best assumption about what your gas price should be for a given transaction at that current time now the problem breaks down and i'll kind of i'll walk through basically how it broke down for us and how we ended up discovering it was a little bit trickier than it seemed at first basically if we look at this these are the logs from chambridge so chambridge

is our cross chain bridging tool that basically allows you to move assets kind of like mark was talking from like an l1 to an l2 in this case this was the avalanche bridge moving back and forth and we started noticing something funny so we have this concept of a nonce and you can see at the top right corner 1724 and it was attempting to basically start this transfer and you can kind of see it progressing it's like hey they created the proposal um and we've we're waiting for the finalization event so we're waiting for it to finalize on ethereum and then there's a couple other proposals that happen you can actually see at the bottom like proposal complete not waiting and you can even see there's a submitted proposal you know on the execution a little further down and then when you get to

the very last log this is where it gets interesting we start we see 1724 again but this time it says we've exceeded our limit so we only watch for so long before we assume the transaction is dropped and in this case we got to a point where it actually just skipped it because it could not find enough votes had gone through and it couldn't find the necessary like event on the contract to actually trigger so this was peculiar because six hours later all of a sudden all the transactions flooded through and the only thing that have changed was we noticed the gas price is really low on all the transactions from this one time and they actually because of the nons on the transactions they actually blocked all of the of the transactions um after it right because you have to kind

of each nonce needs to go first um and so one of the interesting things we found here was that um basically there's two things one um if you're really underpriced you're actually you're you know your node's not really going to make it to the mempool so we didn't even see these transactions on the on ether scan because some miners probably just did not even propagate our transactions because we were so drastically underpriced so the whole situation was really weird it was basically we have missing transactions we don't know what knobs we're at right now in the transactions themselves and for whatever reason we're just constantly skipping the execution now there's one thing to note every single relayer in the system

we have five in this case four of them were running geth one of them was running open ethereum the open ethereum node was working so this led us to believe something was up with the estimated gas and after doing some digging we actually found that these are the results of the what would have been the sampled the sample basically uh transactions from the previous blocks so what geth and all these nodes do is they basically take a sample size they basically say i'm going to take three transactions from the last 50 to 100 blocks and i'm going to basically use that to aggregate a rough estimate of what the gas price should be for your transaction that you for for your

transaction right now because i just made the estimate gas call it's going to do this quick check and build it what we found after doing is that um the blocks that it would have chosen to sample were actually just nev transactions so you can see there's a lot of zeros and ones this is g in g this is represented in whey so if you kind of shrink it down by a couple zeros it's 149 hundred forty nine hundred forty way and then a bunch of zero ways a couple one guys and if you kind of do the mental math here you're going to realize that those zeros and ones are really going to drive the average down because they're doing a really simple algorithm here it was just take three and calculate what it might

be so what ended up happening is actually our transactions are being drastically underpriced because geth was taking into account all these meth which is interesting right because we assume you know you go to estimate gas you're going to get the right value but what we haven't really seen it up until really this year started to like really get aggressive is the amount of mev in every single block because we're talking almost every block at this point has some sort of mev in it and it's affecting the way the nodes were calculating the way geth nodes specifically were actually calculating um their transactions or calculating the gas cost

when giving users an estimated number um so the fix was really simple uh at least so far um we're still monitoring it now we're just basically re-filtering out the zeros and ones uh we're still monitoring it though and there's a good chance we might my original idea was take the standard deviation after removing zeros and ones to try and really push us to the to that perfect sweet spot of light might get there um a little too invasive uh right now but we're gonna keep monitoring it to make sure you know is this gonna continue to be a problem or does by removing zero ones do we actually stabilize the gas fee uh because actually the same day that i went to go make this fixed there was

like three or four issues opened up because people are experiencing very similar behavior they just couldn't they just didn't get down far enough in the rabbit hole to understand why it was happening um and yeah so kind of you know as i said it's something that we're continually monitoring and it's an interesting situation because you know as i said you could be writing a bot any day you could be writing about for whatever reasons and if you're making an assumption to call estimate gas against your own node against nefura against whoever you need to be certain that one there's something a little bit more going in there that they're waiting a little bit better and also too you need to make sure that like you're not getting invalid results you almost have to do a double check and you

know that's kind of like our learning lesson here was if you're going to go and build any sort of automated transaction system you actually the real solution here is probably you know if you want to go to the decentralizer which you really should is you should be asking the node not some third-party provider but because of the immense amount of mev we're seeing in trends in blocks nowadays it's actually probably makes sense for you to maybe even just query the third party provider as just a fallback just to make sure you know am i like in a good range because if you don't and you end up falling you know gases at 200 like it was when we were dealing with it and our transactions were estimating around 80 90.

uh your transactions aren't even going to show up on etherscan it's become really confusing and it doesn't show up on etherscan because i said before nodes are just filtering it out they're just basically saying your transaction's so under price i'm not going to keep it but your local node that you're running is going to keep that transaction continue to try to propagate it saying like hey somebody some miner please pick it up please insert this into the next block so it kind of gets hidden and one of the reasons we found that too is one of them you know nodes flushed their flush their uh node and all of a sudden like everything started going through again because the transaction actually disappeared um and they were able to restart it the non-set you know was stuck at the beginning um so kind of what's next step

uh i went you know last night i kind of said i should probably once the pr got merged in i realized like well i should probably monitor to see how things are going to make sure that we're actually continuing to you know this is a solution that's actually being fixed so i didn't have a lot of time to spin up my own notes can you guys still see my screen i think you should be able to yep cool um basically i said well we should start actually tracking this and we should actually start you know thinking about are the different versions that are running out there different node versions of different clients are they actually estimating gas correctly or do we need to go and actually look at their algorithm on how they're doing it and track it better so i spun up this

website last night with a little like indexer in the background um it's kind of broken in some spots but it's accurately representing the information basically i didn't have time to spin up enough full nodes it would have taken like days to actually you know get that spun up uh since i finished at like 5am uh but so so some of the stuff here the endpoints you're seeing are just like endpoints from different providers different people i was asking on twitter if like anybody had open rbc connections i could connect to but something that we can see right away is that uh if you guys know the based currency they they had their like site map open and i snaked their rpc endpoint so haha gotcha um they're actually running the latest

version much like uh we are but i don't you can see that their estimate gas is like brutally brutally undervalued underpriced um and this is like what i'm talking about it's totally random it's based on the node and how it does its sampling we have if you see based gods and chain save geth chain save geth up here it's running the exact same version as base gods it's actually running a very similar version to infira but we're all reporting totally different values which is expected you know the samplings will be totally different but you know if if this endpoint here was being used to actually construct transactions for an automated service they would completely be butchering them

you wouldn't get the right values your transactions would get stuck and who knows and there it just recovered and came back now something fun if we go to this block number i bet you ether scan will tell us that it has an it's an a weird block oh it did not wait huh no i didn't own this one i'll try this one because it was so low a lot of them like ether scanned out yeah see there you go unconventional ordering so you can even see that like this was probably because this block alone the latest had such an

unco it was even flagged by ether scan as like having mev in it um this api this endpoint right here this node right here was actually drastically underestimating so moral of the story you know and what kind of like what we found is um you're gonna i think moving forward until mev is actually like a little bit more solved i don't think you can actually trust this make ass anymore and i know this is a topic that's been talked about a lot before but um it's something that i think is kind of becoming a much bigger problem nowadays simply because of how prevalent methods that's you know only going to drastically increase here on out um and yeah i just want to say a little

thank you to elizabeth for doing my pr reviews on the standard deviation one even though now it's one line but that was helpful and patrick from eva labs you know he helped me really debug the absolute crap out of this and martin gary and renee from geth because they were actually very swift to get this done which was nice it got merged in in like 24 hours or something they're very very good about it because they saw it as an issue and yeah thanks for joining perfect thank you greg that was uh that was very interesting indeed um i think there were were there some questions uh uh rickmoo shamelessly plugged a gas

estimation daemon tool also yeah probably worth using uh one question here from five five five six five five five five what is a based gods um no idea just uh some d5 app um and then brandon best was really happy to see that you're using brave um so nice um and then here a question from aiden does this mean i should be using estimate gas for a specific node version in mind uh the

i think open ethereum is the most reliable right now based on what i've seen which is ironic at best but um it has the most reliable one i've seen geth is cutting a new release i think in a couple weeks so that'll have the fix in it but what i meant more is that like when considering what what the estimate gas will be um and automating my response the node in which you should just use rick you should just use its tool and blend it with like a third-party api and just make sure they're like kind of close to each other for safety everything always comes back to wreck now

it always is guy like solves all problems cool and then one more question from antoine um does this happen on other chains too or just eth uh i don't think other chains really have mev because they're you know ghost chains so yeah zing cool um thank you for that greg i appreciate it um fascinating talk um and also special thanks to everyone else too that helped you including elizabeth who's with us today