eth_multicall Meeting, May 20, 2024

Transcript

so in the transaction you've sent to me you are trying to like send from yourself to yourself or what is actually happening there because there is like from address to address and it's like sorry and it's like the same address and then you give it Max fee per gas but you don't give it like amount to send or anything like that which one so you sent a link to me today it was the last link in our discussion on on Telegram and so you have in the like calls you have there two calls uh

in each you set explicitly Max fer guess while in the blocks there is like one zero how much it would be like so the max fee per gas in this transaction is higher than the base fee so they should succeed but yeah there's no value of being transmitted so those transaction shouldn't do anything how much inre the default value of like gas that is expected to be transmitted 21,000 is the smallest amount of gas I think you can say

so that I think the trans should consume 20 1,000 gas [Music] mhm but yeah no value is been transaction and for for that same one get crashes so but I don't know why get crashes to that that curing because when I look at the logs it says that it is at index one so the first transaction went through and the second transaction was like out of gas that's fine I think the balance should be high enough for that but yeah what is the calculation I wonder for

the max gas per block in this case it should be the pre I think should be always coming from the previous block so whatever is the previous block depends on that are you running it on hi that that well I'm running that on the hive as well but then I usually try to see if the main produce the same issue and with the main I'm getting the same issue all right all right all right and yeah by the way I was really wondering what should be the like error code for the cases we don't like expect beforehand so

this can be example of say imagine it's a case of something we didn't knew beforehand would happen so is there like error code that you would prefer us to use for such cases I think there's the internal node error I think that something went wrong with node but what if something but if we think that something went wrong with the processing of a block so it's not something something wrong with the node it's something wrong with the request but we just like haven't in investigated that specific case does she have ideas on that I'm not sure how that is

handled um I kind of didn't I missed which case you were talking about uh so on I think it's last uh the like at the bottom of the trailer found issues there's G crashes to invalid nons and never mind uh just cases that as an exception but still it is not to the to the form I mean crash is definitely not good um I said in the chat and yeah get

crash is that and then never mind rtion error zero and I I feel that the no should never return error code zero because we don't have kind of error code and is it this one nether mine treats gas zero weirdly no it's last gu one last guess one gas crashes inv yeah meth handle crashed it's not exactly the same as as the request for NE Mind by the

way there's couple case where get crashes one is the nons and then one is the one I pasted in the chat there's no nons error there it just cres to that and I feel that quy should succeed but it it crashes both it seems to be just something about validation maybe or yeah it's if you put validation off then then it succeeds it's about the validation the validation makes it fail yeah yeah going to look into it but yeah the Alex question was if there's a case that something fails but we didn't know what it is I I think it should be an internal node error or

inter yeah but what if we really really think that it is is not an node error but it's an error of the input so example input is not valid on the main net or input is just not valid somehow uh for this like block processing that we are expected to perform I think we have inval input error yeah but the input is valid the thing that we like try to do with it is not so we don't know before we start processing of this block that the input was somehow wrong and when we started processing of the block we like find it out but it's not transaction not guess not something like that we've managed to

figure out beforehand but something unexpected happened on processing of block can you give a more concrete example because I think we have error cases for most validation checks and if uh the node tries to do something that should work but doesn't then yeah I guess it's internal error for example when decoding or I don't know up something from database and yeah right yeah that would be good enough uh so uh let's begin today probably uh on our side we've uh finished up first review of lucash got

some stuff to fix not too much began the process of initiation of a review by some other peers on NE mind side and we are beginning to do performance check before like being able to merge it in and we would really really want to help kilari on this because so the idea is imagine we are doing like 10 10 or around that concurrent requests to is simulate and each request contains something really really performance heavy performance like uh encouraging and we would like to

test a few at least cases where we can actually see that nodes perform as expected when like say 10 concurrent or 100 concurrent requests come in and for that we need basically three uh we devised that we would need something like three cases the one simplest one it is quite easy to make just transfers it is easy uh but there are a few hard cases so first one would be transactions that try to use as much evm op codes as possible so like diverse evm op codes randomized that are all valid and all somehow like just stressing evm as hard

as possible and the other one is blocks interconnecting with each other like information heavily referenced throughout the blocks for all 256 blocks if possible so something like that would be really really helpful for us to understand the performance characteristics of this method and if we really need or do not need to lower the like basic uh amount of gas or amount of blocks that are allowed this method yeah that sounds good except with the diverse up codes and I'm not exctly sure how I could generate of test

cases yeah that's an issue for us too sadly yeah I'm not very familiar with OP codes and I most all of my test are written with solidity but I I guess with that you would write them in assembly but I was wondering if if you could somehow just random CH random bites and then you make that kind of test that you try to execute random stuff and then some of would be invalid and some wouldn't be invalid but it still but yeah I don't know what is the likelihood that you produce something in a way that is doing something instead of just failing sadly it will be extremely hard to produce anything valid from like just random stuff maybe we can like gather good transactions from the real Network

and just bundle the contracts in the C data Cena you wanted to say something oops sorry wrong button um yeah um I think that will turn into a evm benchmark not necessarily A is simulate Benchmark I don't yeah but the issue is that before is simulate we had no such thing as like possibility of direct evm benchmarking done in concurrent way onto the single client because normally you would have your

single line of blocks and even in like any way you would test it sequentially and here we can have concurrent access concurrent forks and thus different performance characteristics when data is accessed from the past because you one could imagine that say you are on an archive node and somebody is performing this simulate and is accessing some part of the regions from the past and somebody else is accessing another part of the regions from the past and your note just gets gets crushed due to the fact that there are too many random accesses I mean I think it was possible with e call as well um if I'm not

mistaken it I think would be much harder because in E call you would just like make one call and and that would be it and if you had for example say eight concurrent threads each one allowing you to make one concurrent request it would like end quite quickly and here if you would have like eight longest 200 blocks long is simulate calls your evm could really struggle so like in expectancy that every of these calls would take at least 30 seconds uh so my understanding is is that e e will have the same um protections

against OS as eth call does meaning there's like a gas limit on how much you can execute before it just stops you is not like Neath call you can there's some limit hard coded into the client I think you can change it via command line parameter when you start nether mind that if you try to execute an eth call that is bigger than that in terms in the sense that uses more gas than that number then it will just hard fail when it reaches that amount um I thought we were going to use that same value for yes we do yes we currently do yes so I think both of them in both cases a single request can in theory consume the same amount of resources whether it's an e call or E simulate that's true yeah we can generate those 256

blocks and that's probably heavier and then then I was thinking of that with the state over I don't think we have any Li liit on how many state overs can you can do and I guess I don't know how much those abuse Den not but I could create a I could create a transaction that has huge amount of state overrides but I would imagine that those are just that there are setting of value but those are not actually very heavy yeah yeah the state overrides generally should just be in memory I would assume no no dis access and so um worst case scenario you could consume a huge amount of memory I think there's also limits on the Json payload size I'm guessing that's going to be what ultimate limits what how many state overrides you can do and my guess is is the Json size limit is way lower

than any meaningful amount of State overs it's like five Megs or something like that right yeah I think we looked on into it like a bit earlier yeah there is a limit and it's if you do five megab of stad override is that still fine uh yes because the 5 megabytes of Json is going to turn into much less in terms of just raw data just due to Json being not encoded particularly efficiently um and all you can really do is just again it's setting in memory there's no dis access and so you're going to use what five to 50 on the high end is my guess Megs of memory which is negligible for I mean I don't know exactly how much

I'm guessing here but my guess would be somewhere between one and 10 times the Json limit and I think in the past for other things if I remember correctly Json distalizing actually is the most expensive part often of these requests uh the actual execution is sometimes cheaper especially when you have these really large requests and I would not be surprised if that's the case here as well so I get you so basically as if Co haven't get haven't like any problems we probably are more or less safe as long as we have similar boundaries and guarantees on its execution so yeah but still I would love to have at least a few like few cases where we

would try like to push it try to see how one would hack if simulate yeah I do think there are some differences so e simulate um in eath call you you do your call and then everything is basically just evm execution which is um very tightly controlled by evm gas pricing like there are people who spend a lot of effort getting evm gas pricing to be you know at least the right order of magnitude um with e simulate there are some things that we're doing that aren't costing any gas like you know generating blocks is functionally free as far as this goes I do think we have we have a limit on the upper number of blocks you can have though at and so I think just you know having a

test to test 256 blocks and make sure that it's not like obscenely resource intensive for some reason might be useful I don't know how exactly we'd um analyze the resource utilization but um just you know generating the blocks is pretty easy but that would be I think the the area that I would if I was going to do some testing for this performance-wise I'd look at the Block generation because that isn't accounted for in the gas pricing and I guess transaction even even transaction count is still 21,000 gas per transaction so it's just the blocks that are free I posted a tool from Marius that um does

this thing where like it generates random looking transactions um it could be used for not for not for a maybe consistent Benchmark that can be reproduced but just to see um maybe we find worst cases or something like that anyway just discovered it recently I think it's pretty cool is it just used for testing correctness though like I don't know if it's been used so much for performance measuring mhm and it's just generating like transactions from p a to prb or it is like some evm up codes also involved

yeah yeah no it's evm up codes and it is it generates as far as I know valid uh valid evm bite code MH does it require any contracts to be posted before it would be actually usable I think it's pretty Standalone and self-contained um this was used recently to find bugs in the um in devet zero of Prague and it did manage to find a few mhm I'm always terrified when a fuzzer finds bugs

because to me it's an indicative of some other deeper problem um because if you can find bugs via random testing how many bugs did you miss because random testing is notoriously bad at hitting everything but humans are also very bad at hitting everything so this is true humans are very bad at everything I agree get tired yeah I I thought you trust machines more than humans I do trust machines it's just this particular machine is do basically doing a random walk of a search space that is nearly infinitely big and if you can find bugs by randomly searching an infinitely

large space then it means there's a lot of bugs out there and that's what it's terrifying but it's not usually not uniformly random so usually you add that kind of Edge Cas is there like like zero for example is that kind of edge case or negative one or plus one or that kind of think thing and like it's generating also valid op code so it's removing those nonvalid OP code so you kind of like try to make it to generate that kind of meaningful stuff as possible so I'm I'm a huge fan of parameterized testing which it sounds like what you're describing where you basically Define a human looks at the code and says here are the Edge Edge numbers um test the combinatorial set of all the edge cases together in various combinations um that

is super useful in my opinion for the exact reason you describe fuzzing historically though has just meant randomly throwing stuff at the wall until something breaks now maybe maybe what they're doing is actually parameterized testing they call it fuzzing because fuzzing is funner to say I don't know fuzzing includes that you add some Randomness to it and I guess you are just saying that you would Bru for through all the kind combinations that you are given yeah like I If if you're doing if you're doing edge cases you can usually narrow the set down to something that is searchable within finite time finite human time so you you know it might take like a day or two to run so it takes a long time but you can usually

get the Comin tutorial set down pretty low it's once you introduce that Randomness that I start getting scared because like I said if the search space is so big that you need Randomness in order to even start to search it then it means that if you hit something that's terrifying because you've hit something in a search space that's so big that you couldn't actually search it in reasonable amount of time forner case combinations are Ms always it is extremely hard for a human to see all the combinations of corner cases so if you even would have three it would be too difficult to cover them all human right so like I said I'm a big fan of the human goes through and defines here's a corner case here's a corner case here's a corner case here's a

corner case and then you set up a program that will test okay Corner case a without Corner case b or c corner case a with Corner case b Corner case a with Corner Case C Corner case a with b and c cor case b with C Etc like the whole combinatorial set and it takes a very long time to run but it covers the entire search space it's only when you start randomizing um that you cannot cover the search space in finite time and you do not have enough insight to choose which space to search and if so you find something it means you're finding something in a massive search space and if you hit something in a massive search space that should never happen it's basically a guarantee that there's a lot

more that you missed that you didn't hit well with it symol I'm sure that there's a lot cases that I have missed and I haven't thought of and I think it would be useful to have faster running and comparing get and determine to each other I'm quite confident that the fer would find stuff there because well at least I'm not smart enough to figure out all the edge cases that to think of to test but my if you want to voluntary and think of all the edge cases then I would appreciate that it's usually just you know test

zero test negative one test positive one text test un Max test un minus minus Max like and then if you've got conditional that are on some borderline test those numbers as well like test Max gas minus one Max Gas Plus One Max gas exact like you just basically go through and Define all of those and then you set up a parameterized test that just does the combination like combinations of all of those things and that is super useful I'm a huge fan of that type of testing I think it's in fact I think it's one of the best kinds of testing that you can do it's just if you if you just do it then I I'm I'm I'm fine of that there was a c tool um a long time

back uh that I really wish Microsoft continue working on that actually found did this for you and so you would give it your code and then it would go through and find all of the boundary conditions and then it would automatically generate tests that hits all those boundary conditions and automatically generate tests that were combinations of various boundary conditions being hit and it's essentially generated code that would ensure that every possible code path in your uh code base was reached it was very cool I don't know why they dropped it it was incredibly useful it's just a prototype unfortunately was this e related or or something else this is this is like I don't know 10 years ago 15 years ago for Microsoft research called uhz or something like

that I for what it's called how it was called let me see if I can find it in the Ean Foundation is would it be possible to use some resources that that could do fast testing on the it simulate or is it something that we have to do or nobody else does it um I think there's few people who do fuzzing um I i' have to ask around and see maybe I can get some help and to at

least set up the process and then we can take it and run with it yeah I if someone can set all those things up and I'm always I hate setting things up and if but if somebody can set them up then then I can also f with it and try to think of inputs and that kind of things that could be needed to be tested and now when we have neet mind and get and it would be kind of useful easy to do in a way that we can just compare those inputs to each other and then if they find some differences then we can check those see what's going wrong but still there's a lot no difference between get and net mind so I

don't think we would benefit from the fast testing testing that much yet but but after after we get all those yeah after we have those human test all matching the then we benefit from yes first uh take the low hanging foods and then but I think it would be because if we have a tool that works for a simulate then it probably would very easily work for other methods as well basically want something that would Generate random blocks and then if it generat random blocks then we could convert those to each sybol calls and then run those and I would guess that

kind of tool could be something that exist already I don't know if that TX fing is doing that kind of stuff h no TX F not um I don't know the just one transaction not multiple ones uh yeah m i mean multiple like a lot of them but um not sequential transactions so they're kind of independent of each other but yeah I guess we could use to generate 100 transaction and then we make a block out of them and or multiple blocks and then we run those and go results I think the dev will be in the other parameters as well

like not only the transactions but like all the override fields and everything yeah but for those we can also generate some random stuff well maybe not for something it doesn't make sense to gener random stuff but somewhat random stuff you can don't those do we have anything else uh I don't I don't have much I didn't end up doing a lot uh last week but um in one of the discussion I don't know if I mentioned this but in one of the discussions something interesting

came up basically um we it seems like either 3074 or 7702 for account abstraction um are likely candidate for Prague and given that one of their main features is batch transactions I think if simulate can play uh an interesting role in in testing those methods uh well if Mike ask if they brought it back I think 3074 is probably like there was a lot of community push

back against it sorry I was referring to what like pasted sorry so yeah I think um like yeah if simil can be possibly interesting like the intersection of isimulate and account abstraction I think there's room to explore there yeah I see that's right away it should be useful and use the when you making batches you to simulate them first and then send it but do those add something that we need to add to it simulate is there some features that are missing from us I think the only requirement is being

able to simulate from a contract address and um we can't do we can do that already so I think that's all it's really needed and the B batches are actually multic calls I guess and not multiple transactions uh so 430 are you talking about 4374 whatever is getting included I guess that that's one is is so either way so whether it's 3074 or 70 whatever like latest version is um essentially they all allow an eoa to execute code and if an EA can execute code then it means an EA can execute multiple a series of things so like ex execute an

approval followed by a swap on Unis swap as a simple example right and so um you could use something like e simulate to simulate a series of actions using your um using some tool like Interceptor for example and then um those would get batched up into or compiled into a um call to some sort of multi call contract that is executed from the point of view of the EA so the EA is the message sender but you're executing a series of things and so there's an interplay there in the sense that you can use e simulate to prepare the transactions you want to do and then once they're all prepared then you could

use an invoker for 3074 or um custom code for the other one in order to actually execute that onchain automically either of them would be super useful for us assuming people um start using invokers yeah was there some discussion that you were mentioning it simulate Cena about those um yeah

well basically I was bringing up it simulate um um as something that that people should Implement um and and someone mentioned that like this interplay between 374 and so basically they were saying yeah Sim it simulat would be very useful for testing and it's like basically simulating this batch of transactions so when when we have the account abstraction then sorry uh I need to go to the door hey guys sorry for being

late okayo but I guess that load testing is some something that should be done I don't know how much heavier it multic call is or it simol at is compared to e call for example that's probably something that should be done for both get ne mind so for testing um there are two things one we need to make sure that the timeouts work well right the and there are two ways of time I think at least for NE mind there's a just time timeout and there's gas timeout let's say like has depleted right and that it covers across blocks and the second one is that the

actual mechanism of it simulate itself should be fairly light but we need to confirm it right that when we do an amount of blocks that this itself is not heavy right because the blocks are the one the main piece that's not accounted for with gas accounting right yes is it heavier to reference that kind of block that is in the currently in the generated by it simulate or heavier to kind of reference block that is kind of in the 256 blocks or should that be the same like if I create 100 blocks and

then I refer all those sh 100 blocks is that heavier then then just try to look up the blocks like which already happened in the blockchain I guess it would be some much same to be honest I didn't understand that I think the question is is are is it more expensive to work with simulated blocks in memory versus um blocks that are actually part of the blockchain and I think the answer is unless the block happens to be in memory already it's almost certainly more expensive to go to dis to get it than deal with simulated blocks in memory all those blocks in the not in memory or they are all all those 256 that each simet can access those are

probably in the memory already yes it should be yeah so I would imagine it would be the same that there shouldn't be really difference for Access for building off of blocks I think you're correct so the same as um eth call E call can be derived from can build off of any Block in history that the node has access to and so that part will be the same for each simulate the issue the potential issue which I agree with Lucas is unlikely but worth making verifying is that if you simulate 256 blocks then it means the node has to you know generate 256 blocks in memory and you know the node wasn't necessarily designed to with

the idea in mind of having lots of blocks being spawned uh at the same time and so maybe there's some performance degradation there I would more say about you know preserving state with those blocks between those blocks Etc right because it's all in the memory temporary so for example just making sure that we don't blow memory somewhere too much sure my guess is is probably not I can't really think of anything that the evm gas accounting wouldn't already account for that you could blow up yes I do agree that it's just a yeah yeah

well there's this we have we can generate 256 different states of the blockchain I guess those are the going to be difference yeah so just generate um 256 blocks each one maybe do some State changes and each one do some contract calls that mutate State just the kind of normal operations that each do a little bit of work and then um I don't think it needs to be a super complicated test we just want to kind of verify that at the edge it doesn't cause problems if I'm understanding and Lucas correctly yeah just just a s to check that it doesn't blow up in any un UNS in any suspicious

way how how should we go go with this I can do some of the tests like this but then I don't know how we should be running those like I don't want to want this kind of test to that test sh it in a way because then it's just takes longer time to process so I think my recommendation would be we generate a a test that just um you know is Big so to speak um where big means it actually simulates 256 blocks each block has some State changes and or some State overrides and some contract overrides and some um contract calls of those overwritten contracts with overwritten State and then just like one test like that and then um Lucas o and Cena or whoever else can

then just run that test manually like maybe it can be marked as skip or something I don't know how the test framework works with Market a Skip and then someone on a Dev team can just run that manually and just kind of monitor performance metrics externally as it's executing I guess that need to be run many times and multiple concurrent request to kind of like really overload the node unless you want to analyze it if that something in that transaction takes longer than should be I don't think we'd need concurrency like I said before I think the concurrency issue is the same as should be the same as eth call um I do you guys disagree do you think that the if

there's a problem it may only be noticed if you do like you know 10,000 simulates simultaneously or would you notice the problem if there's just one uh so generally we rate limit how many heavy State access calls can be done in parallel and we queue them so there is a limit for that in never mind so you you want to hit if you want to hit the time out then you need to somehow make that it loaded the node so that it couldn't process those in time so those inq can also time out right

so uh sure oh okay but then you should add huge amount of things to the queue so to get the timeouts I guess it should be easy enough right you just copy and paste 100 times the request yeah but then you Al need to monitor the no to do that kind of stuff yeah so I would leave the actual monitoring side I would leave that to and Cena I don't think we have the expertise or skills to even know what to look for but I think we can at least generate the you know the the test data so the the big the big e simulate

payload that kind of maxes out as much as we can in one simulate call like I said I think the the critical things are lot lots of blocks so all 256 blocks State overrides contract overrides and then transactions that actually call those over written contracts um and each block maybe is also full um like Up Up full enough that we can reach almost reach the um default node gas limit for the system system 100 probably right Mega gas or I don't know if it's for G different is it 100 million for the it's for 100 million okay so yeah so basically 100 million

gas worth of transactions spread out across 256 blocks each transaction touches um code that maybe a mix or half transaction touch code that was provided via an override and half the transactions touch code that was not overwritten so we get a good mix of data there some of us going to dis Etc okay yeah does sound reasonable to everybody and then once we get it uh once we generate the trans test data Oleg you can run it another mind and monitor performance however you guys do great do anything

else nothing for me all right thank you all for coming and see you next week thank you Bye by Bye by bye