Dencun Interop Testing #33

Transcript

[Music] J oh

um welcome everyone we're live uh for the 33rd testing call now um so we we wanted to talk about the E bandwidth uh issues around blob gossip um but it might be worth giving you a couple more minutes to see if more folks join um so perhaps it makes sense to chat about um if there's any devet 9 updates and then specifically devet 10 I saw that uh we're sort of getting ready for that so I don't know if anyone from the devops team wants to give a quick update on where we're at with Dev Nets and then we'll hop into the uh El bandwidth conversation and then uh disc discuss blot testing after uh yes sure so de9 is going

forward uh there hasn't been much change since our last call on Thursday um PK 910 might have something to share regarding lobs uh yeah um we've been spamming um the network over the weekend with uh blobs and uh I've just grabbed some stats about um how um blobs are included in uh actual block blocks by clients and uh be expected that all clients combinations would um include blobs at the same rate basically but we have seen

that especially Aragon and ethereum JS clients are producing less blocks with blob inside so they are producing block blocks without blobs so that should be something we should look into at some point sorry so does this mean they're not yeah they're not able to produce a block with blobs is that correct uh they are producing blocks with blobs but not as many as other client I see okay okay so the average has been around 69% and we see e ethereum JS and Aron client

pairs at about 33% so much lower than the average got it and what I think one possible modification sorry so one possible modific modification to the script would also be to just query how full the pool was and then correlate that to how many Globs got um got included because ined mentioned that it could also just be a pool related thing and yeah pools are pretty full from the perspective of yeah we're spamming it but I don't think we've manually checked if Aragon's pools actually accepted all the transactions or any other note speaking for ethereum JS I think

it's mostly resource related to the resource consumption because etherum GS just runs on one thread and uh all the pools as well as its execution also runs on one thread uh but yeah we have plans to make it more robust over time got it and aside from that oh sorry yeah Mario uh we can't hear you Mario not sure if you're trying to speak

you don't seem muted on Zoom but we can't hear anything now you're muted but we can't hear you okay I think someone else was about to say something when Mario oh I PK nothing from my that maybe I can start off um with one other thing um and then hopefully Mario's mic is fine at the end because then he would um fill more information there um he Mario's built this tool

called blobber um it kind of sits in between the beacon node as well as the validator you can I'm just finding a link and I'm guessing that's what you wanted to talk about um I have the link right here yeah perfect so that one um it's been deployed on the test net properly today like an hour ago and I don't think we've seen any proposals with it yet but um that would perform a couple of withholding attacks and yeah I think we just redeployed um so Mario's worked on hype test for it as well and I think um he just redeployed the hi inop instance to include the

blob got it um and then yeah so I saw we're we're getting ready for devet 10 um I it seems like we're just collecting the the the various uh releases or branches from the different client teams um do we yeah do you know when we expect it to uh to have Genesis or do we still want do like do we still want to do stuff on devet 9 and see that stable before we get defet 10 uh launched I think last week we mentioned

that we might do Genesis on Thursday uh I think it would depend on the different client teams if they can make a Rel because now the kcg the minut kcg file I think it's in a PR at this point I'm not sure if it's merged in in the C specs yeah a good question okay uh it's still open it's still open as a PR um I can link it I'm not sure what else needs to go into it before it's muched but um that's the one we should be using and it would be nice maybe it's not a release but but if it's in a branch and you can point us to which branch it is that's already

enough okay and yeah there was some reviews around testing just an hour ago or something so um yeah I suspect that like it's still being worked on a bit Yeah so I think we we should definitely wait till uh we should definitely wait smed and then schedule any kind of time okay yeah that sounds good um and one more thing it is we've um deployed satu with the um it should be collecting blob and block information

with Lighthouse uh prism as well as tuna um so have a look at the grafana dashboard I think I linked it last week I can fetch another link um and that should like I think the bigger issue right now is what are we actually looking for CU we have all of this blobber information and all this block information so if there's clear questions we want answered going forward to test Nets and more stable releases then kind of reaching out with those questions now is a good sweet anything else on the dev before we go into the

El discussion oh yeah Mario are you back yes can you me yes yes we can okay yeah you go through okay so but before I uh so the comment I I was about to say before was that so we have this uh the problem that i' I've seen at least in Hive is that um most of the clients if something is some of the transaction The Blob transactions are reor back into the mol I mean they normally on normal transaction they they are reor back into the mol but that is not the case for blob transactions so what might be happening also is that we are um so the transactions are missed because we are

not resenting them uh from testing side and that is really hard to do um uh from with the Trans testing transaction tools that we have right now um so that might be one of the re I I think the only client that I've seen that reorgs um the transaction back into the mol is go ethereum um and not the rest of them but so this is something that we have to probably take into into account uh at least in h we I had to take it into account and and the problem is that it's not really easy to to to detect that the transaction is missing from the testing side um yeah that that that was all

and so it's the idea that if there's a reorg we want El clients to re-include transactions from blobs I mean I mean it's is it's not required anywhere but I think we're going to take uh we need to take this this into account when we are sending spam transactions oh right yeah okay yeah because I think because we have to R roll back somehow yeah right yeah cuz I I I know we discussed this like on and off and I don't think we never we ever quite got like a a unanimous consensus on like what yeah what the right behavior is so

I yeah I I guess I we shouldn't like enshrine it in the hi test somehow but I I understand that like the Hy test sort of need to deal with that that edge case yeah yeah so and what we did uh at least the consensus on the on the on the hype test was that uh we shouldn't require this and I understand because uh you have to do some fancy things on the mle to Cache the uh the blops just so you are ready when you when it's reared out you can put it back on the mle so it's not so straightforward to do so the hyp test are not expecting this at the moment um but yeah it's something that we have to take into account where we

when we are spamming things we it it might be the case when we are missing some transaction and then we can we cannot proceed with the rest of the transactions because of the nons and whatnot but yeah that makes sense yeah okay um I think we have folks from the e side now um like clients do you want to give some uh background on the blob uh bandwidth conversation we had on AEV last week and then can hear from other

client teams yeah I unfortunately don't really have an specific bandwidth numbers or you know know exactly what Peter is hoping to get the numbers down to but my understanding is that whenever he started running up on defet 9 and was propagating the uh blob transactions the way that our transaction gossiping fetching logic uh works is that we sort of just fully saturate our bandwidth so if there's enough blobs out there the pool then we're going to just naively try and Retreat them as quickly as we can and that's there's like no throttling mechanism there for um

clients coming online or lots of blob transactions coming into the pool and I think what he would like to make happen is have some way that clients throttle themselves so maximum amount of their bandwidth so that we're able to continue fetching the blobs if there's a lot of them to fetch and um you know not inhibit other operations their client operations or um operations for that user's local internet got it um yeah any other El team want to chime in about their thinking around this or or yeah any numbers they've seen

yeah so in NE mind uh the feature for uh like having into into DB process blob transactions and then reing to the pool uh we don't have it on devet 9 it is uh already implemented on the branch but it wasn't touch for some time and will be refactored but it's we generally have it just need to need to polish it and include into into the branch from which we are using image on 9 and uh about throttling we will have it but we

don't have yet got it any other team have thoughts um okay I guess is is this a question around

death like or I guess yeah for the teams that like don't have super strong thoughts do you see this as a blocker um yeah to to to keep moving forward or um yeah are you comfortable with the way your your gossip is set up sorry is this is about um like blob transactions on reor or about no just in general the fact that um The Blob transactions increase the gossip utilization of the node and I think gu wanted to implement a more uh

uh well like like a a a fetcher like something that to basically control the amount of bandwidth that it's using um but I'm curious to see if there's uh if if if there's anyone who like if if other teams feel like this is also a blocker for them basically or if they're fine with the bandwidth consumption as it is or if they don't have they don't know what the current bandwidth numbers look like I don't have an answer of the top of my head we have to look into it I don't know yet got it um yeah I also need to ask

the team I don't have an answer either yeah got it and so if we do have something like a if we feel like the bandwidth usage is too high and then we need to implement something like a a specific fetcher or like a throttling system do people think we would then need an additional devet to test that or is that something would' be comfortable testing on on test nuts directly um yeah because I guess if if if we're thinking of potentially holding definite 10 based on this then we should make that clear now and also figure out like how we yeah figure out exactly what we want to build but if it's more of a nice to

have and something that we're comfortable testing on on test Nets then um yeah there's not as much of a rush to to make a decision or agree on next steps I would make the case that it's probably okay to test it on actual public test Nets cuz technically the way for us to test it on dev Nets would just be to spin up a new node and then I guess sync to head and then monitor how it looks so it'll probably we also show up in sync tests when we do public test Nets I think one thing that's nice with public test Nets as well is that you get to see versus historical data right like um so that'll give us like a good

comparison whereas devet there's just not a ton of historical data um but and and then I guess potentially the risk though with test Nets is just nobody actually uses blobs on it uh so we should make sure though to actually spam blobs on the test Nets once we uh once we for them 99% sure we're going to be spamming Globes yeah um I'll I'll send anyone some Gordy eer if they need to do that gordi um but okay I think I think that makes sense then so let's uh obviously teams haven't had a ton of time to look into that um like we should start looking at it

and then I think testing it quote unquote quot for real on Gordy seeing what bandwidth uh looks like there and and if we want to add any sort of transaction fetching capabilities um yeah let's do that on the actual test net and and even for that though you can imagine forking Gordy realizing we need to add a fetcher and just shipping a different release and running that on Gordy and seeing how it works like we maybe don't even need to wait for another test net the um or for all nodes to adopt some sort of fetcher or throttling system because we' just be able to see it on the new nodes yeah and then by the time we're at Main

net whatever we come up with would obviously be like a default um it could also be just a flag possibly yeah yeah yeah some clients or or some uh operators might have beier machines right yeah yeah you know and then they can allow different metrics so so maybe it's it shouldn't be like an onoff switch but rather like a configurable flag yeah that that makes sense okay I yeah I I think I think that approach is reasonable um is there anything else that people feel is a blocker to Dev Net 10 except

the stuff we've we've already you know discussed on aore devs last week um or is the list uh of what we wanted to see before going live still accurate where um obviously just Hive tests being stable the Mev pipeline being um being more tested uh the kzg setup being in and then we had something around getting better tooling around uh analyzing blobs I'm not sure if there's been any progress on that in the last couple days oh yes yeah that's the zos stuff that's been set up oh okay yeah yeah I see I see the gra fin awesome um I mean the the B base tooling is done

and we have the data the draw data now and then it's just a question of what do we actually want to see we've set up some template dashboards but that's just our idea of what we want to see I'm sure have a better idea yeah okay that's cool that's for recording the metrics and we can yeah have an initial look yeah one more thing that me uh blob inclusion is still not something that's working so it's it's still something that needs to be working before I think we canate it not 10 possibly it's not a blocker do we know what like where the blobs are getting stuck is it's the

Builder yes because every every client payer was able to submit at least one blob okay according to Dora so uh we we we just haven't got any metrics on how many blobs were included with uh M but maybe that's also something we can take a look somehow so do we do we just not know or do we know that the Builder is not including blocks with uh is not building blocks with uh blobs it's a sh ringers blob we haven't found a okay block that had a blob that was

built with a is there a way we can can we just turn off local block production what do you mean by turn off local block production or for which one it would fall back right is not available you start possible using a flag so I can help you with that we we have such flag to in nether so we can turn off local flops as well okay then what we could also just do is uh load star nether mine just disable local block production and see

if we can uh fill up it block pool and just see if can actually subm any valid blob blocks with blobs yeah take who and like as well yeah wouldn't it be easier to go the other way around um the Mev uh relay has a website of all the blocks that it has produced so you can just query them and see which of them have blobs if the answer is nothing then we have a problem right The Flash 3day website exactly CU that'll give us a definitive answer right it's extremely unlikely that a relay has been running for one week and there has been zero blobs that have been for inclusion so I think yeah yeah it

seems easier to query that first assuming the answer to that is zero then we can maybe run this yeah so teu Lighthouse and load star I'll have it and then at least NE mind on the E so we can run those and and try to force Bobs through um I have a or at least I think our working theory was just that the build strategy doesn't take into account blobs that would be the extremely naive um idea and we have M flood that kind of submits extremely lucrative um blocks sorry lucrative

transactions think the Builder is just kind of grabbing them and ignoring the blobs but it's is there a way for the blob transactions to just just submit like a really high like like cuz the Builder would the Builder be like ignoring the transaction type because wouldn't it wouldn't it just look at like the total gas paid for the transaction I honestly don't know what the Builder strategy is we're just running it on strategy greedy and I think me flood submits really juicy transactions so that the Builder always grabs them got it so we know if this me

blood tool is making blobs though CU I think that was an issue that they had a Mev tool making blobs I don't think we have something like that yet well no but if Mev flood does if that's what we're using but also like I think the flash BS Builder should just use the M Pool so like if we just send like a massive or I'll say very valuable Bob transaction it should become clear we can try that out too I guess we can just make some blobs that are extremely expensive and then submit them and see if they I I think we don't have M flood running anymore because we run out of e fund right

now yeah we spent like over a million eat already on the on the expens yeah yeah yeah to clarify on a Dem yeah yeah um okay so I guess we can at least look at the blobs or blocks built on the relay and yeah does anyone know if there's a API for the Relay um where we can qu yeah so there's the specs I just dropped and there's like API specs okay nice um although I will say I've even been trying to build stuff off of these and found some things

that aren correct so proceed carefully I guess there's only 800 payloads so worst case looking at those eight sorry no not manually I was going to say um can't you I yeah I guess you need a node to pick up the transactions from uh or the blocks from yeah I mean we just need to see the slot numbers and then we can query the rest ourselves I guess yeah exactly and so yeah and I'm oh data API yeah okay yeah I think I also yeah so there's like a small view of it on this this web

page the Boost relay D9 yeah barav is just linked yeah okay so I think using that we can probably I haven't even download approach you can just call the web page and it has the data in it and then you can just use set in L magic to get the best yeah okay so with five calls there we should be able to get all the historical data see if um yeah if any of them have blobs and then if not figure out whether it's an issue on the Builder side or whether just running the clients with a

different setting uh will we'll get them through and you're running the Builder from this DB Branch right yeah so the Builder is uh off of like clients latest de 9 perit as in the sh's updated to like L later submit but yeah it's the flashboards branch right yes yeah but it would be good to have another Builder also okay so yeah I think there's a couple uh

threads we can follow up on offline around all this um anything else people want to bring up that we should be looking at in the next couple days before we uh try and launch devet p sorry one more thing just to close the loop here it looks like this Docker image is from yeah like what map merged but if you go look at their D Branch they have stuff on top of that that looks very relevant uh so the docker image is says um from what I can tell it looks like it's off of commit the 6 E6 and if you

just look at the history there's like it looks like all the important Deb stuff is on top of that like blob bundle fixes on which branch is that Alex flash Bots Builder denb okay got it so yeah if you just look at the link I just pasted you can see there's like a few more commits on top I check the yeah nice yeah we can rebuild an image and then try if that helps and looks like Philips already done an initial analysis um the last 200 blocks have zero blobs so that's definitely an issue right yeah

start progress okay you should look at the hash from Matt's commit yeah oh yeah I see what you mean you have four more commits to it okay and it seems like at least the last 200 blocks there's zero blobs so um clearly there's probably something going on there that's not the the clients okay so Vim versus Nano is

something we can leave to the devop standups blob censorship Alex what can we do oh no sorry that was a joke yeah that the builders are censoring the blobs okay anything else is Ed iess possibility that M Rel uh will be censoring blob transactions if if they cannot extract any me from them uh okay this was a joke I was just saying because if you look at the relay there are no Bobs it's

like they're censoring them uh the relays have they're indifferent to blobs they would love blobs if they pay them so it could be if that if there's some fancy nft drop that does a couple of blocks without any blobs yeah I mean whole idea is right that if all the economic activity moves to l2s then like you know in say a year that's going to be the juicy transactions yeah makes sense just pay more if you want to include it yeah okay anything else before we wrap

up sweet well no thanks everyone um yeah let's uh figure out these last couple issues in the next few days and we'll talk oh sorry Mario yeah sorry I I I thought about the topic but I had another topic that I want to talk about that's okay yes please yeah uh I think it might be a little bit better if I just share my screen sure um yeah let me try Okay so can you can you see my

screen yeah okay so regarding the testing of the blops gossiping on the consensus layer so we have developed like this handy tool that helps us um yeah make gossip testing basically on on on on the on the hive and also on the dev Nets um basically what this thing is just like a proxy it sits between the beon client and the validator client and it basically listens to the all the rest API calls and it basically just lets everything go through everything that is not a proposal so when a proposal comes comes by it's it basically intercepts

the proposal and in and it can sign the blops and the block also but it can also do some extra actions um and this extra actions is that what we are currently testing on hive there are many of them um I I'd like to share basically this link I you can you you guys can come over and see what it can do most of the things that it can do is just send extra blops um they are all listed here extra blobs and conflicting blobs incorrect blobs and all that stuff that we can test um and we have a set of uh tests on hive uh that we can just execute and um yeah uh so I can I can I

I will just share the link after after uh after this call on the R&D so you guys can take a look um at at the list of things that we're doing and just if you can um comment on more things that we should test also welcome um the main thing that we are having trouble right now is that basically this thing is um when we try to um intercept a proposal and inject more blops we need to create a a dummy client that connects via that peer-to-peer to the consensus clients and the problem here is that we are finding that most of the clients are simply just Banning us because we do not implement the entire consensus peerto

beer um the the entire consensus peerto beer uh um set of specifications so we just part partially Implement that and because of that we are we are down scored very quickly um so one of the things that would make this tests uh a little bit more more consistent is that we need somehow to uh add this blobber um blobber U client into the trusted Piers uh list and this is um very easy to do I I only tried it with lighthouse and they have this this parameter that is basically just thrusted peerto peerce and we give it a perer ID which is the blobber perer ID because we know it

beforehand before launching the bler and Lighthouse is basically B passing all tests but the rest of the clients are uh well the rest of the clients that I've tested um are just disconnecting us quickly and we don't have like a uh consistent test result at the moment so even even though we have like six tests right now uh um over here most of them are not consistent and we don't have like a like a um a consistent set of results uh to to begin U submitting bugs and all that um so so yeah that's basically it so we have this this tool and um if I I will share this this page so you can take a

look uh every every test case that we have written for Hive at least has the um description what it does uh and what it sense to the to the clients uh and what what it should expect from the clients um we are also planning on uh deel deploying this tool on uh well parsh is is currently working on deploying this tool on the testnets and so we will'll see how the clients Fair there because there are a lot of more clients in there and they're possibly going to be more issues than in Hive but I think this Hive test suit will help us at least with the um um the advantage here is that we have only one type of client per per test run and we can single out if one client is failing with

one specific test uh test type um so yeah that's that's basically what we wanted to share I will share this link and the hive test with so you guys can take a look and uh some uh some feedback is also welcome and if there are any questions I can yep it's also deployed now on the Tet uh but just on one note so L house R is doing weird along with te nice any questions comments also if if the rest of the

clients all already have something like thrusted beer IDs uh parameters that you can share with me that would be really helpful um at the moment I only have a a prism already uh let me know in the in the R&D Discord what's their uh equivalent parameter for this uh but but yeah please if the rest of the clients already have this please just let me know I can just try to include it in in Hive yeah I can send it on the R&D Discord after the chat since we use those flags quite frequency uh just one question Frederick um are you guys uh using the flags on the main branches of the clients or do

you guys maintain a fork with the flag uh it's on the main one uh I think some clients have it kind of as a secret flag so to speak um but uh we don't we don't uh maintain any specific um clients or forks for it it's the Cent team have it yeah thanks yeah thanks and also this is um just just CL clarify this is just gossiping uh the blops we don't at least in Hive there is no no set of tests yet uh regarding the um the the requests the the the peer-to-peer requests for or for

blops so that's still work in progress I guess um this is just gossiping oh say were you gonna say something Fredick uh no just that I will send it afterwards I was going to say that if you want I'm on my phone now so I can't send it otherwise I would do it just right in the chat I will do it in in a few minutes but I was going to say that if you can also look at ETB um if you if you know where that repo is and um it will have the flags um

when launching the um yeah I think it has the flags built in there so awesome thanks anything else okay well thanks a lot everyone and I talk to you all on Thursday's call thanks everyone thanks bye thanks bye thank you bye

[Applause] s