Kurtosis Testing (Breakout #19)

Transcript

okay we're five past we are going to record this and post it to the internet so i would suggest that we do what you wanted to do and then go from there sure yeah basically uh i can give kind of like a little bit of a deeper dive into kurtosis for those who are interested and for those who are watching this in the future um just as really quick uh i guess recap kurtosis is a platform for doing development and testing for distributed systems the idea behind kurtosis is that you can leverage your docker engine docker is your kind of underlying container engine and

then kurtosis provides an abstraction layer on top of that in order to do things various different things with that so the principle uh the principal idea that we have and actually um i usually am a big person who likes to draw so i'm gonna try this let's see if this works let's do this um i think i i think i broke something i sent the wrong link on discord i got it i think i sent

well let's see pooja are we in eight seven three seven right now let me check yeah i think there is a way to invite uh there's the wrong link on the okay i'm going into the other i'll be back i messed up so this is the link here i'll be back we were all in another room waiting for this to start so there should be a probably nine or ten people joining oh awesome yeah we know people are busy you know maybe folks just weren't interested in the session so

hey i'm really i'm really sorry i messed that up we changed this was set up by tim we changed the link to a catheter's link so it would record by default i changed it on a calendar invite i didn't change it on the pm repo i then confused myself today and sent the pm repo link and also went to the other link internet is difficult sometimes i very much apologize i'm okay with that right looks like we got a bunch of

people popping in now cool so this is super exciting okay uh so welcome everybody um thanks so much for uh showing up to our little little working session um my name is kevin i'm the cto of kurtosis you might know me as amy breese on your discord channel because i've probably bothered every single one of you on your discord apologies for the pest drink that was me trying to get everything working with the ethereum module so uh to kind of give a quick recap about what crystal's is and uh what we're doing the idea is that um you know you're able to set up like test nuts and do all these things with docker or kubernetes kind of depending on your flavor but as i'm sure anybody who's done this knows there's kind of a lot of work that

you have to do in order to get that going you have to understand some fairly low level stuff around docker and especially around kubernetes or write the charts that you need so kubernete kurtosis provides an abstraction layer a top atop the top docker and we're currently working on providing kubernetes support as well which will allow you to work with these with distributed system networks much easier so for in the ethereum context the idea is basically we have built a you can almost think of it like a little application that inside of inside of kurtosis which really means inside of docker you can spin up a test net or a local a local devnet i guess you'd call it which

contains two halves contains the execution layer clients um up to your specification as many as you want um with various different flags and logging levels and such like that and then after the execution layer clients are up and running it will create a cl clients a cl client network and then that's going to be with various different configuration parameters that you please in order to get things working and then after you have both of these running you can just leave the network running and it will eventually perform the merge so that was a lot of information dumped at once so i'll take a couple steps back now and just give a really quick walk through of the kind of fundamental concepts of kurtosis and then you guys can

will eventually culminate with running the merge module you guys can go ahead and see this so kurtosis is a client and that client is installed by a package manager i'm on mac so um you could just do brew install uh kurtosis tech slash kryptosis class i believe um i'm not remembering the exact incantation right now but we have installation docs right here i'll go ahead and send to the group yeah here we go we're just text slash tap slash kutosa supply and that will uh get you kurtosis on your machine however if you're on linux you of course will have a different package manager so we have installation steps for those as well let me get this little bit out of the way

so once you have uh once you have critosis installed on your machine we ask you to make a an election about whether you'd like to opt into our metrics program um the you know obviously feel free to make whatever choice you want on that we use the metrics only for product analytics so uh making sure that um we know what what things people are using and what things are worth throwing engineering points behind so once you have kurtosis on your machine you are going to have the kurtosis clyde right here and that kurtosis clay uh the first entry point that you work with with the ketosis clay is enclaves so enclaves are like little walled gardens that you put your distributed networks inside of and very concretely what this is is this is a docker network

under the covers that you start all the different machines all the different services inside of and i'm a big drawer i like to doodle so i'm gonna go ahead and do one of these guys and we can hopefully make this a little more visual too so you have the boundary for your docker engine and the enclave that you create here is going to be a separate network and we can go ahead and see this if we do kurtosis enclave and what this is going to be doing under the covers is creating a new docker network and then you're going to get back a new empty enclave and then if i could do kurtosis enclave ls you can see that i have one enclave

running with this enclave id right here and if i do kurtosis enclave inspect then i can inspect this enclave and it will be empty so inside of my docker engine i have created a docker network and we can see that this docker network corresponds to the name of my enclave id here so kurtosis provides like a couple of tools for for doing things with enclaves like you can we can go ahead and remove this enclave as well um kurtosis will go ahead and do that and if you had any services inside of it it would blow that all away so that's kind of the the enclave concept so enclaves will be the uh the most

macro unit uh object that you'll deal with so now for the ethereum context uh we have a module we call it uh modules in kurtosis are like packaged set of instructions to the kurtosis engine so once you have an enclave of course you want to start stuff in it and that stuff that gets started are going to be containers so we could imagine that we have something like this and you know we could have like a guest service and maybe another guest service maybe we have lighthouse and maybe we got nimbus as well and you would start all these services inside of kurtosis and you could do this

manually um so we do have an sdk for communicating with the engine so you'd have a little bit of like a library and go or in touchscript that's out here that lives in whatever code you want and then that would communicate with the kurtosis engine and you could basically write all the the code necessary to set this up but then you'd have to have that code maybe as a library on your machine maybe you'd write it as a bash script as a bunch of client commands it gets a little bit unwieldy so we have this notion of modules which very concretely are a little tiny grpc server that responds to one command which is execute and that is a container as well

and what that will do is that will package up all the instructions to the kurtosis engine so that all this stuff is encapsulated inside of the module i dumped a lot of information so i'm going to give a quick pause there any questions comments okay so uh this module is the main thing that we've been working on uh it's why i've showed up in all of your discords to ask questions about what are the flags necessary in order to start all these various different nodes all these different types of el and cl clients so a module is a kurtosis because it ends up as a container a little tiny grpc

server it is a docker image which means that we can distribute it via docker so in this case we have the kurtosis tech slash eth2 merge kurtosis module and so this is the docker image that we use that we package the module inside of so you can run a module with kurtosis module exec and ketosis module exec will create a new enclave it will launch the module inside of the enclave and it will execute that module so fairly straightforward now

where things get a little more complex is that modules can be parameterized so in our case we have a we have configured the module to take in json that configures its behavior so this is how you specify the number of participants a bunch of different information about the network that is getting started up like how many validated queues do you want like what different app box do you have you're able to configure the terminal total difficulty as well a bunch of different parameters and the main place for seeing these bits of information is going to be on the module repo page so right here we have a

go ahead and send this to the chat we have a git repo that contains all the code for building the module it's written in go and it is really you can it's a very fancy uh set of instructions for to the kurtosis engine that's really all it's doing so this particular module has is configured for json and these will be the all the different parameters that you can pass in so if you'd like to start your bl node or your cl node you can for example set different images here you can set various different command line parameters if you wanted to set arbitrary parameters

same thing with the cls as well and then of course like i said we have all that networking information here that you can set if you'd like to set it so none of this is required this is all optional and we have defaults associated with that so we have the module we we have the kryptosis client command which is going to do the module exec again because for this as an sdk you could do exactly the same thing in an sdk you could put all this uh this logic to execute a module inside of a unit test for example um so we actually use kurtosis to test kurtosis and what that looks like is the go unit test framework imports the kurtosis sdk and then it tells krita says hey please give

me an enclave do this thing inside of the enclave do a couple of assertions tear the enclave down so what we're going to do here is go ahead and execute that module so what i'm doing here is i'm just passing the execute params to the module via a file that contains the configuration that i want and that is this file over here so i've written this json file i like to keep it as a separate file rather than putting it all in the command line because when i get json highlighting i figure out if i've missed a comma or or something like that but

again you could if you wanted you could just pump in raw json right here so to see what this is doing uh we can go ahead and look through the logs so we can see that we've got creating an enclave so now we're going to have this eth2 merge kurtosis module enclave if i come over to a new tab and i do enclave ls we can see that we have an enclave with that id just as expected and if we do kurtosis enclave inspect here we can see that this enclave is starting to get stuff inside of it so we have one module which is the module the eth2 merged kurtosis modular contains the logic that we expect and it's starting to start things inside of it so this

pre-launch data generator service is actually a container and that container is for generating el and cl genesis information so if we come over to our docker engine we can see that we do indeed have a container the pre-launch data generator container which kurtosis will use for generating the el and cl information then if we come back to the module logs here we can see that we have the parameters that we started the the module with and then the module is just going through the various different steps necessary to set up the network so um right here yep this is the generation step uh concretely under the covers what the

module is doing is it is calling that pre-launch data generator getting an el genesis file and then we're starting to add the el clients which in this case is only one guest client so we add one el client here and now we're going to go ahead and wait for that el client to start mining if i was smart i probably would have flipped the weight from mining flag to false which would which would speed this up at the cost of breaking the network slightly the cl clients would behave a little bit strangely but in your your production tests or in your use case testing you can go ahead and leave this as true which will basically say wait until the el clients are producing

at least one block and then after they are producing one block we can go ahead and start adding the cl clients so any questions so far um while we wait for this uh well wait for this guy to start mining any questions thoughts comments concerns or maybe anything that's confusing anything that i'm running through super fast perry had a comment in the chat okay so um so while we're waiting for this uh i can go ahead and give a little bit of a demo of the of the uh features that kurtosis uh provides for debugging kevin we have a

comment in the chat um when you have a module or when you excuse me when you have an enclave here kevin a very common use case is that the you'll want to dump the enclave information so i think i've chatted with all of you about this about uh doing a a log dump of all the containers that are running inside of the enclave so i do see a raised hand your audio is off oh no you guys can't hear me can you no we can but you can't hear us oh no okay give me one second uh i think my airpods did something goofy

[Laughter] i can just expand on my comment in the meantime um since then the entire e2 module is just a docker image if any of you modify any of it at whatever time if you want to add a test or ovulate the most common use case has been in my client i've changed to flag so i need to change the flag in the module um you can just compile it yourself with scripts build sh and then test it and make a pr for them and then punch it for you okay i can hear you guys now can you hear me yes we could always hear you i apologize for hearing people smiling in the little side window here and then i was like something's something's going wrong here

okay uh where did i lose you guys at no you didn't lose us we heard you the whole time there were just like a couple of comments that you didn't hear like for example perry uh had mentioned build scripts okay okay cool i'm gonna i'm gonna shut up then because i'm sure i was talking over people um not too much we've generally been quietly listening but if anyone has a question that's a great time um i was curious how you're generating the cl genesis states like what tools yeah 100 so there is a docker image which is put together um by and probably

correct me from ron but i think skyline is one of the core ether folks uh yeah okay so there is a docker image that is i'll go with repo here um this repo right here is a dockerized version of the tools necessary for creating validator keys and creating genesis uh cl genesis and el genesis here so i'll post this in the chat and basically what we do inside of kurtosis is we [Music] wait a minute

i should go to everyone um so we basically call these bundled tools that are packaged inside of the docker image and then there's a flag for pulling cl genesis here and um we pump in a templated uh cl genesis config let me get over my computer's chugging a little bit so merge module this is the downsides of a 2015 mac while running a proof of work miner while having golan doing indexing while screen sharing so apologies for a little bit of slowness here

eventually we're going to get over to the eth2 merge module so i can show you a little bit more about the code inside as well and i can give you guys a little bit of a walk through in the meantime also one other question can i give the like kurtosis uh module a git repo rather than an image of the client and how they compile it or do i have to basically do that with myself and create the document locally you will have to you have to do that yourself just because we or at least like when we've started we made the assumption that we don't know anything about how that docker image is getting built so um like which sub-package it would be in and inside of

that um i actually think that's a really cool feature and it's something that we saw um like a complimentary company doing where they basically like always push a doctor image um from every git commit but uh but yeah currently we don't do that i'm just interjecting that there's another repo where um i've got workflows for i think four different clients if you have a client you want to build that isn't a part of those workflows just make a pr or let me know and i can add them and the ci for this runs nightly so you get an image spit out um at 2 a.m and if you push something later on and you want to just manually run it you can

just run a manual workflow and specify which branch you want it will then clone the branch of that client repo and then spit out an image i thought j slash client name nice all right we're gonna do this old-fashioned way um so we have the cl genesis um so the um so we have uh we're inside of the ether merge module right now so this is that module that i was showing before and we have the

templates here in this static files subdirectory so [Music] actually it goes smart i'll be showing this to you guys so um inside of this ketosis module uh subdirectory we have static files and then uh we have the genesis generation config cl and then we have the information so this is really just a go template um ago template that the module will pump in the correct values for so that's actually how the um cl genesis file gets concretely generated so

we can see now that the el node has started mining and now we've proceeded over to the cl client uh generously generation so that's going to proceed normally this is quite a bit faster because my cpu isn't maxed out but uh you guys can play around with that on the machine and see how fast i know that you have the m1 machine and then things are going like insanely efficiently um so i i dream of getting one of those one day so uh over here we were talking about having um dumping enclave information so uh usually if something goes wrong for example inside of cryptosis it's very useful to be able to see the container

logs so um we have this command which is ketosis enclave dump um and kurtosis has the abilities let's see enclave output here so basically this is going to start dumping all the information inside of the enclave out to this output here but if i go here we can see that everything that is currently inside of the enclave gets spat out and we can see log files as well as the docker the docker container inspect information um and this is super useful on my end for example like say kurtosis does something that you don't expect like

there's an error a bug or something like that if you can provide me with this enclave dump output that makes things a lot easier for my site so i can see exactly what pentosis was doing on the inside and we also have the ability here to um shell into a service if we want to so if you wanted to do kurtosis um service shell and you grab your enclave id and you grab say a given service or let's actually do the death note here and you go ahead and you can get a bash

shell we try to be a little bit better than docker and that will try to give you bash first before you do sh so you're not always stuck trying to like land up in the asset shell and then lastly you have logs as well and if you wanted to say follow the logs then we would go ahead and start tailing logs as the start showing so we can see that we're actually watching the nodes live logs any questions about that so far okay so we should be getting pretty close to generating client data co client data

and the very last thing that i can show you guys on the code side is uh how you would modify your particular client's launch within kurtosis so maybe you have a flag change that needs to happen or maybe there's some sort of structural change to the way that the service is launched inside of kurtosis module inside of impul we're going to have this participant network and i'll show you guys uh to find the separately um afterwards so inside our participant network we of course have el ncl and let's say we wanted to modify

the gef um the guest node right here we go to this death launcher file so every client has a xxx underscore launcher file which is for modifying the way that the container is started inside of kurtosis so every single one of these files um they have just a bunch of constants here um corresponding to uh various different pieces of information uh that we use for giving or launching the file or launching the uh client so you know we can see minor awards accounts and port numbers and that sort of thing we have a map of used ports which are going to be the ports that are exposed for the container so kurtosis wants you to tell it about hey what are the ports that this container

is going to be listening on and uh the reason for that actually let me do a deviation here the reason for that is because if we come back to this kurtosis enclave inspect here we'll see that every single service here actually has local host port bindings so you are these services are running inside of a docker network so they are isolated by design so they can't interfere with each other you don't have port conflicts and things like that going on but you still want to be able to access these so we can see that each of these ports that is now on the geth node right here has been bound to an ephemeral port on my machine so for as long as this enclave is alive i will be able to

access this depth node via um by these various ports and i'm going to do a and do the bad demo thing which is i'm going to try to remember a a command off the top of my head and all right maybe you can you can help me with this one um if i wanted to get the death um if i wanted to get the geth block number then i think block i just sent it to you you have to change the ip it's on chat

thank you so much okay so let us grab so 8545 yes and there we go let's clean that up a little bit now we get we can see that we can communicate with these stuff inside so um we've seen various use cases for this you could actually run a node outside of kurtosis connected to the stuff inside of kurtosis um you could uh obviously make rpc requests in order to do any sort of debugging that you wanted to like make the network do any uh anything that

you'd like and yeah so that's that's pretty much the host for binding so coming back to my uh what i was originally talking about here the um the reason that krytos has asked you to define the ports both is to expose it and to allow you to uh bind things on your local machine so um this is all just various goal logic um in order to define basically a class for those who come from java and this is the real meat and potatoes of what's going on so this uh each of these launchers implements a launch interface and that launching interface contains a bunch of information that the node can use in

order to start out so enclave context this is actually the handle to the ketosis engine itself so if you want to manipulate the the contents of the cryptozone clay this is how you would do it and the the this is the kurtosis sdk uh we have the image that the user has requested to start to start this with we got the log level we got the global log level as well which is all corresponds to the this guy so all these different parameters get piped down to the launcher so um if the user specified a new image right here if they specified a log level um i specified the global log level right here you can see that all of those parameters get piped down right here as well as extra parameters and then it's basically up to

up to us up to the launcher to do whatever we want now um we've written all of this for all you guys that you don't have to you don't have to deal with any of this but the the real work is getting done with this git container config supplier so every every time you've launched a service inside of ketosis ketosis says hey uh please tell me how to start this container and if we go ahead and scroll down to this function we can see that this is a function which returns a function um a little bit confusing but uh this is the function that actually has starts your actual uh starts your actual container

all this stuff which looks kind of complex is actually just sending um is actually just initializing the file system on the container we're not super happy with the way that this api works so i would say don't um don't fuss too much about this we're actually going to be simplifying this in the near future or to make kubernetes compatibility work and then uh you have a big command here to launch the actual death node um we add a couple more flags like if this is a boot node for example add the boot nodes flag pretty standard and then we go ahead and tell kurtosis the information so we say hey this container is going to use those ports that we declared up top we have an entry point override in this particular case

which is uh you know if you're not familiar with dockerfiles it's basically just a thing to tell docker how to do a various starting a way to start a container and then a command is basically an additional an additional part of that so once we have this this function we basically have a factory for manufacturing yet notes so we come up back up here in the launch function we have this factory so the container config supplier and then we just give it ticker toasts and that will uh we'll go ahead and um create our node all this extra stuff is just go ahead and making sure that we wait for the thing to become available before we return it to anybody else does that make sense was there anything

confusing about that okay so uh the kind of meta the the meta point that i wanted to explain with this is that like let's say you had a you had an extra flag let's say there's a kiln specific flag that needs to get added uh you could add it right here so um in that json we saw there was a extra grams um these extra params things so you could add it right there but if you'd like everybody who's testing with your clients to get that that upgrade then you can go ahead and actually modify the each to merge module

itself so this is open source first this is a critosis tech youth to merge module right here and then you can go ahead and find your launcher i usually usually just use the um go to file i'll go for a launcher and we can see there's all these different launchers with bezu launcher here there's gap launcher prism nimbus restart et cetera and scroll down every single one of them has this big long flags section here and you could be like oh you know what i just want to i want to add another one of these right here um go ahead and submit a pull request we have ci hooked up for this repo so you don't have to worry that you're going to accidentally break something um because everything will be really groovy

and actually uh pari has been submitting a couple uh prs very kindly to make sure that things are going ahead and work so feel free to feel free to modify these that means that anybody who tests with your um with your client will basically have everything uh good to go and working and party i think that if i'm not mistaken that you're actually starting to run this module inside of ci for the various different clients so that will also help with with price testing and making sure that every night that you're building your client so lastly we can come back here and we can see that the module has finished so uh we were stuck here

um this took about 10 minutes in on my machine when i'm not screen sharing when i don't have gold land open and everything like that i would say it takes probably about one to two um so i promise this is not this slow normally and uh we can see that we're just adding one lighthouse plant uh we also add a transaction spammer in order to make sure there is some load on the network um marius if you're on if you're online thank you so much for helping me with that uh so that will just be submitting just a bunch of small transactions um we have a couple pre-initialized accounts in the genesis and it just moves it just shuffles stuff between them it's not really doing anything useful um but it is adding is that its own docker container it is yep this is i'm not wrong it's the same file

we use on the test nets so it just does transactions and then switches around fields um oh excuse me actually marius's version is a library and we have doctorized it so i'll add that too so that gets started uh that gets started with princesses and if that's problematic for you guys it would be pretty simple to just add a flag to the module that doesn't add the stuff um pretty much anything any of this uh admin stuff i guess you'd say um doesn't get uh we could flag it off if you wanted to

and then lastly just a couple of debugging tools so we have pokemon as well as prometheus and grafana so folklon right here we'll go ahead and see that it is showing things working here um [Music] and i'm not 100 sure why this is sometimes it seems like uh probably i don't know if this is a bug with pokemon where we get to the zero zero zero zero group but this is in fact working um even though it doesn't doesn't show it but i use this for the epoch a lot and when you have a lot of clients it's pretty useful to see if everybody's on the same route because when it is reporting roots i think there's like a little bit of delay in torque monitor show i think it might be like a small bug um so if you have like say four clients

going on here then you can see if they're all uh agreeing on the same four or not and then we have uh we have grafana um so this is almost definitely gonna be still broken through that bug in the demo from last week our engineer is on vacation so he's uh he's gonna be coming back on wednesday to fix it so um this guy uh once it's working then you're going to just go away we're going to uh you're going to see pure information and you're going to see um excuse me actually i need to take this dashboard

link here um and the module by the way um you can think of modules like functions so they took in the json parameters and they spit out json parameters as well and these are just kind of like useful things in order to interact with the module we can really report any information that we want here um there's probably going to be a world in the future where like the the pre-funded accounts in genesis are also reported here um that's like really not a big deal so um we can see right here that uh right now we've got the bug um but once it's going to be working you're gonna be able to see all the different clients um see what their beacon head slot is see what their peers are we've got el peers as well this is currently just the um seal uh cl clients

so that was kind of the that was kind of the overview of uh ketosis like what ketosis is doing um what the module actually does and um how to interact with the module uh how to debug with the module if you wanna you know if you have the desktop you can go ahead and feel free to do logs right here um feel free to use the keto supply if you want um shell back and forth uh we've got the enclave dump in order for you to dump information about an enclave outwards um oh one other thing i should mention is that we actually have the ability to attach rebels so um

here we can do ketosis new and then add this and what this will do is this is going to start as a node repel as a container running inside this enclave so you can start interacting with stuff directly so for example if there's anything you didn't that you weren't able to do through these ports um you can go ahead and do it from directly inside of the enclave and so you know we can see that we have this node rifle here we've got tab complete here we have that enclave context object so

again the kurtosis sdk which is imported inside of this ripple right now so if we wanted to do thing something right here and we do have the awaits available for those who use typescript and we can see that any information that we would that were able to do inside of the module using the ontario context we can go ahead and do uh right here inside of the rebel and the last little bit of information um kind of a feature that i'm personally proud of because i think it's kind of neat is uh rather than like you might not have a package that you want so for example if you wanted others js inside of here uh we can go ahead and do ketosis enclave inspect on that enclave and now

we have both the enclave id and the grid of this interactive grapple and then if we do the process repo install grab the enclave id grab this and then pump in the package name uh kurtosis will actually in the background install the package inside of the um your replica so then it will be become available to you and you can start using that inside of the nova so if you wanted to like communicate with um if you wanted to communicate with the services inside of the enclave from inside the enclave um then that would be uh that would be like that uh benefit of communicating inside of

the enclave each of these enclaves have a service id here which is the first part of this without the um without the stuff after here and we can see that the service ids are already available right here so el client zero client zero etc when you are active when you were working inside the enclave so when you are when you are not going through these ports uh you can actually use the service id as a host name so i'm not super familiar with javascript but um if i had like a ping right i could do like ping cl client zero and that would work that would actually um that is the domain name of the uh of the ceo guys hear me again

this guy's there that's a real pain um okay so that was that was kind of a quick overview um [Music] we're getting we're getting close to the end here so the last thing i want to give you guys is uh just a couple bits of docs in order to make your life working with qriklis is easier so all of our docs are found at docs.kurtosistech.com i'll submit to the chat so we have everything we have a quick start which is basically if you just want to be going from zero with kurtosis uh you actually run um you do set up an ethereum network and you actually write

a typescript unit test against that ethereum network uh we have the installation docs that i went ahead and showed earlier we have temp completion on kurtosis as well because basically it's a real pain to remember everything we have a using the cly docs so basically all the common operations that you end up doing inside of kurtosis you can go ahead and use this as your reference for the ply everything in the ply as well takes in a um [Music] takes in a h-flag so we have the closest desk age we got closest enclave dash h we got those on clear.h all this different stuff to show all the different flags um

so anything that you need for that you can always pass in the dash h and then we also have right down here uh we have documentation for the actual uh ketosis sdk so just in case you want to use a contestant sdk like let's say you wanted to use kurtosis to write your own unit tests uh you could go ahead and see how this is done here so you would always start with a kurtosis ketosis context this is kind of like the top-level connection to the kryptosis engine it's what allows you to create enclaves destroy enclaves that sort of thing and then when you call create enclave you're going to get back an unclaimed context and that enclave context is what we saw

before here and that has a bunch of enclave specific information so you can go ahead and load module in this case the ethereum 2 merge module if you wanted to set up kurtosis inside of your enclave inside of a unit test which could also be done inside of ci if you wanted and unload module and all these various different things you could add a service if you wanted you can simulate network repartitions um so this is uh this is one of our uh let's say a little bit more advanced features is that we could actually simulate partitions between arbitrary uh various different services that are going on inside of the network and that's pretty much what i got um doc's.kurtosistech.com

we have a discord as well so that should be on our it's here yeah here we go so we have a discord uh we have a merge channel on the discord here so we got the youth merge feel free to show up there or just show up in general um again i am uh on on this discord i'm kevin today uh my handle is new bruce which looks like this um or you can just have mentioned with kevin today uh if you had any problems at all please please please don't hesitate to scream and let us know um we are a startup so we're trying to make sure that our product is as awesome as

possible and we'll make sure that we're responding to you guys pain uh if there's any pain at all as possible so please feel free to shout at us makes the product better and makes you guys experience better um so don't be shy about that and that's all i got pari i don't know if you wanted to briefly mention the running nci thing or if there's anything you wanted to chat about with that if people have questions we can do that first but otherwise i'd love to give you guys an idea of how that looks in the ci i'm curious about how you think about hive versus kurtosis yeah so um so we we started kurtosis

uh actually in the avalanche ecosystem so that's kind of where ketosis started and how it grew out into um so when we when we started engaging uh with pari and folks the it sounds like the hive um from my understanding of five hive is is focused a lot on like unit tests it's kind of like or not necessarily unit test but but mocking very specific things in order to do very specific scenarios so um [Music] the you know when we were talking through the high code base i think there was something where um there was one uh like drop certain packets uh to simulate like block delays if i'm not mistaken uh so hive as i understand it fits very well

for the use case of where you have like one to two clients um where one of those might be one of those might be mocked for example uh and kurtosis right now is doing the like i guess you'd say like the one to n so you could you could set up one client inside of kurtosis but um you also have the ability to set up all the way up to end up to the limit of what your machine can support all right okay i also have an uh a question uh is it i mean this is great for for ci but i'm trying to figure out how would be used also for for for a dev perspective while you

actually developing things and you have something running on kurtosis and you at some point you want you are gonna you know you want to deploy a bug fix in in your running uh your running network see but is it an easy easy way to do or you have to actually deploy the um a new image and when you start it again actually the network needs to be restarted completely uh yes actually your um so if you use the hostname as the um as the connection parameter in between uh in between different nodes

you can stop um so and you would just do this with the kurtosis sdk so you could do this with the node um with the node reple or you could do this without the go rep or anything any or go sdk you could shut down one of the running services and then tell kurtosis to add it again and what with the same service id and then it would get the same domain name and then all the traffic that was previously erroring when trying to connect we'll go ahead and do that again um it'll basically like the connection will heal because the domain name will exist again so that would be your use case for being able to like pull out a service and add it back and you can use a different image in this case because you have to

redeploy manually specifying your new image right uh yeah you can you would basically just specify whichever image you want to drop into place and kurtosis will always try to use the latest version of the image so um uh the dev flow that um and you know i'll caveat that i haven't done this before so um so i know the domain name i know the domain name will work because that's a docker feature it's not something that we do but um the the dev flow that i would imagine is that you have your repo which is building the image and then you have a graphics network and you're like hey there's a problem with this so you tell crypto says hey please rip down this service and just right cool done got it um container is stopped now so that domain name is now free and all the other

network is failing to connect build your new image and then tell kurtosis again hey can you please add this new service with the same service id services back started with your new docker id from your new docker image and then the domain name is backups i mean this maybe this kind of workflows could be highlighted in some additional uh yeah dogs so maybe that then then help us as also the developing part and you can we could leverage it yeah yeah yeah it's a super cool workflow and actually like one of my lasting regrets is that we haven't had the time to build a kryptosis service like new or service remove command so that you have

to go through the sdk because i think it would be really awesome if you could have the cly where you just specify enclave id and service id and kurtosis will tear it down for you um it actually wouldn't be that hard to build so if that's something that's super useful for you uh i don't wanna i don't wanna promise like in the next couple days but like i think like maybe in like two weeks time or so i think that could be something that we could definitely um we could definitely build yeah this definitely helps also the the dev side of it instead of yeah the the main objective which is the ci i guess and also i was wondering i still i saw that in your json you have the way of add additional um

parameters but i i saw during your presentation that is not possible to override existing parameters coming from launcher or is something that we it's possible to actually specify completely new set of parameters and overriding everything so this is going to depend a lot on the type of client that the binary is using so the um some of the clients i know that if you just add new parameters at the end it will actually keep the latest parameter i think this might be a posix thing so you could just add extra parameters to override but i know some of them some of them don't yeah it depends right yeah

but that would be that would be in my opinion like um i don't know if there's like like a way for for you folks to specify like hey like take one's at the end that would be basically the easiest way because otherwise we would be um uh otherwise we would probably have to specify no parameters then people would have to specify everything there might be a proper way to do it though but i think adding them at the end would be the simplest i mean there is the override but then the additional parameters but maybe an overcomplete or override option where you have to simply don't rely on the default one you want to spin up completely manually it could be maybe useful [Music]

okay um yes that's everything i got um probably i don't know if you wanted to run through the ci workflow really faster sure i can do that um maybe i can share my screen yes i guess you guys can see my screen now okay so what i've done is i've taken the kurtosis stuff and tossed it into a small integration test and this will be divided into two parts so the first part is the merge client build because there's currently some clients

who i rely on to provide me the latest image so instead of relying on them to provide me the latest image we can just build them ourselves and this gets triggered every day at 12 in the 2 a.m in the morning so i've what i've done here let me just show you the workflow for get for example so there's two ways you can use this first one is you either manually run it and when you manually done it you can specify which branch you want to use or it just runs on a schedule at 2 am and when it runs on a schedule it just default builds the merge kiln branch so i'd get the latest get image from the merge kiln branch then this ci gets triggered at 3am

automatically and i've basically listed the base parts of everyone's um docker hub repo and then there's a small script that says fetch latest image tag along with uh what the base image is so it'll go there it'll fetch the latest image which if you recall would have just been pushed from the previous ci and then tosses all of them as environment variables then i've taken the kurtosis config as a template and that spits out the untemp like the templated version essentially so now we have the kurtosis config with the latest images and then it just runs the module with the image and what they've done is

there's a wait for finality check right now that's only only test so essentially final lap finality has to happen by epoch 5 which is one epoch later than the best case scenario and if it doesn't happen it fails so essentially in about 30-ish minutes you'll know if your ci passes or not and in some scenarios that won't happen and just to trigger such a scenario i switched out the the deco image to the latest and taco's already implemented the renaming whereas get hasn't so tech get takuget will actually break and as you can see that's exactly what happens uh if you go to the ci run

you'll see that the final finality didn't happen in 32 minutes and because we want to later figure out what actually happened if you watch this repo every time it fails you get an email if it passes you just get an email saying it succeeded so if it failed um i just run kurtosis dump so you can get the entire logs check it out yourself let me just show you what that looks like and for example cl client 0 is tikku and if you go forward you can see oh yeah look underwrap response thrown take was basically not happy and then figure out what actually happened um

it just does some cleanup stuff i'm running everything self-hosted right now um i've reverted it so it will work again and when it works it still dumps the logs in case you want to have a look at what happened and yeah if you guys want to stress test this or have your own cis doing different combinations like previous version plus current version or something then just copy out this template change whatever you want to do and you can have your own integration test i would need i guess once we have the v2 spec deployed this would be a lot more useful because currently since we're changing the spec this will break a bit but maybe in a week or two it'll actually make sure we don't have

any regressions anymore yeah i think that's it anyone has any questions hmm and one thing to note that's really cool is that you did the um you did the enclave dump at the end of the ci run we found this to be super useful so basically if your enclave fails if your ci fails then you basically have the log package debug um we do that inside of our own internal ci

for those assemblies exactly and then i just ran a cleanup script later on but yeah feel free to rip it off use it in in your own ci one place i see it being really useful is testing your past version with your next version so you don't have incompatibilities between your users and we definitely need help writing more tests so we currently just have the finality test if you can think of other scenarios you want to put an integration test through then then we can figure out how to make it happen uh okay so kevin i have a question uh

follow-up question with regard to the development scenario that enrico talked about and so basically everything is uh docker network right yes behind the screens so it is possible that you know while the roses is running and has its enclave up we can manually spin up our own client and plug it in the network and sort of interface it with other nodes it should be possible right you mean uh using the interface of the domain name or so yeah we'll definitely make sure that uh that if it needs to hook up to other ports it will be able to hook up but

it should possible that you know i can spin up provide all the relevant uh end points api endpoints and cl or el should just work right yeah totally you can you can do this you can do this either inside or outside if you want like you could actually run um so for the near the near foundation that we're working with people run it outside of kurtosis pretty commonly actually so you'll have the network running inside of kurtosis you'll have those public ports on your host machine that um for connecting to it and then you can just tell your node that's running say like outside of docker uh say hey please like connect to those ports and then it'll um the connection will just travel back and forth right so then we even have to spin up

our docker image yeah you wouldn't even have to spin up a docker image yeah yeah sorry maybe i was misunderstanding um that's like that that is the easiest uh way to develop that it depends if it is exposed also the p2p part and udp i mean if it is exposed i think you can can join yeah the only thing that might be weird is that because docker does like ip multiplexing basically um the gossip might be a little weird but like if you're connecting directly to the network um like if you give the rpcs like several nodes then you should be i should be good to go any other questions or anything else i

can help answer okay well as always i'm on our discord i think i'm probably on a couple of you guys discords as well if you have any questions or you know want to ping me definitely feel free all right everyone hope you have a great day and uh we'll chat to y'all soon thank you guys for joining today and i'm gonna upload this video uh recording later on the ap and cathode is youtube sounds great

have a good one everyone thanks thanks kevin thanks bye