The ETH2 Medalla Data Challenge - Team Presentations

Transcript

05:25 - Opening & Introductions

hey everyone uh I am super Fizz or Hank from the e Staker community and we are here with the madasha data challenge submission showcase brought to you by E Staker uh e Staker is a community of users that supports uh staking on the the new ethereum Beacon chain Network um we'd love to have you join us sometime at reddit.com Staker or in our Discord at invite.gg Staker today I am joined with my co-host Michael Geeson from Tica uh our friends from East AER Buddha and Patricio um Buddha is with Beacon chain

and Patricio is with the Fantastic POA platform uh I don't know if we've mentioned yet but we are offering uh po apps for submitters to the uh madasha data Challenge and I think that's not just the people who show up here but we're offering to all submitters is that what we agreed upon Patricio great yeah so uh if if you're a submitter and you're watching this uh we'd like to reach out to you and get you a a PO app for providing a submission to that challenge uh we're also joined by Danny Ryan from the ethereum foundation um Danny has had a a recent fantastic success of getting the deposit contract uh for the beacon chain

up and running so congratulations there um and great with that I'd like to give you a broad overview what we're doing today uh the madasha data challenge was a data data visualization challenge proposed by the ethereum foundation um there were about 23 submissions and uh East Staker is kind of curious we wanted to know what those submissions look like from the submitters so um we sent a request out and got quite a bit of feedback from submitters who um who wanted to share with us and so we're excited to share that with every once a day um here we

have uh and actually yep I'm going to do my best to nope let's let uh all of the uh presenters pop in really quick and say hello where they're from because I'd like to get everyone a chance to say hello before we go on we'll use the same popcorn style we used in prep um so Michael if you want to say hello really quickly and then we'll pass it on yep Michael Geeson here also known as ana. e on the redit and Discord uh eaker communities I'm one of the moderators there hi guys enan here also known as B um I work on the beacon chain Explorer and spend my time on Discord in the you

Community yeah okayy I'm patrio I run po up um I do many things related is and I'm here to learn about all the submissions of theha data Challenge and I pass it to spun sight hi I'm Nate mccurvy from Splunk and online known as data physicist I'll pass it to Sid hey I'm Sid I uh part of the E2 data uh uh GitHub Challenge and uh I worked with Ali simos on on this challenge um I'd like to pass it to

him hey everybody um I'm Elias uh and I work for bison Trails as a protocol specialist and worked on this uh submission with uh Sid passing it on to gordal hi there uh I'm an associate professor of business at UAE University in Abu Dhabi uh UAE United Arab Emirates and we are a team block block data team together with Joseph Joseph heno he is a data engineer at Memorial slon cering Cancer Center thank you so much please uh next I guess sang

I I'm Shang from Omni analytics group uh we submitted two submissions to the medha data challenge can pass it on to Lawrence y um so I'm Lawrence at Omni analytics group as well and I'm here on to sportong I guess we'll pass is Anthony hey I'm Antoine I work at spun with Nate um that's it on to I think that's it okay thank you all right great and from there we're ready to get started sharing submissions what we've asked from submitters is that they take about five or eight minutes um to give us a broad overview of their

submission this is not going to be exhaustive um it is a supplement to the actual submission so the intention here is to give people a an easy introduction to the submission so that they can go and find out more later um so Michael yep so first up we have presenting uh cang and Lawrence from Omni analytics uh cang will be the one presenting they've got two submissions uh one is the proper proposers and uh skipped slots the other is the art of slashing uh so with that I'll give it to cang hi um let me share my screen real

22:57 - Siang and Lawrence from OmniAnalytics

quick okay hi um again I'm Shang from Omni analytics group um with Lawrence here and we are here just to talk a little bit about um both of our submissions um for the ethereum medasia data challenge just a little bit about me I'm a data scientist at Omni analytics group uh a boutique statistical consulting firm specializing in statistical analysis machine learning artificial intelligence and now cryptocurrency I graduated from Georgia

State University with a PhD in mathematics and please join us over on Twitter at Omni analytics to learn more about us our first submission is on analyzing validator performance titled proper propos ER and Skip slots a ranking and segmentation study of validat behavior and most of you probably heard that um eum 2 phase zero Gone live yesterday um we can get started on get being a validator today um to participate as a validator one needs 32 ether to be placed into

activation epot then wait until the first epot it is eligible after that a validator will be activated and may begin to propose or attest and you can exit voluntarily or be forced to exit our goal was to characterize validated behavior in order to construct sub type of scoring methodology and in order to do this we must collect data set that acts as a representative of the various behavioral aspects of a validator and we use data collected from the beon scan blockchain

Explorer our analysis track the behavior of over 880,000 validators the data consists of the following a row index public key index number the current balance the effective balance the number of block assigned executed and skipped the eligibility OT the activation OT the exit EOP the withdraw EOP and the slash indicator here we have a simple graph and we can see that around 20,000 validators were activated during the Genesis block and since then there is a

constant growth of four validators for Epoch however we also notice there are two anomalies one around 3,000 and another one around 14,000 in both of these instances no validators were on boarded uh these two small hiccups in the network just showcase that any technique that tries to characterize performance must also be aware of that exogeneous factors can influence validated behavior and therefore affect the analysis to C ize the validator behavior we use a weighted linear rank scoring

algorithm using the current balance number of successful execution the ACT active status how long the node has been active this number of skip assignments and slash indicator as inputs we would just calculate a score using this formula shown here it is just taking the rank of valuator along each of these variables and multiply each length with some weights and add up all these weighted ranks to form a new rank score then we can then sort uh then we we decided to settle it

on a selection of seven score tiers to differentiate Network behavior for instance tier one is the top three% that has a perfect track record of no skip slots and no slashing while tier seven is the bottom 6% who are mostly inactive and have the lowest current balance our tear all possess distinct behavioral characteristics useful for discriminating between them however there is also a deeper level of heterogenity that exists within the tears themselves across

the landscape the scores within most tiers correlat around one another forming localized region as you can see in this graph in tier four to tier seven all have multiple regions where validated scores rest here we are able to highlight some validators within the data that possess representative characteristics and behavior native to each localized regions the table here shows a representative samples of most common valid profiles within each tier and as mention there are seven

tiers with one being the highest and seven the lowest we start with tier one we have our perfect proposer a validator who exhibit the best combination of behaviors with track these validators have the highest number of successful blocks to go along with their longer than average active time on the network next we have tier two validators they though they have similar characteristic as tier one particularly perfect proposal rates they have been active for a slightly shorter time and have proposed fewer blocks in tier three we see the trend continue where they've been on the

network a shorter and shorter period of time and tier four is where poor performance begins to manifest itself in actual skip blocks we can see new validators and those who have ski one or more blocks proposal in this tier then in tier five and tier six we start to see more and more skip blocks and lower current balance in particular validators in tier six have skipped more block assignment than they have success successfully proposed yet they have managed to stay on the network for a longest period of time and lastly we see new validator who

has been who have been slashed regardless whether they have skied a block in tier seven and with only a few months of data we we expect that these findings will continue to evolve and as such the tiers defining relative performance of validators will continue to need adjustment over time next we is our second submission on slashing title The Art of slashing to prevent attacks on the network validator can be slashed and ejected from the network if found to have broken a set of rules slashing means that a significant part of the validator stake is removed

potentially up to the whole 32 ether our analysis is based on 1751 slashings captured on beon scan and the variables are the row index the epoch number that the Valor was Slash the slot number that the validor was Slash the amount of time passed since the Val was Slash the index of the validor who was Slash the index of the validor who did the slashing and why slashing sounds pretty scary um but the likelihood of a slashing in any one slot is actually

very small on this graph we see one huge bike around 3,000 OT and another around 12,500 excluding these spikes there's approximately 63 slashers per 1,000 e which is not a lot and among these slashings um 97% are due to attestation rule offense and the rest are due to proposal rule offense and note that most of the proposal rule offense occurred after 13,500

eot here we have a simple distribution of the number of slashings received by a validator so you can see most validator only get slash once or twice so it's very surprising that EV valuator was Slash seven times and that's the highest number where one was Slash and similarly sorry and similarly a validator who have done most validator who have have only done slashing once or twice and eight out of these top 10 validator who have done the most slashings turn out to be in tier

three uh according to our ranking system the highest number of slashes done by valid was actually 93 the nature of slashings allow us to visualize the slashings we are directed graphs where each node represents a validator and directed Edge represents the act of slashing and interestingly no validator get slashed by the same validator twice so we have SIMPLE directed graphs and also we have this star pattern that we can

see over here on the on the right side and it's also very interesting that there was no Revenge slashing and thus there is no double headed arrow in our graphs it's all one it's only one arrow pointing to the other as the network of violators continues to grow we expect the number of interesting subgraph to grow with it and represent some interesting Dynamics in terms of the interaction between validators as it pertains to slashing and if you would like to more details on our submissions please reference our articles and to end this presentation I

would like to talk about what our next steps could be we would love to automate the data collection process to create an easy toanalyze data set we've improved data sourcing will be able to expand the set of variables we use to characterize the performance and finally we're ready to create a visualization and early warning tools that would flag po performing validators improving the overall security of the etherum network and lastly I would like to thank the organizer for putting this together and everyone for listening thank you does anyone have any

questions I I I do have one and I always have to acknowledge it might just be my naivity how does a validator get slashed more than one time does that is it um getting slash maybe it wasn't a a huge offense I would guess maybe just some small stick they is cut off but maybe Lawrence can can explain more yeah so um not exactly sure why they're multiple slashings but it does it kind of does make sense I think that it's just either one violation of the attestation or the proposal so maybe they submitted multiple blocks so they've been assigned to their slot

right they submitted a block wasn't valid they got slashed and punished forward and then maybe they had enough sufficient balance so that the balance stays high enough so then okay they get a next shot at it they propose another block boom they get slashed again so that's how they can probably accumulate multiple slashes okay that's I I was under the Apparently false impression that um uh a slashing immediately put you in the exit CU so I guess that's something that I will need to learn more about maybe right we I need to look at what exactly is happening there but you can only actually be slashed once but um suppose you part of uh suppose you're

part of an aggregate atation that has you a tester one and a tester two and then another aggregate ass station where you're it's you a tester three a tester two and a tester three um both of those could be submit on chain although you would only be slashed once because there was um additional information in the second one but uh it might might depend on kind of little things like that but ultimately you only get FL one you know what you can do Danny you can also look on Beacon scan because they have the SL slots um I guess that Tab and you can actually kind of verify that some of these some of these yeah some of those validators have been slashed more than once so even though it probably

shouldn't be a current you can clearly see it in the data I was I was I was curious about the validator tier list um because um if a validator just joins joined recently he will have a higher balance than a validator that has been validating for since Genesis and maybe he wasn't offline so how does that weigh in into the tier list do you guys did you guys um calculate it in when yeah um SE can you pull up the slide really quick because it actually is highlighted in tier number four let's see if you can it yeah the slides yeah you bring it up

34:52 - Nate McKervey and Antoine Toulme from Splunk

yeah yeah um next slide next slide yeah so your question Buddha so if you look at the very first one in tier four that's basically a brand new validator right so at the time basically like 60 F I think that's what it is but they've had no executions and no skips so they basically haven't even been sign signed to anything yet so the way the scoring ranking says that you know what we can't we can't give you a high score yet because you haven't proven yourself but we also don't want to dump you so we throw you in the middle we throw you in the middle of the uh military list which is kind of cool like this was actually a result that came after the fact that we were doing took a little bit of

calibration but after we fit around with the waiting we came up with something that was really nice so basically you start in the middle roow and as you get better you rise in the ranks pretty cool eh thank you yeah awesome great um and so if if you will disable screen sharing um thank you so much uh you guys your submissions were very interesting I was a little worried um it would they would be over my head and I wouldn't be able to grasp them but I I thoroughly enjoyed that thank you so much um you're welcome yeah so it was great uh next we're gonna move on to Nate mccurvy

Antoine tme from Splunk um looking forward to seeing what you guys have to share all right and I assume everybody can see and here yes all right great so our goal was to bring together like a really wide variety of data and then use that to investigate monitor alert act on it and so in this case what we did was we brought in data from uh three different clients so we had prism Lighthouse and a a teu archive node and then each of these clients is generating multiple types of data so has we're bringing in off logs system logs Beacon chain logs

validator logs as well as system metrics like CPU Network and disk then we used a couple different methods for data uh regarding nodes that we didn't have access to or full access to so for example we're scraping E2 stats and that provides data on like version heads sync stats memory usage perer count and uh and for prism and load star add stations but not all nodes are exposing this data because it's an optional flag so to get more data we use we uh customize chain D to send data to Splunk instead of a database and so this gets information from the beacon nodes including proposer duties Beacon commits validator blocks and blocks which those blocks include

like the structure the attestations proper proposer slashings attest attester slashings and vol exits so there's a lot of data there but then we also added the Splunk connect for ethereum which is an open source um connector for getting data from Ledger and nodes uh and on this case uh we we were getting data from gry Network including blocks transactions events and metrics and the nice thing here is that this little connector will also derc serialize transaction events if it's given in a contract ABI and then it also performs contract fingerprinting to match the abis with function signatures and then automatically decodes the parameter names as well and then there's a bunch of other data sources that aren't listed here like uh from Discord

data and so on but the tldr is that there's a lot of different types of data that when you combine together can create really rich analytics um and so what we did was we created this uh operations center which provides a high level overview of a bunch of stuff um and then each one of these areas you can you can drill down into other dashboards or the raw data itself based on whatever is interesting to you now I set the time range for these most of these dashboards to around the October 13th time frame since it was a time period where things started getting interesting on the regarding the health of the madasha test net um when it began to degrade a bit but note that all of this data is actually streaming in in real time so these dashboards if I set them to real

time would be animated there's no need to scrape data put it in CSV and then Analyze That static data set this is all all real time and so at the top we see just some basic statistics across the board um we can even see like the top graffiti here so this is you know some of that automatically converting like the input hex into something that's human readable so if you're trying to earn po app you'll actually see the PO app um information here but that's not really useful for health all the other stuff is is more useful for for health um and so this health score is calculated from a combination of the health of many other scores think like Genie coefficients online validators a bunch of other stuff and so pretty much

everything drills down so let's go ahead and take a look at a few of these starting with like the address Explorer and so this is just a basic dashboard just showing like the ad addresses the total transactions in the last 24 hours um a lot of 32 eth going to the deposit address which is what you would expect and then we can see that uh you can do like real-time Force directed stuff so lot of of people are sending to this contract address but then we can go and do like second level relationships what happens after that the thicker the lines the the more e that's being sent and all these you can drill down into the raw data itself so this is act the actual

um uh data from The Ledger uh and this this is coming from spun connect for ethereum so you can actually connect it to multiple networks instead of going to a different block Explorer for each network uh you can just send it into one place and search across it so I could send die to the XD Network and search one address and I'll see the transaction across layer one and Layer Two or you can connect to eth one and eth2 um and search across them so I just put in the the eth2 deposit address and we'll just search for the number of transactions last 24 hours and there's been yeah 1,752 events with a bunch of different fields and you can see that the majority

of the deposits are 32 e but there's other Stu going on uh and so if we go back to like the lower left quadrant here you'll see like a version of E2 stats um that we recreated and and enhanced a little bit so we can see all the different types of client types how many nodes are online the versions of those nodes are they syncing or not remember this is from I think October 13th yeah and you can see the locations of these nodes and if some of them are having issues they'll they'll show that they're offline and then you can see very I information about them that's being exposed their number of peers their history attestation history and this was kind of useful because even my note one of my nodes I was running prism node and it

started having issues um and I was able to debug it because we're bringing in all that data and I could see all the errors and warnings and what the what the root cause was it turned out it was a a bug that was introduced in uh the alpha 29 release um which I upgraded around October 13th and so did a lot of other people and the alpha 28 went down and Al 29 went up um and they the Prismatic Labs people made a hot fix and it was it fixed the issue but most people waited everybody knows the story I think by now so it was kind of cool because what happened was I saw that when I even when I upgraded and tried to get my node back online I started I ran out of memory um and then I just increased my VM and salt and then it

started working again but it was nice to have all those logs in one place and so then we thought it might be useful to show the memory usage by the different version of the different clients and you can see like which releases are using less memory and bu other stuff and then you can do slashing analytics too so we can see who's been slashed the most and uh since a lot of people are requesting gorilla eth on Discord we just do an export of the uh addresses people are requesting with their username and then you can start to associate who is doing what and you could combine this in force director graphs to see to really get a lot of in need information and the slashing information you could also do Force directed live in real time and we did a lot of Genie and Loren

stuff too um so the we want this obviously to be as close to zero as possible the genie coefficient so we have the most even I guess distribution um everyone's probably pretty aware that there's a lot of prism nodes out there and if there's an issue with this then it's we could have problems getting finality because I don't know there's a lot of different reasons you wouldn't want one node to be overwhelming but what's pretty cool is this is arbitrary you can create the uh dep the genie and Loren Genie coefficient Loren curves on any arbitrary field we have a simple macro where you just put in the name of the field here and it'll calculate the luren curve and the genie coefficient so

like literally any of the this information I wouldn't really like you could how many peers are connected you could just put in peers here and it automatically calculate all that yeah and so we detailed this out on our blog it was a lot of fun and that's it that's a very impressive uh project and dashboard um can you tell us how how I can get the dashboard on my wall if I unmute myself yeah so everything we did was um using like data that's out there and

Splunk you can download the free version of Splunk um uh we put some information in oh I'm not sharing my screen anyway we put some information in the blog post there's a Docker compose example where you can start getting all the data off of um I think yeah by whatever ethereum compatible network you want to connect to um and it'd be cool to collaborate and build these dashboards together like just it's just like Jon and XML and the whole Community could run their own um instances looking at the data setting up alerts too so like you can automatically send alerts to slack Discord mobile phone email based on whatever happens and I'll be honest as as an end user I'm not I'm not obsessed with

collecting all that data locally but I am very interested in being able to pull up that display and seeing the health of the network in snapshot without having to do all of the labor of managing the back end uh so that's really exciting to me we just released um the ability to publish dashboards so and so we're publishing one here if you go to MDC madasha data challenge that Splunk DLT you'll actually have access to uh that dashboard um and it drills down into our live instance so once you click on this it's not like you're going to get a login prompt but yeah you can we can the community can build and publish these down

too sure and you know again as a home user this is really what I would like to mount a screen to the wall and just have up all the time um just so I can wake up in the morning and see how things are going so yeah and I want to do that specifically like with my my nodes that are staking too I want to know a lot of information about my nodes like are they being attacked or are they uh being offline or lots of lots of invalid login attempts or even the service maybe I'm using some sort of staking service and I could monitor them um how did you guys get the count

for the prism nodes and the lighthouse nodes because I I think you showed um statistics that from E2 stats right ah okay that wasn't that wasn't a crawler okay that is correct we also crawled U but this not been released with the new version of Discovery V5 uh we're not able to to crawl uh right now but we did crawl we um we fought up to something like 800 nodes uh over a period of time uh but that is um not a definite number because you can have notes that you know would come out and come back into the network um happy to pass on to you or CSV with the with a note we found

yeah because U we have a similar product or website where you can see the notes and it was not working properly that's why I'm [Music] asking all right well it was great uh thanks for presenting guys uh next up we have Elias who works for bison trails and Sid who works for coinbase they'll be presenting eth2 data. github.io so with that go ahead and take it guys thank you Michael and thank you everybody let me just try and share my screen

57:06 - Elias from Bison Trails and Sid from CoinBase

here um all right here we go oh can everybody see or not really uh I don't think so yet I can't see your screen uh give me a sec okay Sid can you can you try and pull it up on your end because I'm getting uh yeah let's see if this works yep can you guys see yep awesome all right um so I'll kick us off

then Sid Sid will take over for the first section and then uh and then I'll close it off with the rest of it so I'm I'm Elias uh I did this work with Sid we had worked uh in the past uh together when Sid was running the token analyst he was getting data out of you know blockchains I was I was doing analysis and so on so um we saw we saw the challenge being posted and and and felt that it'd be a good time to kind of like you know drum up the old uh token analyst um spirits and attack it together um super fun great experience um doing it and and and a super great opportunity for us to kind of like you know learn more about you know the inner workings of of of E2 and how the whole system comes uh comes

together um and kind of starting out we wanted to kind of do something fairly comprehensive that kind of both merges uh and and addresses to uh an audience that is not so familiar with these two and kind of you know tells a story through data of what happened in in in maaja um but also to kind of having that broad overview to maybe try and find kind of points where we can really um add value either through new metrics or uncovering things that were not visible you know at first uh at first glance so um I'll pass it on to Sid to walk us through the first kind of pieces of of uh of our research and then uh I'll take

you back on yeah thanks Elias yeah just as a short intro yeah my name is Sid I currently work at blockchain research at coinbase and uh yeah it was absolute pleasure working with allias to dig into some of this data um uh shout out to uh bua and the beacon chain team too for providing us a lot of this data so that we could uh we could you know get the ball rolling so I mean yeah as mentioned before uh initially our goal was to run all the major uh node clients and essentially get a picture of how they performed and also uh pull data from them uh in order to actually do this analysis uh however we found out that

that feat was uh you know quite extensive uh mainly because a the node clients had slightly different data sets when we took samples uh and just formatting them into a normalized format for analysis is is in in and of itself Quite a feat so um it proved to be pretty challenging however we did see in terms of performance uh so just for context we ran all uh you know Lighthouse prism Nimbus and teu on uh you know pretty large instances VMS on Google Cloud uh uh where we also ran a GU node on the same VM in order to make it as you know close to a optimized validator setup as possible uh and uh saw that Lighthouse was you know by far

the fast sync in terms of sync times um and and we we we didn't notice anything substantial in terms of differences between U memory or CPU usage uh from our trials uh so just that was a short takeaway from our experience with the different clients and uh um we'll we we'll go more into depth in terms of the clients when uh we discuss uh grouping so this is part of the analysis that we really tried to hone in on um essentially we wanted to get perspective of who's actually participating in in the test net uh at the given point in time and and what are the types of entities that are running validators and so uh there are several points of data on chain uh and offchain that we could

use to you know group validators together into clusters or into entities uh starting with you know the validator index AK it's the equivalent of the public key for the validator the withdrawal credentials uh which is essentially related to uh that validator index which can be used to withdraw and can be mapped to multiple validator indexes uh and then the E1 deposit address that was another data point that we used where uh if an E1 deposit address deposited into uh the contract uh and could then be mapped to uh potentially multiple validator Keys as well and then finally uh with the poop rewards that came with broadcasting graffiti for signaling what kind of node

client you are running uh that was also a very very helpful indicator for us to essentially use graffiti uh to uh Target and and label what kind of node client a person may be running so we used all these different factors and we were essentially able to Cluster together validators into into different sets of groupings uh one set was what types of clients were they running uh uh as you know a lot of you already know a majority of the clients uh that we were able to tag we're running prism uh followed by Lighthouse and then uh some of the others uh and then uh Additionally you know perhaps more interestingly uh we were able to kind of narrow down the groupings in terms of uh what type of entity they were into

largely around 22 groups and so these kind of spread out between uh staking as a service firms you know infrastructure providers ethereum Foundation uh several several names you can see in the pie chart here if you can uh uh uh if it's visible for you uh you kind of pop out where like distinctly identifiable as you know belonging to a certain group and uh these were effectively being uh you know mapped to multiple validator indexes which could then be derived uh to represent what their performance was how did they behave on chain how do they attest how do they vote and how do they propose and all of that is stuff that uh we dug into and allias will'll go into

in more depth later on so grouping was a big part of it and big picture I mean uh all in we took in the network as a whole uh to see you know when it was healthy and when it's not what's Normal and abnormal in terms of uh we we provided a definition where if three4 of the block slots available were proposed we uh essentially deemed it as a normal Network condition and uh you know one of the key takeaways for us uh through this time uh as some of you may know there are kind of two periods of abnormality uh one between you know EPO 2000 to 4,000 during the rough time incident and then around Epoch 13,000 13,400 uh where essentially both of these periods um were marked by client

syncing issues uh you know uh with with uh you know with prism in the first uh rough time incident and then in the second one uh around UPAC 13,000 with with Nimbus and so uh that seems to have been from our data set the the the kind of uh led to the majority of proposer slashings uh where all the slashings that we saw were attributed to double votes uh and uh not exactly to some of the other qualities such as like for surrounding voting that uh are responsible for slashing so that was an interesting takeaway in terms of big picture and uh in terms of Network Health we also defined a series of different metrics to essentially Define

Network Health as you can see in the first graph you know missed over proposed blocks there's these distinct two periods between 2,000 to 4,000 and then uh in 13,000 you see the orange uh bit dip a little bit uh there's kind of abnormal outlier periods and then um in the second graph uh we wanted to map out finality so finality is defined as essentially when two-thirds of uh you know U of the active validators essentially are included on chain uh the the line over here you see is is what we defined as a finality threshold and then we essentially mapped the lags the distance the epoch lags for when uh active validator votes were included on

chain to essentially see what's the threshold and and you know how how efficient was the network as a whole in terms of including valuable attestations on chain and this graph really gives us a big picture view especially during the rough time incident you can see you know a significant dip there where you know less than 15% of the finality threshold was uh you know included on chain so it was very interesting to see uh and then finally yes aggregate inclusion DeLay So when uh you know you know it's essentially mapping out this inclusion lag uh over time across Epoch uh so these are some of the big picture metrics we use to derive Network Health uh but then uh perhaps more interestingly Elias can dig into some of the uh you know attestation specific

metrics super thank you Sid uh so the next bit that we uh we took a look at was was the performance of aggregation we wanted to get into the PTP layer but quickly realized that given time and so on we wouldn't have have enough time to crawl a meaningful uh sample out of it and so we then went on chain and disaggregated a bunch of group data stations to figure out what actually is going on and we found that uh there over the course of meta apparently according to twice as many active validators per Epoch uh were uh uh uh included uh so twice as many stations were included on

chain as activated or active validators uh per Epoch and you know this visualization here on that stations uh included versus active validators kind of shows you how that panned out um over time we initially called it attestations blo but then kind of like you know we discussed it with with farious people in in the community perhaps it's a little bit uh too aggressive as a term and so we called it Surplus um but then you know we didn't have a lot of time to to get into actually why that's happening and whether that's bad or not because some of it like you know Danny described before is you know totally expected um uh but uh you know we did a little bit more research some of the uh the results

uh We've recently published uh on the Bison CH blog um and and we actually found that there's a lot of singles being included on chain calling singles at stations that uh are composed of only one validator index that Target the same slot that arrived at the same time and yet they were still uh included as multiple uh different ad stations and in fact we found that the mean was around 22 um only looking at a sample of the first 1500 EPO which as you can see here kind of like the Surplus situation was a little bit exaggerated but you can you know extrapolate what that means for uh for the rest of of of the test net um uh by by looking at kind of the Surplus uh

that we observed here in aggregate go for the next slide s you with me thank you all right we also looked at uh proposal Effectiveness and trying to figure out um how to best sort of score a validator for how well they they perform as a as a proposer um we understand that roughly kind of for every one time so the actually the rewards from being a proposer are about like an eighth or between an eighth and a tenth of the rewards that someone stands to

earn as as an attester so uh while it's not like a a big chunk of of the rewards uh that a validator should expect it's it's actually pretty important because on on kind of like the nominal per unit basis the proposer uh reward is actually significantly larger than the each of the attester uh rewards and we kind of came up with a methodology wanting to uh control for uh both the time active in the test net and also the probability of proposing at least once kind of trying to capture a little bit of the difficulty um with more validators having entered the test net over time um and and came up with uh kind of an interesting methodology which you can

find uh in in the report um and then through that kind of you know blowing out one of the findings when we uh segmented by by client type we found that roughly uh while the uh in the slots proposed per validator the the client sort of distribution was very similar to what we observed on on aggregate when when we looked at slots missed we actually found a higher proportion of of nimbus and and and Lighthouse uh uh validators uh uh being uh exposed to M slots let's go to the next slide we also looked at uh a tester

Effectiveness and kind of tried to simplify things a little bit here of of how one goes about you know uh scoring for a tester Effectiveness and we thought that kind of looking at the aggregate inclusion delay uh over an a tester's lifetime and the uptime ratio um is probably kind of the easiest and and and more dense in information way to uh come up with with with a metric like that with uptime being sort of the moderator of how many times you um uh you participate uh in attestations uh the maximum number being one every Epoch for which you're active and the aggregate inclusion delay being the moderator of how much of the max reward

you stand to get as a uh as a validator so the longer you take to to to get your vote included the less of a vote uh you get and and and these are the results here we also looked at um the valuable to nonv valuable attestations ratio so how many Surplus attestations each of the validators um uh uh got included on chain we're kind of I've got this crossed out here because we're not entirely convinced that this should be a performance metric for an attester because you know the fact that there's Surplus stations could be uh uh one some of them are totally benign and expected two uh some of them you know it's not it's probably not the mistake of or or

or or probable cause doesn't go back to the uh the validator node but maybe the aggregator or maybe the proposer um but we you know we included that for for reference to just like create a little bit more of a of a richer uh data set and this is what uh what our results uh look like so you know a lot of a lot of people it being a testnet and and not having real money at stake you see that there's a very highly dis proportionate um uh segment of ineffective ATT testers and then like a roughly normal distribution um with with still quite a few lards and and and actually quite a few standup uh um performers next

slide and then we also kind of mapped uh how the score uh mapped uh against the uh performance on rewards we only had data available for kind of the uh the performance uh uh of of a validator index overall in the testnet and um as such we only in this view we only took the first 20K valers that were active on Genesis so that we uh we have a clear and not not biased view um and we found that it's actually a pretty good uh predictor about 85% uh correlation and so on uh I think we have one more slide and then we uh we also segmented by uh

client type on the tester Effectiveness score and also you know all the other scores like for uptime ratio and for for inclusion lag and so on and you know found that uh prism prism and Lighthouse um are probably the most popular clients uh for for a reason um and then we're looking at the client distribution among the top 30 operator groups so kind of going through the same exercise size but you know instead of taking clients we took the operator groups that Sid talked about uh earlier we found that kind of a crushing majority uh at least from the onethird of the whole that we were able to map to client Choice uh where uh we're using uh

prism thank you um that's us yeah um I don't really have a question but I just wanted to ask you or tell you from the community s if you could do the same thing um after a month after we launched on the main net because yeah since we had so many issues on the test Nets and so many releases the data is um yeah not not not me meaningful is not the correct

word but uh I think there will be will be more valuable on the main net and if you could do that that would be great sure sure thing and and by the way like all all our code or most of our code is on is on GitHub um so if you go to our submission there's there's a link that links out to a repo with all the notebooks and all the raw data for anyone that wants to kind of like take modules or replicate the analysis and so on but we're more than happy to uh to go take a look again yeah maybe one question how did you guys get the client types the prism and the light was probably from the graffiti or all right okay yeah yeah basically from the

graffiti was the main source uh from the poop identification from the graffti okay yeah many people cheated on that so yeah yeah I'm sure valuable data yeah thanks no no I'm not saying that the data is not meaning is not meaningful I'm just saying the fact that people um Have Cheated but in combination with other stuff like um the up time so if someone had a load star uh graffiti and um didn't have any or wasn't in the or if someone has a lighthouse client and uses a prism graffiti and we know for a fact that the

prism up time is very High the likelihood of him cheating is very high if that makes sense I can repeat if you want me to yeah I think I get what you're saying um we tried to group it where uh it's not just graffiti wasn't just the only characteristics but also other things right um like validator you know withdrawal Keys public Keys eat one public keys so on and so forth so uh there were cases where you know we saw guys trying to spoof where they just put you know random pups on their graffiti just to uh try to get rewards so try to exclude those from the

analysis I think it's the case of there when there's no perfect data available you use the best thing you can and try to make sense of it and I think I think you all did a really good job of that yeah so even we as Explorer had wrong data so I'm not blaming you guys at all I know it's hard we we we know it took us it took us like two two or three days to get like the correct data out again but you know props props to you for kind of being super responsive and and and on top of this um yeah so we we had since we only had one client running which was prism um there was no no chance for us to actually um

back test it with a different client but that will be that will change soon uh with the new API with the standard API and then I think all the challenges and all the um analysis will be more accurate all right uh thank you all very much um for our next presentation we have blockblock data um you can find them at blockblock data. github.io we have gerdle ertek from the hello B Sloan ketering Cancer Center in New York USA so we're looking forward to hearing from you all they have three uh

1:19:15 - Gurdal Ertek from UAE University and Joseph Kholodenko from Memorial Sloan Kettering Cancer Center

consecutive presentations yes can you see the can can everybody see the screen and hear me correctly yep yeah yes I can see and here all right all right everybody Welcome to to our presentation and uh this is for the Medela data Challenge and this is our analysis by our team our cons of two members I'm I'm gal Ark and I'm a business professor at College of Business and economics at UA University in Abu zabi United Arab Emirates and Joseph Joseph heno is a data engineer at Memorial slon cing Cancer Center in New York New York and we will be presenting our three uh articles here article 01

002 and 003 so those are the three articles that we will be presenting so what are those articles the first article is about the ethereum to beacon block so this is a visual documentation of uh the ethereum 2.0 beacon block which is basically the most important um most important uh the block on a most important chain in in ethereum 2.0 so that's our first article and then our second article is analysis of uh data propagation latencies and our third analysis is clustering and profiling of time EPO in ethereum 2.0 so again this is our tee

and we'd like to thank everyone in the community and uh everyone who helped us as well for this project so yeah let's start with the first one uh and I'll try to finish each one of them in about five minutes so I'll try to finish everything in about 15 minutes all right so what is the first research what's the first article what's our first contribution the ethereum 2.0 beacon block a data schema and visual documentation so what we did here was we provided a visual documentation and even more important than the documentation itself we came up with a way of documenting such blockchains and uh so it's not only the specific documentation that we did for e

e 2.0 but the generic methodology that we uh developed which can be applied to pretty much any any blockchain so our focus is the beacon chain and a block on the beacon chain so the CH the chain of course consists of blocks so we focus on data on a single block and we focus on the data rather than the metadata rather than the data itself so rather than you know the whole data we focus on what is the data structure what is the data schema and uh since a picture is worth a thousand words our approach was visualizations and especially interactive visualizations the first one is 3D visualization of architecture the second one is data schema diagram again

interactive visualization and the third one is a zoomable presentation and we did the documentation using a dbml database markup language using actual data from medel so when we look at the content the first one is this uh threedimensional visualization of the architecture so it's you know it en enables you to zoom in zoom out and then you know play with it so you can just uh see and focus on areas that you're interested in okay so of course this could be used for visualizing any blockchains architecture the second one here is the uh interactive uh database schema and we also put some simple

instructions so you just click on the image and hold over fields and links so you can just hold over the links um and then so you can see the relationship between the different fields or we can also look at the relationship between the uh specific tables and this is the on the left hand side this is the dbml code right there okay so that's second interactive visualization and you can focus and then you can look into specific things all right and the third one is the um is a zoomable presentation that we created using prey of course we used as you can see we used the screenshot from the earlier one so here this is the interactive

visualization so let's maximize this and then we just it just focuses on every single component of the data and it also provides actual data as you can see it provides actual data from the med medha test net as well all right so the beacon block proposal slashings and then all the other different uh different components different areas of tables the groups of tables all right so that's those are the three visualizations and then later in our paper of course we also went into the detail of each one of them so for each table we describe what is in each table and then uh for each one of them also we provided uh the

database structure including uh including one to one mappings and many to one mapping and other one to many mappings and then also actual data from medala medha test all right so that was our first contribution and our goal here was to flatten the learning curve you know many people developers Auditors validators data engineers and scientists they need to learn about e e too and how can we make it easier to learn there's wonderful documentation out there and can we just make that documentation easier to digest if you will uh through the understanding of meta data and this I believe I mean as far as we could tell this was the first uh time that we have

seen all these different approaches being used uh to describe metadata in a blockchain in an interactive through interactive visualizations and it's applicable to in blockchain all right so that was our first contribution now our second contribution is an analysis of data propagation latencies delays basically in the ethereum to medesha test net and here uh we appli not only visual methods but we complement the visual methods with with formal statistical analysis because I mean especially if we're talking about let's say differences okay let's say we have one group of one sample and we have another sample and you know the means

are different uh but then if one of them is higher has a higher mean than the other one is that really you know is that really you know just when we look at the averages is that a conclusive answer no we need to do statistical analysis because you know the difference may be due to some statistical variation statistical due the randomness okay so that's why statistical analysis is important in our opinion that's why we implemented that and then we came up with some actionable insights as well so uh we started with the medesha test net data and Mr Jim McDonald kindly shared that with us and it was a post post SQL dump and we basically had um some SQL

queries and we went through certain steps and eventually we obtained these data fields and we find inclusion distance for each slot as the maximum inclusion distance over all attestation aggregations and over all committees that participated in the voting for that slot so this is like more like equivalent to maximum inclusion distance now we know that in some of the other entries uh for the data challenge the definition is different even though the term is the same so this is our specific definition of the term and uh it is actually a metric for plays in the system okay uh so but then even if you know one prefers another definition the

methodology that we will present is still applicable to that uh to that other definition of the metric as well so we start with univariate analysis and then uh we we we had a distribution of uh average inclusion delay and the average inclusion delay it's of course it's a bit skewed because it's there's an upper limit right there's an upper limit of 32 so this by way what is this 32 this is the number of slots okay so uh so this is number of slots but then this is per Epoch so per per Epoch what is the average inclusion delay in terms of slots and now we have a similar uh histogram and distribution fit for uh

for uh the number of missing blocks as well and turns out uh the software we use suggested that beta distribution would be a good fit uh for both both of these all right and next analysis was looking into correlations because correlation will uh motivate us to come up with the hypo hypothesis and to see which pairs of uh variables which pairs of attributes would be interest interesting to analyze together and here we immediately see the uh higher values here here um here here so those are definitely worth investigating all right and when the first analysis that we had was I mean the next analysis we had was patterns over time so over EPO now again

when we're measuring inclusion distance the inclusion distance is measured in terms of slots but then uh the analysis that we have is over EPO and as our colleagues earlier mentioned we ignored we removed uh any data from uh before Epoch 5000 because of the mentioned uh problems in the system uh in the blockchain network but then we'll focus on anything after Epoch uh 5,000 of course we couldn't include the latest one because we were still working on the data at that time we already had the data plot that so one of the analysis that we did was voluntary exits how many voluntary exits do we have in each Epoch now the maximum number is 22 and then we

also see that there are no exits no exits no exits and then boom suddenly there is like a you know narrow time interval where there are many many uh voluntary exit so what's happening there especially it's very obvious here right you know in such a short in a few EPO so many um so many voluntary exits happened so what is that what's happening there again we don't know what happened but this is something that points us to that direction now the first conclusion that uh we get actually is that the average versus standard deviation of attestation inclusion distance are inversely Associated as we have this as we can see from the scatter plug when we plug the uh inclusion distance standard deviation

against inclusion distance average we see immediately the inverse relationship oops sorry I couldn't throw very well so okay let's try again okay yeah there's there's the inverse relationship and of course software we use also gave us a nice fit for a polinomial function so we could express the relation as such as well all right then the second another analysis uh which gives us our second uh result uh is inclusion distance versus commit size okay so again inclusion distance basically propagation delay uh how does it CH change with respect to number of committees and then turns out it first decreases and then it increases

okay now again coming back are these statistically significant differences or not to that end there's a well-known test uh in statistics called uh dun test and this will allow uh us to make multiple comparisons and there is Bon feron I think this is yeah there is a bon B Boni inequality that we uh also use uh to come up with the correct Alpha values anyhow to to you know to return back to the Practical aspects uh so there is there's a difference as you can see you know it seems to change over number of committees so depending on the number of committees the inclusion

distance seems to be affected but is this statistically uh supported and the answer is yes so for example what does this mean this table us gives us the results of the dun chart and we also visualize the table over here so what this says is this for example whenever the uh number of committees is 256 which is this case uh the mean inclusion distance average is higher than 288 and yes it's statistically supported so one means it's statistically supported if it's a pink one it means it's less than but it's still statistic supported but what about the difference between this and this for example so here it's zero so it's not statistically support even

though there's a difference so since this table is a bit hard to read uh we applied a methodology that I I mean we had developed with with another group of researchers I I was part of that research team uh we visualized it as a graph and we had a hierarchical layout so this tells us that whenever the um committee size was 352 and let's just calculate what that means so committee size over an EPO being 352 means and there are 32 um slots in an Epoch so that means um if in a slot the uh the um the number of committees in that slot was 11 that was the lowest that was the lowest inclusion

lay so that's when the uh propagation delay was the minimal now this is not saying that you know the committee size should be 11 because why because the committee size the number of committees I'm sorry the number of committees should be 11 is it uh saying that no because the number of committees the committee size the epoch they are all related to each other but this is definitely telling us this commit size is associated with average inclusion distance and another result that we obtained was with regards to voluntary exits so when whenever we observe um voluntary exits okay so if the voluntary exit is not zero but if it's greater than zero so we observe that inclusion distance average uh is higher right it's

higher than the average so so uh we did formal statistical test uh first we we tested for uh param parametric U we we tested for normality and we decided not to use parametric test and then eventually uh we applied man withney test to come up with a P value of so B less than 0.01 so basically what this says is voluntary exits are positively associated with average inclusion distance in other words if there are voluntary exits in a in an Epoch uh there will be uh more propagation delay all right why is this important again it helps us understand on chain data we we adapted formal

statistical methods to this domain and also we also came up with a software stack I mean all these is they were not done you know only with python or R we Ed many many different software and we also provide definitions and descriptions in a very concise manner and again the methodology is applicable to other metrics and other onchain data as well now the third analysis is clustering and profiling of time EPO uh so here the clustering is on timebox not the not the validators and we both cluster them and we profile them so our is something like this remember the four seasons of Vivaldi so now we have the seven seasons of madesha and we again

applied Visual and statistical analysis interactive three-dimensional clusters and then basically we found that there are unique clusters of time Epoch and these clusters are can be profiled and there are many implications for future research how many clusters should we have we we came up with that using the silhouette score and the best number was nine but then eventually we used seven because some of the Clusters had too few uh observations so and then we have a interactive visualization which you can go ahead and analyze you can just interactively analyze how the Clusters are mapped in these three dimensions and then uh we also analyzed um these

cluster we profile these clusters with respect to each other and then uh we see that for example cluster two has the highest inclusion distance and then statistically significant here also you can see that and then it also has the um highest number of attester slashings uh and then uh second highest number of mean voluntary exits and then so this this this is basically this cluster of time EPO is where there are significant issues all right so this is the first research of this kind that to our best to the best of our knowledge because we used machine learning of unsupervised machine learning and combined it with formal statistical analysis for uh time EPO that's what it makes it different

and it opens many many issues uh so what did we learn from this whole project number one cancel everything during the data tone because you're not going to be able to find time for anything else set up validator NOS on your own don't wait for data from others because you may never get it but eventually we were very lucky and we got it uh so engage in each Staker uh Discord Channel share ideas share data share everything public early on your results so you get feedback and ask people don't be shy and ask for ready data there are amazing people who are ready to provide that data just is in our case Mr Jim McDonald did that and understand architecture mechanics and

metadata first rather than you know jumping into analysis this is what what we will tell to ourselves in the future okay start small and then increase in the analytics maturity Target visual analytics first and use a toolbox rather than a single tool of python because implementing everything in Python may be too too too much time consuming and the data tone basically calls for Frugal data science so here are the slides that support these Lessons Learned this is the these are the um stages of analytics maturity as suggested by Gartner and then uh this is which visualization can be used in what type of situations and what type of purposes and this is a more

comprehensive one and then this is the software St that we used for our second analysis and the third analysis was also similar and this is Frugal data science again it's best to apply these principles uh in uh data tones which we did and um this is also one of my working papers with a colleague so hopefully we'll we'll submit this soon all right that's all thank you so much and that's all from uh my uh from our side thank you so much for listening to us thank you wow nice thank you um first of all where can I find the slides slides uh please drop me an email and we' be happy to I'd be very happy to share um my email is my first and last

name with no no space no. nothing gmail.com okay and we are we are planning to put it on the internet for now though I can at least share with you on on email yeah all right um yeah like the the the part where you talked about the inclusion distance in versus the committee size was something I wanted to do with my team but we had no time um but I think I'm not sure if you talked about it or if you mentioned it at all but do you see a correlation of specific validators so I have to back go back little bit so the tations themselves um I'm pretty sure they're also referred as the sharts that we are

going to see in the future so the sharts that we are going to have um in phase one so the one committee is representing one shart and if we have those sharts in each of the epoch um is there any correlation of having a bad inclusion distance if you are in always so if you're always in um committee one does it mean that committee one um and committee one has a bad inclusion distance is it is there a correlation between the inclusion distance and the community or is it correlated to the validators okay I I think uh it's uh I as far as we could tell again we we we we learned everything in these 40

days we didn't know anything about ethereum to you know we just we just dived into it and we tried learning how to swim so the way we understand is that um the assignment of validators to the Committees is random okay so I mean uh so I mean it's not something related to the index of the uh committee but it could be the number of committees you know how many committees are in each slot that could that could have an effect also especially when it comes to uh you know e to phase one when the sharding The Shard chains are being introduced yeah that's that's definitely the analysis that we implemented here could definitely be um used for for that

I mean it's important to look into number but then again not only the number of committees but also you know the committee size and Epoch so maybe that was down because something else happened in the epoch not necessarily due to the committee size right yeah I think we had to dig a bit deeper so if um a specific validator is always causing a bad inclusion distance um yeah yeah that goes way deeper but yeah thank you sure sure thank you for the question thank you so and thank you for your help by the way during our during those 40 days and nights you were one of those who helped us uh answering our questions thank you eat right yeah thank you no

problem thank you thank you I do have a a question and this kind of goes out to all of the teams really um it through all of the the work and the dashboards that everyone has created were there any was there ever any point where you guys the teams were like whoa that's that's really neat or or something unexpected happened were there any of those moments at all yeah sure for example I mean I I was personally very impressed with the other uh submissions and what they did and you know the depth of analysis you know the interesting insights that that was

1:26:07 - General Questions & Wrap-up

especially I was very impressed with splunk's u Splunk dashboard because I'm coming from a PM I I also have this side PMS Performance Management Systems so I I really like that dashboard it was very interesting very rich thank you yeah we had a lot of fun um our team likes to do our offsites at places like East Denver and we haven't been able to get together for a while so I said let's spend a few days and let's just do a hackathon this madasha data challenge is perfect timing for us so we just sat down learned a lot and had a lot of fun and the a surprising thing for us was as we were building that operation center dashboard we started seeing the health score go down we're like what's going on and we were sure

our searches were broken or something and then we ended up finding out then my my prism node like crashed a little bit later anyway it was just really fun that it was the perfect time for us to do this hack on because that's when the network started having issues so that was a really fun cool surprise um one thing we wished we had done was we were really excited about rumor which I just put in um me the chat but uh we didn't have the time to get it going to the level that we wanted to but that looked a pretty cool way to get a lot more data um yeah we we thanks thanks to everybody actually like all all the

submissions were great and and we we also kind of like learned a lot from from everybody else um the most surprising thing I think that we found was uh the uh the Surplus uh at stations or what we called Surplus at stations what would seem like Surplus at stations so you know our our pride kind of coming to look at the data was that we would find approximately like uh you know as many attestations as uh uh validators active uh in in in every Epoch and actually what we found was that it was you know towards the later stages when when the network normalized a little bit was was that it was twice the number right and and kind of like that raised an eyebrow because okay we started like

you know talking to the community and reading docs and and and reading the research and so on um and we we realized that okay some of it is uh to be expected by Merit of how the BLS uh uh uh signatures and and the aggregated proofs work but then also it it seems like 2x was was was actually pretty high um without having any yard stick to to to compare uh against right or or like like a really objective uh Rule and so we we're we're still playing around with a bunch of data and kind of like trying to figure out what what the ual sort of uh Truth uh is there and like getting even more granular into figuring out you

know what what types of uh uh uh uh events lead to that uh to that result all right do any other the do any other of the uh teams or moderators have anything before we kind of close up here I want to say it was really nice to have things like chain out there to really like investigate whether we were what we were building was working and was right and so thank you Buddha thank you so much really nice experience thank you awesome well before we close up here uh for those of you who uh tune in and

are watching this on a recording um we do want to ask to subscribe to the E Staker YouTube page uh by searching for E Staker all one word on YouTube by subscribing in addition to being notifying uh when the East Staker Community posts new content it also helps us with uh YouTube feature sets like being able to stream live events in the future uh so to support the East Staker Community please consider subscribing to the east Staker Channel today uh also if you're ever overwhelmed by the speed of ethereum news and want to learn more about events and programs like uh etherum ethereum study master or the collaboration groups or just uh information in general all that more

can be found at the E Staker Reddit community at reddit.com SLR Staker also our e Staker Discord server at invite.gg eaker uh we're just a group of exuberantly friendly people who are passionate about ethereum we welcome all ability levels to join our community to learn and have fun so with that uh I personally want to thank all the teams for uh one joining this call and and presenting everything but two uh doing this actual work uh extending off of super fizz's comment as an end user this stuff is incredibly exciting because we can sort of remove

the layer of abstraction that is ethereum and put it into these real data sets and visual representations and I think it really humanizes the network and it makes it more fun it gets more more people involved that aren't just end users that they're data scientists they're people that maybe don't have the knowledge or skill sets to develop these tools but they know how to analyze the data that the tools represent and I think that's incredibly Great and Powerful for our community at large to help mature and and get to those next steps so uh thank you very very much for the the time and hard work and dedication that all of you have put in for this and I think with thanks everyone um

yeah thanks for coming out yep thank you very much take care by have a good one byebye thanks