Michael Sproul - Update on client diversity and blockprint

Transcript

all right I'm very pleased to introduce you to Michael Sprout Michael is a developer with the lighthouse client with Sigma Prime and he's going to give our next presentation hi everyone thanks for having me um yeah today I'm not going to be talking about Lighthouse but another project devours called block print um and this is a project measuring uh client diversity on the consensus layer and I've never actually given a talk about it so I thought this would be a good opportunity to sort of look back on the last two years of uh client diversity history and block print and present sort of what we've been working on and how that's going uh so client diversity is this idea that probably many of you are

already familiar with that we don't want a bug in any single client to cause damage to the network so ideally on the consensus layer we have five six maybe soon seven clients or something and we want the the split between these clients to be sort of Equitable enough that no single client has like two3 of this of the stake um because if a single client has two3 and then there's a bug in that client they can like finalize an invalid chain and if this happens it's like Doomsday basically like the validators that are running that client probably lose all of their e it's really bad so 2/3 is like the death threshold and then one3 is like kind of bad CU then they can stop liveness so liveness is like being able

to guarantee finality going forward so pardon me we need about yeah one3 per client Max um is the real limit and two3 is sort of real danger territory um so two years ago when I started working on block print um the situation was basically that we had these Network Crawlers for looking at the consensus clients and seeing you know what they were running and how these crawlers work is they basically just go around the peer-to-peer Network connecting to nodes and then saying hey what client are you running and the nodes mostly respond honestly CU that's the default Behavior like if you haven't hacked your client it's going to respond honestly and say yeah hey I'm running Lighthouse um so from these metrics which are still very important you get a sort of view of the

nodes on the network but you don't get a view of what the validators are doing so you you don't know how many validators are running on each note you might see okay there's a split between Lighthouse prism teu Nimbus but you don't know that there might like 200,000 validators or something crazy running on a single node so for the purposes of client diversity in the consensus layer the thing that's really important is the number of validators um so these Network CWS are still very important and they are still running today um and they also give us this sense of like Geographic diversity which is also important because if there's a if there's a network outage in the US or something we want the chain to be able to sort of limp along with the rest of everyone else around the world

um so so this was this was the sort of status quo as of September 2021 and we thought this was pretty good um but it turned out we could do a little bit better with blockprint so block print was born out of some other analysis that I was doing um I was looking at how the consensus clients were building blocks on mainnet because I wanted to understand the profit profitability of the block building process and whether clients were including redundant attestations whether they were um being as efficient as possible when there were like skipped slots sorry um so I was looking at all this data and I basically had a big Google spreadsheet of data about the different

clients and what I noticed was that there were certain patterns between the clients as to how they were doing things so uh each client basically has a characteristic fingerprint um and for the first version I basically just like eyeballed what the fingerprint was and I wrote this if condition that's like five lines um and this is really really dodgy but it mostly works like it gives this graph which turned out to be pretty much accurate so just okay we're back sorry um this is the client diversity on Main net as of um September 2021 so this is when I first uh came up with my estimate um and

you can see there that we're really close to the danger threshold for prism it's at 64.8 um but with the caveat that we didn't really know how accurate This was um but we this we this was like a rough ining we could see from the network metrics that prism was big um and then this sort of confirmed that yeah okay prism's really big and the other clients are sitting there at barely above a third all combined um so this this first version as they said was a bit dodgy it was like my expert knowledge that was sort of uh encoded in this program that manually looked at various things um in AI this is considered sort of quite bad practice to be relying on like sort of expert input to to build your system

it's much better to have something that learns and works out what the pattern is itself um and we didn't really know how accurate this system was as I said so we were sort of Flying Blind but I tweeted this thing out um you know on like in September and everyone took my word for it and then I felt like I had to then develop that a bit more and make it more accurate and actually really Pro to myself that this was valid um so with this in mind we um dedicated some time to building a sort of 1.0 version of block print that was a little less dodgy um and so we took an off-the-shelf machine learning library in Python called psyit learn um and it has a implementation of a Cane nearest

Neighbors classifier which is very much what we want to do um how this classifier works you can see there it sort of plots the training data in in an N dimensional space in this case threedimensional um and then when you get a new Block in what it does is it will just uh measure where that block is in the coordinate space and take the uh the five closest or the seven closest points and work out what the what the block is likely to be based on that so you can see um I'll go into what the axes mean in a minute but you can see there's sort of a clustering of purple and a clustering of pink which are Nimbus and prism Lighthouse is sort of concentrated up the central axi um and then there's uh teu in Gray is over sort

of near nimus which is why they were getting confused in the initial version they looked quite similar to the um to the machine learning model um and then on those axes there these are like uh various properties of the of the blocks so when we fingerprint the block we use a little bit of expert knowledge to um to like pull out some numeric value that represents something about the block so the Zed axis the vertical one is like a measurement of the block reward so you can see that uh at the time when this happened Lighthouse was doing well for Block rewards which I was very proud of and it was sort of a bit of an outlier on that Z axis um so that would sort of flag that something was lighthouse um the x-axis across horizontally was the

Redundant attestations um ideally clients shouldn't have been including redundant attestations but due to various quirks of their algorithms some of them were they're not doing that so much anymore but occasionally um and then the other axis was about the ordering because attestations are often ordered from most profitable to least so we were looking for okay is this client being very strict about how it orders the attestations in it in its blocks or is it being um is it being more loose and using some other algorithm um yeah so this first version was a lot better than the than the very Alpha um spreadsheet version um we collected training data based on graffiti um so we had a big collection of Lighthouse blocks prison blocks teu

Etc um we could configure the features which are those axes that we were using um and find out which ones would the most effective and for measuring accuracy we had a nice system of cross validation which is a standard machine learning thing where you keep a portion of your uh training data aside you train the model on the rest of the data and then you measure it uh how accurate it is on the remaining portion and you go okay did I get that right and you get a sort of percentage value of oh yeah you're 95% accurate or something often not 95% but um these are the features I'm I maybe won't go into this too much because they're kind of um there's a bit of this is like getting into the weeds but you you can see that

we've got a bunch of features um these are just metrics like you know you take the block you analyze it you pull out a number so the percentage of the attestations that are redundant is one example uh how sorted the rewards are from profitable to least profitable is often the sort uh the ordering um some clients also sort by the slot of the ad station um if they're including old ad stations sometimes they'll put them at the end or sometimes they'll put them at the start um and a few other things like the number of uh single bit adest stations which is like unaggregated adest stations from a single validator sometimes that turns out not to be a very good feature but it was something that we tried so using this new 1.0 model um we

blazed ahead I I kept tweeting these things out every couple of months people would ask me oh how's the client diversity looking is it better is it better is it better and I was basically like no it's not better um so in January 2022 which is really not that long ago we were still sitting at prison um around 23s of the network which is it actually in this graph crossed the danger threshold so people were really starting to get worried at this point that if there were a bug in prism it could really bring the network to its knees and that would not be good um so off the back of this there was a really strong uh sort of social coordination campaign to improve the client diversity um on the chain and there were various websites uh the top one there is now

defunct but we uh some uh in viz from I think uh where is you right now Beacon chain um did some really good stats on what clients the staking pools were using uh using our data and then client diversity. org was showing the total Network breakdown uh some of the staking pool produced reports about what clients they were using and then sort of Pinky promised that they would diversify and then they did diversify which was good um lots of tweets lots of uh Community leaders people probably in this room were helping lead the charge against that which was good and the prism team were very gracious and also started to promote this idea of um uh client diversity and getting people to use other clients because for them they even though it's a sign of their success and

obviously a testament to the great engineering they're doing it's really not what you want to be the developer of the majority client because it puts a lot of stress and pressure on you and we've seen this with gu as well on the execution side it's a really high stakes thing to be working on this majority client and like you don't want to make a mistake when you're doing that it's much more uh uh like robust and relaxing for you as a developer if there's someone that shares the load with you and you can uh not make mistakes but like if you do make a mistake it's not the end of the world um and yeah this this campaign worked so this is the client diversity uh as of today on the consensus layer you can still see prism and lith housee are over

the one3 threshold so we still have a bit of work to do we could ideally spread out another 10 to 15% of the stake I actually suspect that this will improve um at the next denb hardfork because there are some changes to how blocks are built and I think Nimbus will probably um bump up because Nimbus blocks are generally very profitable um and I think people running multi validator setups will probably start producing more nimus blocks and we'll see that in the stats um but yeah we we're doing a lot better than we were and we're no longer close to that dangerous 2third threshold for a while Lighthouse was above prism and we were worried that it might get to the two3 threshold but thankfully prism has like risen back to the top and we're not

close to 60% or anything so it's all good um yeah now I'm just going to talk a bit about the development that we've been doing on block print since then um because we realized this was quite an important piece of infrastructure we really wanted to get it as robust as possible make it as accurate as possible and um keep building it out so some of the features we've been working on uh we've got an API for researchers to use uh it turns out this data is quite useful and quite interesting for doing analytics of the beacon chain um we've also improved our accuracy monitoring so that it happens live um we're looking at rolling out the model so that it supports more clients um in particular loadstar and granding

are not really properly accounted for in our current model but we're working on that and we're also trying to onboard uh more developers and more infrastructure operators to sort of grow the ecosystem get more people involved um and then like really like make this a core part of uh ethereum's diversity infrastructure which is which is cool it also means less work for me hopefully um yeah so the block print API we have to strike a balance here between um producing useful data and also uh preserving privacy for the validators that are being sort of exposed by our um our fingerprinting um the risk here is that if you know what client somebody is

running you can potentially Target them with a vulnerability that's specific to their client um so at the moment the compromise we've got is that there's an API that gives aggregate data um so data about the whole validator set but if you want data about specific validators then you need to DM me and ask for a password um and this is I don't know it's not it's not very open but like the code is all open source but getting access to our instance is permission so it's sort of this this Middle Ground between being open and being um private some of the apps that have been built with the private API are really really cool um this one rog. piix by Tony uh who you'll probably see later

today um is it's really cool it shows the number of reorgs happening over time you can actually see there's a big uptick recently which is a bit concerning um and you can see the breakdown by client so you can see that actually Lighthouse which is my responsibility is there getting rored quite a lot and that's something that we're aware of and working on but uh having this dashboard is really invaluable because it means we can see that this is happening and and address it um another one and I only added this slide this morning so I'm glad it's here um is the EF devops guys have built this cool dashboard it looks a bit washed out there but it basically shows the number of online ATT testers classified by their client and also the number of

offline ATT testers classified by their client so the the top one is the online one you can see it's basically flat over time uh this is just a short time period so we don't see any changes um but we can see that the offline attest is there's a few blips here uh the Orange Line being prism that's probably just a couple of staking pools of something that uh you know had an infrastructure issue went offline briefly and then came back online and you can see the the Y AIS is just like a couple hundred validators so it's not a big issue um but if there was some bug that knocked a bunch of validators offline using a particular client we'd be able to see that show up in this graph um so this is a this is like a private dashboard at the moment but the EF develops people

are like monitoring this and we'll raise the alarm I'm sure if something goes wrong here um yeah so and then the other main workstream that has been part of block print is like improving the accuracy over time and improving the the monitoring um so we need to do this because the clients are constantly improving they're changing how they build blocks and that could change their fingerprint uh in Lighthouse we're about to change our fingerprint slightly um and the other thing is that doing the cross validation that I mentioned before that was just something we had to do manually it's much better if we can do something continuously over time um and the final point is that we were previously limited by uh the graffiti to get training data so um this is fine for most of the clients

but um prism doesn't actually use any graffiti by default so prism was kind of actually underrepresented in our training data because it was hard to like find enough blocks that were made by Prism um and the way we've addressed this is this kind of complicated system that builds blocks continuously um and measures how well they're being classified so I'm going to show you a horrible diagram now but then I promise I'll explain it in smaller pieces um so this is the system that we came up with um you can follow these steps in sequence but basically there's a thing that produces blocks using a bunch of different um clients so I've got this like cluster that runs two of every client it's like no is Arc um and then

it feeds these blocks into block print and it says hey blockprint did you get that right and then it keeps track of whether block print classified them correctly so we produce a block using Nimbus we feed it into block print block print says hey that looks like prism to me and we go naughty blockprint that was wrong and we write that down a key part of this is that we have a um an execution multiplexer I kind of wrote this for fun I kind of wrote this to save on um infrastructure costs but it just allows us to connect multiple consensus clients to a single execution client um which is something that since the merge has not been possible it used to be possible to sort of share an execution client but since the merge it's been one to one uh and this now

makes it so you can have many to one and there are other multiplexes that do one to many I think Nimbus natively supports one to many um but the important part of this for us is that it also takes care of building the execution payloads because we don't necessarily want to go to the execution client and force it to build a valid payload with all these transactions that we don't really look at anyway and that takes time and effort um the other part so the next part is the thing that actually builds the blocks um or actually requests to the blocks this is called block dreamer um and it goes just via the HTTP API to all of the clients that are in your cluster and says hey give me a block give me a block give me a block give me a block

and then those consensus clients go to the multiplexer electric eel and they say hey give me a payload it gives them a dummy payload and then block dreamer gets all of these blocks from all the different clients that it can do stuff with and what does it do with them it sends them to this other micros service called block gauge um and this is the part this is the real uh where the magic happens it sends those blocks to Lighthouse to convert them into the form that block print wants and then it sends that data to block print and classifies the the blocks so this is where if block dreamer produced a block using Nimbus it sends it to block gauge block gauge sends it to block print and block print hopefully says yeah that's nimus or it might say something else and then we

keep track of the the true positives the false positives the false negatives Etc um inside of block gauge we've built also a nice um user interface so this is a web UI that you can access now at blockprint dope. um and I'm very grateful to my colleagues who have front-end design skills um for building this for us which which looks really cool I think um and we show the same sort of breakdown as client divers unity.org we are yeah very open about taking inspiration from that website for this graph um and then we also have a nice table of numbers which is very exciting to look at if you if you know what these numbers

mean oh the red ones are very hard to see but um I'll go into briefly what this means am I Doink the time yeah okay so um one of the main things that we we measure for our model is the true positive rate um and this is sort of like a a statistical property of any diagnostic test it gets used a lot in like medicine actually like if you do a covid test it has like a a guaranteed true positive rate which is kind of cool um so here we're looking at for each client these are the ones in the rows we say Okay of all the blocks produced by that client in our cluster what percentage of them did you classify

correctly lock print so if you look at Lighthouse there you can see of all the lighthouse blocks produced 0% were classified as Grand which is good 98 or 96% were classified as Lighthouse and then 0% were load star 3% were classified as Nimbus which is a little bit of an error but not too bad and 0% as prism 0% is techu and you can see this table uh the true positive rates for all of our clients are pretty good except for grandine and load star because they're actually disabled in the model so there's no way for it to get them right they're always uh they're always classified incorrectly unfortun but we're working on that we on that

soon the other like statistical measure that we um that we keep track of is the positive predictive value and this one is like a little less intuitive than the true positive rate and I think the first time you hear about this you're going to be confused probably I was confused for a long time um but it basically sort of flips the framing and says okay of all the things that are classified as Lighthouse what percentage of those were actually from Lighthouse so it's like the sort of you've you've switched around some of the um some of the numbers in the calculation and um this one turns out to be very sensitive to um things being classified incorrectly because we've got a lot of grandine blocks getting incorrectly classified as Lighthouse Lighthouse is um positive predictive

value is actually really low but it's sort of artificially low because we know that it's mostly Lighthouse blocks that are in there but there's a lot of um false positives from grandine so I think once we get grandine and load star sorted out this graph will actually look quite good you can see that the bottom row for teu is like 98% um and yeah ideally we'll have all of the clients around that like 95% plus level which is really good yeah so the minority clients this is kind of like um something that I've been trying to deal with for a long time but we don't quite have a perfect solution yet the problem is even if you have like a 99% accurate test you're still going to get 1% false positives

and this is a thing called the base rate fallacy that's like common in statistics um so yeah if you have 1% of a million validators that's like a lot of validators right um and this could be larger than the client's actual share which means that if you see something classified as load star it's like 5050 whether it was actually load star so we're not really doing um it means that like if we're guessing if we're like like the positive predictive value ends up being like 50% if something was classified as load star it's like 5050 whether it was actually load St um and it throws off the accuracy for the other clients as well which is a bit frustrating um but our hope is that by using by getting enough synthetic blocks out of our cluster we're able to build

like bigger models that are really really accurate for all the different clients and find features that um discriminate between the between the clients and then once we've built this model and tuned it with all six clients all seven clients however many they end up being um then we'll redeploy it and continue monitoring it with all of this uh infrastructure that we've built so we're I think we're much better set up now to continue growing into the future which is exactly what we need uh and yeah and then this is the final thing we've been working on is sort of opening up um block prints development and um its sort of organizational structure we're just starting a collaboration now with migal Labs who have done a lot of work on um

Beacon chain health and also have their own network crawler they're now running their own instance of block print um they're going to help with the development and we're going to share um training data models um we're going to share some of the clusters for nodes that are building blocks and um yeah I think this is really cool and we may also onboard more um people in the space maybe get some block print instances running on test Nets get one on heski maybe even on um nosis chain which uses the same clients and yeah just basically building out the infrastructure and making sure that it's really solid into the future thank you um very much for listening to my [Applause]

talk does anyone are there any questions from the audience here um one okay do you have plans to spread the block print on the execution clients as well at the moment I don't have plans to do that and part of the reason is that it's hard to fingerprint the execution clients because like 90% of blocks are built by block Builders now so you'd really probably just be fingerprinting the block Builders not the actual execution clients but don't they use the same

client software to build blocks they so the thing is they use the same clients but they don't use the same same client as the proposer that's signing the block so we'd be able to get a measure of what execution clients used by Builders but we wouldn't get a measure of what execution clients are used by validators which is what's more important okay thank you any any other questions from the audience here I guess I guess I had just one quick one about maybe that similar comment have have you thought about uh with the client teams adding a bread Chum instead of a fingerprint that kind of unique identifies as a signature just asking for a field or a declaration yeah um so

we did sort of think about this approach um the reason we haven't taken that yet is that it tends to work quite well um already without like a an explicit fingerprint um and the problem is that with an explicit fingerprint it then becomes much easier to gain like um loadstar wanted to have a a program recently where they' incentivized people for running loadstar and if it was just like put load star in Your Graffiti then you could easily do that without actually running load star or if it's like some complicated property of their fingerprint then it's much harder to fake so but that would be an option if um yeah if the fingerprinting stops working at some point yeah any other questions all right oh wait one more here in the front

please thank you thank you for your work um are you collecting of blocks so you fingerprinting or you collecting statistics on reorganizing organizations yeah so uh how the yeah how the thing works at the moment is that we'll get just every block so um we've got a a thing that's just monitoring blocks as they come in and it just fingerprints them and then puts them in the database um and then if they get orphaned later they're still in our database so we still have stats on them so yeah it's only if there's a slash block that will just have to choose one yeah thank you all right seeing no other questions please give our speaker a round of applause

here