Ethereum Client Security

Transcript

to the def to the defi security submit webinars we're going to have a webinar every month and actually we are starting this event with a a fantastic line of speakers thank you Nathan for leading the marketing actually also in last year in Paris and also this year we are starting this month we have Mario we have seen and we have Amed and I let fck introduce himself and the speakers thank you very much mly um yeah so I'm Frederick I uh work at the EF in the protocol security team I um have the

honor of being the moderator here together with some amazing people I uh I have a few slides to share what the protocol security team at EF is uh is doing but I'm going to run through them pretty quickly because I don't want to take too much time away from the from the actual speakers so let me see I can fix this hope you can see it so um yeah so so the protocol security team at the EF is a group of about 5 to 10 people depending on how you count it uh

we have people that work on full-time on protocol security and we have some people that do it um as a side hustle uh to what they're doing um with their primary time we do a lot of security coordination together with the client teams um Whenever there is a security reability we will reach out to the client teams speak with them um try to handle it uh in a secure way and we also coordinate with a lot of other people we do a lot of research on things like client diversity l2s threat analysis and so forth we also spend quite a bit of time on grants so uh we

we provide grants that are relevant from a security point of view to uh various actors it can be universities or it can be just random people on internet um we also do some instant response in the team and uh a lot of manual reviews so we look at the kind of the ecosystem of the um of the etherum ecosystem uh meaning that we look at all the clients we look at uh different libraries being used uh we look at the sated compiler some smart contracts such as the deposit contract

we look at um uh critical infrastructure Etc and then we also do some fussing um which is a presentation of its own um and one of the key things that we do is that we we manage the bug bouncy program which I think is uh is very important uh this is the program that people can uh can use to submit bugs um if they find a bug um in a client for example they can submit it through the bug Bounty program and uh we will review it and if it's an actual issue that would have an impact then we will provide a bouncy reward and uh as of now we actually we have the uh

the Bounty reward doubled up until the uh up until two weeks until the main net hard workk of dun happens so at the moment the max payout is 500,000 um yeah with that I'm not going to steal any more time um and instead I will hand the word over to Marius from the goian team hey thank you uh yeah my name is Marius um I'm working on the um go ethereum team and I'm going to talk a bit about etherum client security and how to make sure your blockchain

doesn't break so I've been with it with the EF for um almost four years now I um I joined when we didn't really have a security team yet um the security team was basically Martin and um I tried to help him as much as possible and since then the the ethereum uh the security team grew quite a lot um but since there was like basically in the beginning there was only Martin a lot of the a lot of the like security related stuff um was historically part of the GU team and so we've been we've been maintaining a a

a bunch of tools and things there and I'm yeah I'm just going to go through it bit so um for those who don't know G is one implementation of the ethereum protocol there are uh multiple implementations in different programming languages um gu or go ethereum is written in go later on we will hear also here from nethermind which is uh the net implementation um in C and uh gu is what we call an execution layer client um that means we maintain the accounts and balances uh we execute and verify the transactions um we make sure that um that once you start up your node you

will synchronize with the with the other nodes and and download all of the old blocks all of the old transactions all of the all of the state and we also provide apis to interact with a blockchain so if you want to send a transaction you will send it um to an execution layer client or most likely you will send it to infura who will then send it to an ex execution layer client and um there are a lot of different issues within G um um the most common types of issues are denial of service resource exhaustion and other security um related issues so denial of service um go is a language that will panic and

shut down on runtime exceptions so if you for example um access a nil pointer then it will um create a panic and um shut down the client and because we don't want um other people to be able to shut down the the ethereum network with one uh one transaction or one message um uh we have to make sure that these denials of services are not happening and we use unit tests um most of the time to help us catch these scenarios um another big chunk of um of problems are resource ex extion basically if someone can create uh create a transaction that

will take 10 minutes to run um and send this transaction then every Noe will take 10 minutes to run this transaction every gu node and um so the network will not progress further and now with proof of stake we have a 12C block time and so if the block is not executed within 12 seconds um then the next block gets created um but if we have an attacker that can create blocks that uh that are 10 minutes to run um then they can continuously send these transactions and make other people uh exhaust their resources and um that's really bad for

the network and uh because we have so much API surface um both in means of like transactions networking messages but also the Json RPC API um attackers can very easily uh hog resources so we have to be very careful about um the design of the code and we have a lot of uh fuzzing in order to make sure that we um we don't hit these issues where where some transaction takes 10 minutes to execute and um another thing is the security of the overall network uh of of the overall system um because ethereum nodes are are

Services they usually run in the background on your system um but they can interact uh with other things on the system um and so if for example you you are able to make a guess node use three times the amount of memory that we usually use um then this will have impact on other other services that are running on the same system um so unit tests um um unit test is very important for us because it um it's the easiest way and the quickest way for us to test the functionality um we have unit tests for for small functionalities we have uh components and we have

subsystems and the cool thing in go is that we have great unit test support it's built into the programming language and in gu we have 12, 446 unit tests uh which takes uh less than 2 minutes to run on on on my machine so it's a very quick way to cover a lot of um uh a lot of ground and these tests are run on the CI uh whenever we propose a change to gu and so if um it's very important to build your components um from the ground up with testability in mind so it doesn't make sense to create a component

and then think about testing uh because it's too late then then you have to refactor the whole component again and um yeah so our components we we try to think about how we can set them up for testing with um different like mocking out different parts of the component um from the beginning and that makes it uh very easy to test things in your code um another thing that we use in in go is uh is fuzzing uh fuzzing is basically you call a function that you want to test with random input um or interesting input and um try to find errors in it either um just uh find

panics so we have three different ways of of fuzzing um in the first way we we try to find nil poter exceptions area out of bounds exceptions things like this um so trying to fuss for panics um we have tests where we faster functionality and its inverse so for example encoding and decoding and we make sure that what we encod what we decoded matches what we encoded in the beginning and um then we have um uh then we have another mode where we F two implementations of the same functionality against each other um this is called differential fuzzing and um so for example we have three different

Library is uh implementing the signing functionality so if you sign a transaction in G we have three different libraries that that allow you to do that um one of them is written in C and the two others are written in go and we make sure that all of them do exactly the same thing on the same output and um yeah so what my whole Implement uh whole presentation is about is basically testing needs to be a class citizen in your project um and we make sure that uh whenever we build something we always think about testing as well um Frederick already talked about the buck

Bounty program that's uh why I'm going to um going to skip over it um just I just want to reiterate that right now um all of the points or all of the payouts are doubled um so if you find a bug and submit it right now you'll you'll get twice the output uh twice the reward that you will get otherwise um and then um I'm going to talk a bit about cross client strategies so in the first part we we talked about testing strategies for GU the client and now we're going to talk about uh other clients and one thing about these cross client strategies it's very important to have consensus so the ethereum network is run

by five execution layer clients and five consensus layer clients and um so you have at most 25 different combinations of clients and all of them must work with all of the other implementations and they have to do exactly the same thing if one implementation does something different then we have what we call a chain split um so for example um when I get a transaction I'm I'm I'm I'm get a trans get a block with a transaction I have to execute the transaction and make sure that the person that sent a transaction had enough money to um pay for the transaction if um if now two uh implementations um calculate how much uh

this the person has to spend to send this transaction um differently then it can occur that one one node says yeah this transaction is valid and the other note says oh no the person doesn't have enough funds for it and um so they are stuck in two different stat that they that don't match each other one where the transaction was executed one where the transaction was rejected and so for ethereum to work it's critically important um that there are no differences between these implementations and one one thing that we that we use to find those um or to test those is differential testing um basically we take the same input um and

we run it on all of the different evm implementation um all of the different clients um in this picture there's only um there's only some clients um we do this also on other evm implementations so for example on the on the python imple evm imple implementation or on the JavaScript evm implementations and then we collect the output of those implementations and verify that they do exactly the same thing we have various different uh ways of doing that and um this makes it very easy to see if uh if something if something broke um because

if if two of them uh if the output stiffer then you know okay they decided something differently and we need to make we need to investigate um another technique that we use is malicious nodes um basically we set up a network a test net or Dev net and in this network we introduce malicious nodes these Mal malicious nodes can insert or alter transactions they can change Fields um within the networking messages they can send big values or nil values um in the consensus layer we have uh notes that can double vote of signatures and um send just send weird stuff so in order to try to uh

break the other nodes and then we have uh something that we call shadow fors um basically this is something that um that started around the merge uh where we um where we thought about how can we test the merge with um so if we if we only test uh our software on the test net um then it's it's cool but it's not really tested because on a test net you don't have as many transactions you don't have as much State you don't have as much history and um so what we want to do is basically um test everything on mainnet but we cannot test

it on mainnet because then we would like people might lose money if if we up so what we started doing is is Shadow Fox um basically we configure a subset of nodes with new rules and then at some point we activate these new rules and then all of the transactions that are valid on the normal main net are also valid on our shadow forks and are being replayed on the shadow forks and um this helped us uh tremendously around the merge where we found issues that we would have otherwise not found until mainnet and um this is an overview of some of the testing tools that we use um in internally and and with uh also with

the with the other uh teams um so for example uh go go evm lab creates um creates State tests for the execution layer uh we have malicious CL and El clients uh we have uh kosis and anthesis for running um deterministic fuzzing and nightly tests um so we have um we have larger scale test Nets being run every night uh we have hi where we have I think [Music] 150,000 tests that are being run on every

um uh whenever someone proposes an update when uh uh whenever someone proposes an update on on any client they are run on hive um I think it's it's nightly or something and um yeah so we have a lot of different testing tools and this slide is also kind of old by now so we we build even more um yeah and uh I want to spend the last time that I have to go through some of the uh stuff that we found uh some of the trophies that we that we collected mostly during merge testing um so this SP was an issue in uh the gas calculation and bisu um it was

found by go evm lab where we created these the state tests State tests are basically just a transaction um and some State and you can you can execute the transaction and see um and see the result and here the problem was that there was a issue with the gas calculation and uh this gas calculation would overflow and so what would happen is you would create a call and then you have basically infinite gas and this would be really really bad because you can create blocks that will never stop being executed and

um so um here this bug it's also an older bug um but it also has a has a bit of a history um there was uh an issue in return data copy in the GU um code base uh where the memory return uh was shallow copied and so uh modifications to the return data um modified the memory and the bug wasn't wasn't like it was kind of bad but it was quickly fixed and discovered um but we did not announce our uh the bug publicly and uh after a few months someone found the bug

and deliberately triggered it on mayet um because they said that only 1% of notes were affected um but the notes these nodes were uh in fura notes um by chance and so my I'm I'm yeah my it would be really nice if you ever find uh bugs in one of the clients uh even if it's already fixed in the news version Don't Go and Trigger it on may not uh we had a bu in bisu last week that was also deliberately triggered on mayet um and yeah you can go to the you can go take that bug if you find it uh

you can go to the um to to the bug Bounty program and get get some money for it and it's way easier and better for everyone um than triggering it on may not um here are some B bucks that we found during the merch with the the shadow Fox first one was the default gas limit so in G the gas limit was set the default gas limit was set to 8 million so when we had the first man Shadow fog uh the gas limit was quickly voted down from 30 million down to 8 million and if that would have happened on mayet uh it would have been kind of bad um because

only like this is from the beginning of the merch the blocks would be uh smaller and get smaller by by 2/3 more than two 2/3 and that would have been really really bad another issue that we found during the shadow forx um was uh there was there were two issues one of them made us um uh do a 60 600,000 block reok and during this 600,000 block reok on a on a girly Shadow for um this exceeded our uh available memory and so we were able to rewrite the reor operation to to fix this issue and

another thing was the ESS of the base fee um basically um the values that we have on the execution layer have to go through the consensus layer and uh prism used the wrong ESS for the base fee and uh just creating a bad block on a shadow fog when the base fee was over 255 um because um if you have a value over 255 it uses more than one bite and the endianness only comes into play if you have more than one bite and um yeah so on our test Nets or on our small tests that we had um we never went with a base fee over

255 um because we just didn't have enough transactions but on these shadow Fox we did so um that was that was a good find um this one is a bug that we found um uh with our fuzes and basically um there was a there was a panic in the uh math library in the go language and so uh yeah basically if the modu was larger than 6,000 bits or six kilobytes uh on uh kilobits on 64-bit architecture then the standard Library

would crash and this would like if you had if you knew about this you could send a transaction that would crash all of the GU and so we had to fix it in the stal library um this was a consensus issue in nethermind um where um the modulo would fail to handle the case of 2 to the power of 255 correctly um this one is a bug another bug in the mod x precompile um and this was found in bisu and recently it was also found in Nimbus um where basically you have these three

parameters um mod x is X power of p mod uh x to the power of Y mod P uh so you have three three parameters and um in the precompile uh you have the base modulus exponent and then the base the modulus and the exponent length and um so if you set the length and the modulus to zero um then the the the other parameters have to be zero as well um but uh some clients did not uh do the shortcut and you would result in an overflow um those were stuff that we

found in nethermind um where we where there were was an issue with the copy op codes um if if length uh was Zero for any of the of these of these copy of codes um and now if you want to join us um we are still looking for people um for the so we have two teams we have the security team um that is uh very focused on um pen testing and fuzzing and active security and we have the testing team which is more focused on providing test cases um for all the clients uh making

sure um everything every everything works they maintain um Hive and they maintain the the execution spec tests and um for that we are still looking for people so if you want to join this effort um you can contact Mario at mario. e.org and um yeah that was it thank you for having me thank you very much Marius um very good presentation as always um there are some questions from the from the audience um I I have a question as well actually

that I want to to to lead lead with and then we can ask some audience questions um so there are a lot of different layers when it comes to Security in our ecosystem so you have you know end users doing doing stuff on Shain you have the smart contracts you have validators and the infrastructure that the vales are running on um and also on the like on the validator Node layer you have like sub layers such as networking and and so forth um so I'm a bit curious of what you feel is

the is the weakest part uh at this this point in time and what we should kind of focus on improving security wise um yeah that's a that's it's a great question uh we have um we have a lot of expertise in testing the consensus like the evm um like it's it's um if if there's a bug in the EV evm it's very easy to find for us um so it's not sorry it's not only uh it's not only um like finding the bugs um or or fixing them before they happen but also if they happen can

we like quickly quickly figure them out and within the evm we have um we have a lot of tools for that for finding debugging consensus issues um it's it gets a bit harder if you have issues in the surrounding code so for example if you have um an issue in um in the way the block is Con constructed um that is um more complex um to to find at least in the execution layer I know in the consensus layer they have ways of dumping out the raw block and and and things um as far as I know um and so uh that's very cool and that's something

that we should uh think about as well um and um I think the the the most interesting part for me would be the the networking um the this is something where we do have some tests uh we also have some fuzzers um but we don't have like a very comprehensive test suit uh for those um we have a test suit for sying um so um we but this is also not really a focus for for client teams unfortunately and so people people are

not not looking too much into into these areas yes and um on the consensus layer I I cannot say I'm I'm mostly focused on the execution layer yeah it's interesting because the um if you look at the vulnerabilities that have been discovered the majority has actually been on the network layer um the service attacks Etc um so from a bug Hunter point of view it's probably one of the good areas to look at I would say um and then we have two questions from Tom um the first one is about uh when Shadow Forks are being done um he's wondering if all the different clients

are part of the Shadow Forks or if it's just like go ethereum at one point and Des at one point perap you can explain yeah it it kind of depends where we are in the cycle um sometimes we do these shadow Forks very early um and then it's only the clients that Implement uh the change up until then so we had shadow forx where it was only two or three of the five clients um um on the execution layer U basically the shadow Fox are only on the only on the execution layer unfortunately um because for the shadow F to work um we have to create a new consensus layer with a new Beacon State

um and uh with a new Beacon chain and so um that's that's kind of how we do the shadow Fox um and but if we do the shadow FKS before a hard fog um then there's Everyone is always on board there so U we try to be as early as possible with these shadow Forks um because they give us a lot of information um especially for example around um around for 44 now um there was a lot of questions about how like how is the bandwidth impacted on a a node that has um like mayet load uh with the blobs and everything The Blob transactions and

the transaction pool and um so it was very important for us to do a shadow for uh quite early in the process uh with all of the implementations that we're ready for that and um but yeah if if we do it uh like closer to the to the actual for date then there's always everyone there yeah yeah and then there's the uh the duts um parure that and so forth yeah so another question from Tom then is about uh denial services on the minority clients uh Tom used Bess as an example here um so he's wondering a bit about let's say a attacker triggers a service onu he's wondering about the the actual

impact um like some people obviously the ones that has uh Bess um Bess node or client they would not well they would go down um but what would the impact be on the network itself um so we we actually seen this uh before um there was uh in the very early days there was a Dos on G um where G nodes uh the it's called the Shanghai attacks um where guas nodes were not able to follow the chain anymore and fortunately we had um back in the day parity um and the parity nodes um they

were able to progress the chain so ethereum never went down um back then I I'm not I don't know how how the percentage were uh between the different noes um but um if a client uh like a minority client um has a Dos um it's it's uh not that big of a problem as long as the the minority client has less than 32% uh or 33% um because um then we are still finalizing the chain um but even if a ma majority clients has client has a denial of service uh it's not the end of the

world um because even then the the chain will continue on um but um yeah it would be it would be definitely bad um and the the client teams that see that there's some some does happening uh will have to scramble and fix their software and get out the new release for their for their to support their clients and um but yeah usually um like the dos and resource exhaustion exhaustion attacks are quite obvious why it's happening and so writing a fix for it is is very uh is

very quick so um like we should be able to write a fix in for for an attack like this in like 4 to eight hours roughly so okay perfect thank um next up we have Amad from the neine team who is going to do his presentation hello hello hi uh okay so I'm going to share my screen uh right

okay my name is Ahmed barar I'm an ethereum uh core Dev with the nethermind team uh today I'm going to talk about uh ethereum client security and the possible effects uh of issues in this domain um I'll start right in no introductions because uh it's a Hot Topic uh so in in in in in a client uh in an ethereum client uh talking especially about whether it's an execution client or a consensus layer client but mostly I'm talking about execution clients here uh there are a lot of moving parts right and and for each part uh there is is there is uh possibilities of of different types of

attack on on each one of those uh Parts um and and we like uh the execution client has obviously a database where we keep the storage of blocks State uh Etc uh it has the evm and the consensus to process these transactions uh there's also the P2P Network can with uh nodes around it and uh also uh it has a Json RPC where uh the users can can query uh this node and ask it about the state of the chain um and there are

others obviously like transaction pool for example that will have transactions uh up before they are submitted to the blocks uh etc etc right so because we have all of these moving parts and and they all these parts interact uh together there are the attack service be the attack surface or or the security issue surface becomes Higher and Higher and and we can we can look at uh example of security issues uh that we have seen in clients and other uh uh also uh uh examples uh so for example uh there was one that was mentioned by Marius

that nether mind you and 256 where you wanted to do a modulo on uh that large number another mind wouldn't want know how to handle this issue and this is a consensus bug there was also a recent bug in uh B smart chain where they uh they had two two clients I think one of them is a fork of G and another Fork of Aragon I think and and basically the fork of gu was their uh uh main client mostly and and it had a they had a consensus issue between the

clients that eron wouldn't accept the the blocks coming from G and what they had to do was they had to uh hard code the state transition that happened uh the buggy State transition that happened uh uh uh in the uh in the binance implementation of the fork forg and and and and that was like that the exact example of a very bad consensus bug issue that that we want to avoid in ethereum as much as we can um there is other other issues in clients sometimes uh in the previous uh previously nether

mind would suffer from a lot of uh out of memory and it would be killed by the operating system uh uh sometimes it would be um uh it would have its it uh a memory leak and uh that is because of uh some bug in the code not not handling memory properly uh another issue was uh one other uh example of of of security issues is that a client a peer would be disconnected continuously because for example request or response size is a bit too big and uh bigger than other clients would be uh expecting for this

request or response and they would drop the connection from this pier and as such this peer would lose a lot of uh uh uh uh other peers that was con to and it w't be able to progress with syncing the network um you have also examples where uh database Corruptions happen uh data data and the database become unreliable uh nether the client would try to for example uh read from the database uh but the database won't have the data or the data would be corrupted and the client won't be able to progress by processing blocks Etc there is also um the issue that was I think also mentioned by

Marius where the the go ethereum implementation transaction pool uh on Gurley uh maxed the CPU usage and this is a a very simple example of of of Dos denial of service attack uh Ledger even though Ledger is not an ethereum client but it it the the hack the way the hack happened was very interesting where the hacker utilized access on a GitHub repository on a GitHub account to run a a a a GitHub action that would publish malicious and vulnerable implementation of The Ledger connect whatever Library

the the library that and because Ledger at the time used to allow clients to pull this implementation from a CDN uh then these client uh these users and the browser we pulling the V vulnerable uh uh JavaScript library every time because the hacker was able to publish this vulnerable uh uh implementation um so this also shows uh uh how how bad it could be if if for example someone with authority to publish a a client for example uh can can publish a client that is buggy uh that's why we tighten the security

around our GitHub GitHub uh repositories as much as we can um now like let's Explore More uh uh on these components that were mentioned before what what possible issues and these are just examples again uh they're not everything right and we will look into each uh uh how how big of an issue each one is right how big of an issue in in each situation uh for a minority client for a majority client Etc we're going to look at it so for the for example the P2P network if you if you're able to do a Dos or a denial of service

for a minority client it's not that big a deal uh for example um nether mind can drop the network can still finalize it's not higher than 33% for example it's uh the network can still finalize everything is fine it would be bad for those ones running nethermind for example uh or or any other client that is a minority client but it it it won't affect uh the network uh as a whole uh the network would still be able to finalize and progress normally uh synchronization issue for example um so uh if for example a client is not able to new clients not able to sync from scratch uh then this is not that big of

a deal because only new clients are affected uh uh uh or new nodes are affected but existing nodes are progressing normally so this won't won't have that big of an effect um same with database corruption and consensus issue if a minority client has it uh that's not a big deal it will just Fork on its own and the the the the network would still be able to finalize uh this client would probably fix the issue later and it would join the the correct fork and would start attesting to the correct Fork again not big of an issue um same thing for consensus issue uh evm with with evm and consensus not

that big of a deal uh if we look at Json RPC transaction pool we already talked about denial of service for a minority client and what would happen uh it would just be denied till till till the issue is fixed uh like Mario said probably four to eight hours for denial of service if if it exceeds I I would be surprised if it exceeds one day to be fixed and and and released to to to to the public um now looking at these uh uh issues from a perspective of a majority client is a completely different story when we look at uh a majority client having a denial of service uh attack then we would uh suddenly have um

finality dropped and and that's that's one of the cases right uh finality dropped and when finality drops something happened called an activity leak uh in ethereum and when an intivity leak start uh uh uh happening all of these not being able to attest because they they they got dos right uh and they were running the majority client would lose uh start losing a part of their stake that would increase quadratically uh with every Epoch uh till they lose possibly could be all of their stake so

it's really scary um synchronization issue is the same like only new nodes can't uh sync so it's not that big of a deal now um let's talk about database corruption so if database corruption leads to a consensus issue and not just an an activity leak then that's a totally different story for for a majority client because here we're talking about um more issues uh depending on the percentage of that majority client for the validators uh we will we will discuss this uh uh more in the on the next slide same with consensus issue it's very high

critical uh denial of service as discussed before for for a majority client would lead to an activity leak and and and uh the more time that these clients are offline the the the higher the the the penalty for being offline and not attesting to the network so this table here specifies uh uh uh what would happen depending on the client's uh uh validator uh uh market share right um so if the client is below 33% if it went offline uh uh or it had a consensus issue then uh there's no significant an impact the chain will

still be able to to finalize uh there will be uh there will be no big penalties because all the penalties that these clients would endure is just the the normal missing attestation penalty which is not that high and it does not increase quadratically um where if the client percentage is between 33 and 60% 66% then uh the chain will stop finalizing and if it's above 50% so there is there's three thresholds right there is the 33 there is the 50 there is the 60% if it's above 50% then we would end up with uh two chains and and that

would start being problematic right uh but either neither chain could finalize so if if things depend on finalization uh uh like bridges for example or deposits in De centralized exchanges Etc then then then we are still safe kind of right if they're built right um and the inactivity leag kicks in resulting in penalties that increase quadratically every Epoch and that's really tough because you we will see in the next slide what happens with with these increases and how bad they are um and also for the rest of the stakers they stop receiving attestation

rewards uh uh but they don't endure any penalties and there is a reason for that the reason is that if they if they uh keep receiving attestation reward then they they have an incentive to keep uh uh uh the rest of the network uh offline uh uh because they would uh get more uh uh so so it had to we had to also prevent them uh that that already attesting uh uh uh uh nodes from uh from from from trying to do that right um so what would happen if we are talking about a more than

66% of of of of the market share of validators then we have very problematic outcomes right so what would happen is that the majority client would be able to finalize an incorrect chain and if if exchanges and and like for example uh stable coins uh that have backing etc etc there will be a lot of consequences because they can depend on transactions that land in this incorrect Fork uh to do uh uh to like to deposit money for people to allow people to withdraw money uh uh etc etc and and and we would end up with two forks one that

is finalizing but it's incorrect and the second that is correct but cannot finalize and it's it's in an inactivity leak uh uh and and what happens is that so we have two choices we either accept the bug and we make all the other clients buggy just like the majority client is buggy and then we forsake the good Forks that don't have the bug right or we have another solution which is to forsake the uh buggy chain but then it was finalized and probably some people already did certain or or a lot of

transactions that uh possibly uh uh gave people money and and Etc and they can't go back on these transactions after we decide okay we're going to forsake this uh finalized chain right finalized Fork so it's it's a very big deal uh uh so there is okay now looking at an activity leak what would happen uh uh uh when when there is a majority client that goes offline uh meaning anything anything above 33% would lead to an inactivity leak and and and and here we look at this is this is a chart from dank red

uh uh article called uh run the majority client uh at your own Pearl right and and here um this graph is actually very very old it's from 2022 uh march to 22 uh so the actual the actual graph of what's currently on the beacon chain the value of the slope is way higher so what what you're seeing that the the the blue one is is is is for sure not not the one that is running right now and not even the red one it's even sloppier and higher so it goes way faster uh uh uh uh with the inactivity

leak to the point that in batric the value was set that the validator the inactive validators would lose 60.7% of their stake if in only 18.2 days uh which is 4,096 aox right and and that is and that the reason that the inactivity leak happens is to pre if if a majority uh uh of the chain started going offline the chain can start finalizing again once uh these offline uh uh uh validators uh are uh uh their their stake goes uh low and um they uh they

the rest of the validators can then attest and they become the majority again so they become more than uh uh 66% and they can finalize the chain um so as you can see the even even this red very scary uh line is actually not representative of what the value is set right now it's really scary for all the validators out there running a majority client and and like um I would highly advise any any any validator uh uh in especially institutions running a majority client to switch as soon as possible uh uh uh

because they will the in their risk assessment they need to keep that in mind keep this risk huge risk in mind right um so right now the state of client diversity uh you can look at the website called client diversity. org and and unfortunately this is very unreliable data um for consensus client the data there is multiple ways of of calculating the data so one one uh one of them uh actually crawls the P2P Network to know how many CL nodes are out there and what they uh what and what is the reported client that they're running uh but this is not representative of the validator node

it's representative of the just the CL nodes that the number of Cl nodes that are running uh another uh way of calculating these numbers is using uh the format of the block to use an algorithm where they would know if this if it's formatted more like uh the produced block if it's formatted more like uh prism Lighthouse ticko Nimbus lster Etc and obviously this this mechanism it's not that accurate right because there there could be a lot there is obviously a lot of similarities between the produced blocks and and and cl clients and the accuracy of this method is very

low uh the last uh one of of of the ones that are presenting the data we do not even know how how it's collecting the data rated network uh it's it's closed Source um for for the execution client data um there's a website called exclusive diversity and what they did is that they went to institutional stakers and they started asking them what what what type of of of execution nodes are you running and this is the reported data that they got and they only covered like 60% of the network and that's not even that much updated so institutions could be uh uh changing this data without informing uh uh execution diversity due to info that they have

changed execution client so so this data is super unreliable uh I will give some Alpha that this is this will is about to change uh uh hopefully uh soon uh and we will hopefully have reliable data but I can't say anymore um yeah um so what I was saying is that so client diversity does is is is is a bit complex because if you if you go and look at nodes uh and the work and you ask them what their what what what client are they they will report it but that doesn't mean that that that's representative of the validator

diversity because validators uh you can run uh thousands of validators on the same node uh uh uh without issue so that means that uh just knowing what the nodes are running uh does not mean that uh uh it's it's the same representation uh for validators same thing for Builders especially with uh Mev uh uh we have Builders always uh uh uh building the the blocks most of the time uh we're talking more than 90% of the time Builders are are building the block nowadays uh uh uh and that results in if all the builders are using the same

exact client then we would possibly and and this client has introduces a bug there could be a possibility that we will have always no blocks because every block would not be able to be uh uh included in the network because the the other clients would reject it right and and this is problematic obviously um and that that's a wrap I would like to reference uh T gr's article uh and the E2 book uh is scker article about Cent diversity also uh client diversity.

org uh for the statistics and thanks to my uh uh colleague Merrick for his help and review on the presentation and yeah that's all from me thank you very much Ahmad I think uh that was great I think uh the way you made the security issu surface very security issue surface very was very clear I think that was really easy to digest and I think one of the key things is probably that you show the risks of actually running a majority clients that it can impact you um in a very negative way if you do if you do

that um so there are of course risks of running a majority client and people should definitely consider running a Min minority client um you uh yeah on the diversity part I think yeah that's that's really interesting and I think you can you can probably expand it even more like thinking about you maybe don't want to put your validators on AWS for example because a lot of others people might doing the same thing and if AWS were to go down for example that could also impact your 100% so so home staking is definitely something that's uh worth considering um we had a question from uh

Shia Kwan I I hope I pronounced the name correctly um Shawan was wondering about a bit about client diversity so basically wondering what is the major reason that people are staying with get as the majority clients and uh not switching to neine for example what do you think about that right so first gu is a very good client right and uh uh uh it's one of the best actually if not the best um and and that the the GU team are one uh are one of the best client teams that that can can can work

on a client uh so it it performs really well uh it uh it has very high performance uh levels for for validators and and like when when you look at institutions which are uh unfortunately uh mostly uh what who run uh um the the the validators especially with Lio dominance being around 33% of the network uh you would uh see that um uh you would see that these these these uh validators are always optimizing for the highest Effectiveness and if if even uh get is better by uh uh 1% than another client and in the

performance perspective for example then uh these valid institutional validators would choose get because this 1% to them is a lot of money uh uh uh uh left on the table and they wouldn't want to leave it on the table of course this is only one aspect right now nethermind has been uh seriously improved in the the in the performance perspective uh previously nethermind for example used to require uh more than 12 gigabytes of RAM for example to run now it requires less than four um previously nether mind had uh uh for example a lot of uh uh uh

missed attestations now almost never missed attestations right uh there was a lot of optimizations done on nether mind so till people are used to seeing these minority client uh uh uh in the lead when it comes to Performance it will take some some time till till they adjust to this change uh but yeah that that's about it yeah and I guess today a lot of uh blocks are being built by external Builders as well um and I think a lot of those or if not all of them are using get which is is could be a bit of an issue as well I guess um that needs some work um you mentioned the uh the Le

Ledger connect hack which was a very interesting supply chain attack in my opinion um never mind like basically all software today uh use various libraries um so you mentioned that you know you you secured your um I'm guessing your themes accounts for example making sure that they're secure and so forth but how does never mind protect against supply chain attacks um you know on the libraries you're depending on so right so so we basically don't don't just take the latest library and start running it NE mind whenever it's available right and nether mind does does not have this automatic way of

pulling these libraries uh nether mind needs to be built with a specific library number uh that's how net and C work you have to specify the packages with versions specific versions so we tend to uh test run all all of our updates uh uh when it comes to these packages uh when when we update these packages we test run the client make sure it's running smoothly we we we run the fuzzers and and we make sure that no issues uh uh are happening because of this nugate update whether it's a regression and performance or uh uh anything else more serious okay perfect thank you very much

almad um the next speaker that we have now is Sean from Sigma Prime hey how's it going everyone let me just share my screen here okay so I'm Sean I work at Sigma Prime um I've been working on the lighthouse client for the past uh I think 3 years now maybe three and a half um and I'm going to talk about Security on the consensus side of things so a lot of the concepts will be similar

to what we've been talking about with uh the execution layer um but there's a bit of a a different context that we think about them in um and there are some unique sort of problems that arise from dealing with uh things like Fork choice and the consensus layer and we've been able to develop some uh interesting tools to deal with these types of problems um to try to like uh find bugs um and issues and implementation uh more quickly um okay so first I'm going to start with just um understanding the sort of core issue that the consensus layer specifically is

trying to solve um and that's related to like deciding which chain is the correct chain so coming to consensus on what is the correct chain uh so if you start with a single block and this Block's valid and then another block is built on top of it pretty easy to come to consensus like block two this is this is now the head of the chain um however we have a third block that enters the picture and this block is also valid but shares the same parent as block two now gets a bit more confusing do we pick do we pick the latest block or do we pick um the block that we saw first um

because these two blocks sharing a common parent they're going to lead to different chains okay so in previous stake we have this vote voting mechanism to determine these sort of tiebreakers and this is the uh Collective weight of the the stake behind attestations that validators are making um and I'm not going to go too deeply into the details of it um but this sort of waiting mechanism is how we resolve these types of issues um and other ways in which uh determining the correct chain can fail um we've talked about in previously but U it's related to chain

splitting um and this would have to do with message validity so when I say message validity what I'm talking about is do different clients or different notes um when they see a message do they have the same rules implemented about like is this valid or not because if one determines it's invalid for example block two they might discard it and never look at it um another node might have identified it as valid imported it and started building on top of it um the the node considering this invalid would never be able to come back and import that block and reor to the correct chain um this would be a chain split um we

mentioned it in the execution layer with regards to like making sure the evm uh is functions the same um across clients uh but in the consensus layer we have similar like we need evm correctness but we also need um and we rely on the execution layer for that we also need to make sure that um the management of the validator set and the interpretation of attestations that validators make are consistent um and so that's related to like the message process processing the State Processing that we do um uh making sure that these are on the same page uh limit chain splits that are

uh persistent but there's also a component of message propagation so making sure everyone sees the message um if you aren't seeing votes on block two in this scenario then then you would think that the waiting on block three is higher than the rest of the network thinks so this may cause you to remain on the minority fork for longer um and there's also this timing component to um Fork choice and uh picking the canonical chain and this timing component introduced by the fact that validators are required to vote at a specific time in the slot so that's at 4 seconds so which you're really getting

is like an aggregate view of nodes with different Stakes like what is their view of the chain at this time in the slot and from that is where you make decisions about which chain do I go along with um and failures in any of these previous errors can lead to things like reorgs um in this scenario what would happen is anyone who had set block two his head and voted on block two um um if later block three ends up getting more weight they would have to reor to block three so sort of discard block two for now um change to block three set that as their head uh

and in this scenario typically like whoever is building block four would have um a lot of weight in determining which of these two chains ends up being canonical there's a lot of details related to that um and then there's also the impact of offline nodes on consensus and uh we talked about this in ahmed's talk where once you hit certain thresholds at that point you can no longer finalize the chain um and so for example if onethird of the nodes are offline then finality won't happen you start getting the inactivity leak and the other very important component of

this um from the perspective of the consensus there is that the longer you have these periods of non-f finality the more possible um blocks you have to keep around in order to like potentially make a reorg and also the states corresponding with these blocks you have to keep around um so this is the same sort of structure but introducing um what what are the the beacon States um so these are the states associated with the consensus layer these are the current state of the validator set and the um uh state of the chain based on uh how the consensus

layers applied all the blocks leading up to this state um so each one of these states on the left is sort of like an intermediate between different blocks um and in periods of non-f finality we need to keep more of these states around in order to potentially apply these blocks um to find different states um because there's more possible chains that we have to consider when deciding what is the head of the chain and these states end up being like a big deal for us because they're a lot bigger than blocks so blocks are somewhere around like um like 100 kilobytes whereas the

beacon state right now is somewhere around 100 megabytes over 100 megabytes um how this changes in denb is we're adding blobs which are more similar to blocks in terms of size they're also around 100 kilobytes each and we're adding six of them um but they don't actually increase the size of the beacon State um yeah so managing these states becomes like a big problem um from my point of view and in terms of like thinking about how we want to design parts of the client because uh the easiest way to take down a client might be making it use a lot of

resources and either holding a lot of states in memory or making it dump a lot of these states to disk um and this would typically happen in periods of non-f finality once we have finality we have like some set of assurances that we won't actually need these states again so what we do is we we end up pruning them pretty aggressively um so we we by default store one state every 2,48 slots um once the chain is fin finalized um and this is because like we have assurances that we won't reor to that chain unless there's like a mass slashing event um okay so yeah in terms of security I

think about like how would I do the most amount damage possible to the ethereum network and we've touched on this a few times but um a chain split I think is maybe the most damage you can cause um and the specifics the specific version of a chain spit that is the worst is when something that's invalid is finalized um and this is because when you have something invalid that's finalized um you now you can't vote on a different chain without being slashed so um as Amad mentioned in the past in his presentation it's either you keep going with the um invalid but finalized chain

or all these people are going to lose their stake a large portion of their Stak um yeah and then the other component here is you can do a lot of damage to the network just by inducing an outage so this would be like a Dos like been talking about um I think of this often times more in ter in terms of resource usage generally um and that's because you can cause clients to um miss their caches frequently if you have things like a lot of Forks or maybe like unexpected scenarios related to Forks um

that could cause clients to do like a lot more processing or maybe look at their database more times than they expect and then at some point you can get to where the machine they're running on can't actually keep up with uh catching up on this old processing as well as like doing new processing related to uh staying with head of the chain um so there's a scenario where you're slowing chains down enough to where they can't keep up and also um there's obviously the risk of just fully exhausting resources on the M machine that they're running on um and so this isn't necessarily as bad as um finalizing something that's

incorrect but it still is bad in terms of if enough of the Network's offline all of a sudden the chain isn't finalizing and then that sort of like compounds things because all the nodes have to keep more blocks and states around and handle um uh the potential for longer reorgs and this increases like the RAM and dis all nodes are using so there's sort of like this compounding effect there um yeah so generally we have a strong focus on like correctness um in order to avoid a chain split and Mari talked about this with um validity a lot where like you can do

things like differential fuzzing to determine our clients like acting the same and a lot of different ways um I found that with the consensus layer we have like a common set of tests related to message processing um and State Processing um that all clients run against and this has been really good to alleviate the potential for if a single client is writing their own test for all these scenarios well one there's the risk of maybe you wrote the test incorrectly but then there's also the risk that you're testing your implementation more more than you're testing the actual fun functionality um so having this common set of tests that all clients run

against helps with us understanding functional differences um and it also to some degree reduces the amount of like duplicative test writing that we end up doing um yeah and so another thing related to this is multic client testing generally so whether that be on like local test NS or just um uh Network simulator type tools um testing across client types but also client versions because it's possible that across different versions of your client you change something in consensus that causes it to be incompatible with either a prior version of your own client or someone else's

client there's this interoperability component that gets like pretty complicated and tough to test pretty quickly um um yeah and I wrote this point fuzzing um generally this is good to find stuff related to like invalid data in your specific implementation um how does your implementation differ from others um and I mean that in terms of like more a more granular granular level related to specific messages whereas um we can also get into fuzzing in like a different way where you're fuzzing related to U maybe messages that are valid but coming at different times or valid but

also intentionally able to like uh induce Forks in certain scenarios things like that um and that's where this like timing propagation type chain chain split comes in um and this can be related to like a fork Choice implementation which um I've done some work on in lighthouse in the past and the fork Choice rules they get pretty um uh tough to follow because there's um this degree of like a self referential uh like a self-referential nature to them where you have like an attestation that votes simultaneously on the head of the chain and the part of the chain that's

finalized um the head of the chain can never diverge from like uh the finalized part of the chain so it can't like uh switch from that part of the chain but also the weight at the head influences what becomes finalized eventually so there's a lot of scenarios with this um especially related to like if different views of the chain like different nodes are seeing different messages then they might have different like work Choice States and these states might need to be resolved if for example like there's a temporary Network partition um so tools for Network level simulation are really helpful for this

um cytosis is useful for this so is antithesis um I think with the anticip antithesis you can even manipulate like the timing of each node and also like the latency of of messages across the network um which is cool um and then this tool blobber is one made by Mario at the EF which uh I thought was really cool we had found an issue in Lighthouse where on one of our denb test Nets we were trying to um import a block from a pier using the same rules that we use during gossip

propagation and uh what happened was our gossip rules um prevent you from importing like the second message you see um so if you got the second message then the first message was the one that was actually voted to be valid you'd have to go get it from someone else but if we weren't able to import this block uh this this this blob then all of a sudden we can't import the block generally so then our chain or Lighthouse noes we get stuck wouldn't be able to like proceed with the correct chain um so I brought this up on a call and Mario he he went out and built a tool to test not only this specific scenario but also other types of

scenarios related to um blob message ordering and timing and validity um so that that was really cool uh and this next bullet point is Hydra fuzzing so Hydra is this tool that Michael from our team is created and I sort of alluded to this before but um what it does is it creates it it tries to Target Fork Choice specifically so it creates valid blocks and creates valid attestations and it's aware of the types of thresholds related to um finalization uh in order to like get you near those thresholds and also induce

Randomness around those thresholds and this can help with um understanding if Lighthouse might behave differently in in more extreme situations uh to other clients or um if we have a fork Choice bug that we can't recover from um but yeah okay and so this other um aspect of client development is staying online um so I think in terms of like uh resource usage a lot of times when I think about resilience there are others on our team that are more focused on for example like making sure like a random packet can't crash Lighthouse um

but for me I work a lot more in um this Fork Choice area where like we're dealing with M States we're dealing with what happens when we have unexpected reorgs with um like maybe five different chains when we only want to store like at most four CH four states in memory um typically we try to just have one state and in like normal processing we're just mutating the state and keeping it in memory um but every Fork essentially adds one state for us but there has to be some which like we don't want to use Ram um and then that thres threshold will

either have to like just dump a state and later if we end up needing the block related to it um load it load the most recent State on that chain from the database and reapply blocks so in that scenario you're like you're trading off ram in order to um do more processing so you're using less Ram but doing more processing and um loading more from the disk uh like uh reading a state for the disc for example is something like I think 250 milliseconds up to like 500 milliseconds maybe longer and then processing a block

is somewhere around 100 milliseconds so you have to think in terms of like okay if we save 100 Megs of ram like what are the odds we might actually have to do this much more processing later that goes into a lot of like the um caching decisions um and then there's also these same sort of issues with um like timing and message Pro propagation related to just staying online generally and this is because like a lot of these scenarios where we have uh like either Network partitions or we have um messages being withheld and released later these are what can induce the sort of worst case

resource us usage in our clients um so the same sort of tools are also good for understanding worst case resource resource usage and that's something we like end up focusing on a lot of the time um and then this last Point tree States is actually um an upcoming set of changes that Michael from our team has been working on for a while um and to illustrate it I'll bring back this slide um and so here you can see that every time we mutate a state we're essentially treating it as a new state if we do a reorg here from uh block two to block

three we have to dump state two and we have to reload State one to find out what state three was um but but in tre state we would treat state zero as like this is the base State and then every time we apply a block we just track the diffs um so this would allow us to use um a lot less Ram instead of using like 100 Megs per state we would end up using uh I think aund we'd be able to store 128 States in that same uh amount of ram this would make us like a lot more resilient to reorgs it allow us to do a lot more caching we wouldn't have to

worry as much about cash Miss cash misses um and then on top of that we can have the same sort of storage structure in our database so um the states that we are storing um and the ones that we have to keep around when there's long periods of um non-f finality um those also get smaller um yeah this is an exciting set of changes M's been working on for a while um our tenative plan is release this after the denev hard theb hard work um at some point yeah so that's about it for me thank you thank you very much Sean um yeah so it's a very complex and very Niche topic

so yeah very impressive that you can make it so easy to understand and yeah digestible um given that it's it's a very Niche and Niche topic do you have any suggestions on uh on where people could start to read more or like research on it if they uh if they're interested in it um so I really like the annotated specs um there's a set of annotated specs that um falic wrote um and then also Ben Edgington has um I think it's called the E2 book um I find those both

really useful um and then other than that for me personally I didn't really get a deep understanding about a lot of this stuff until I tried to actually make some of the changes um that's sort of how I fell into client development I think is that I hit a wall with um yeah I don't W with how much I thought I could learn by looking at the specs and then trying to make a change you end up learning a lot more yeah okay um there's been a few questions uh mly asked um what kind of problems does uh Hydra fussing detect yeah so we can actually use it

for a lot of different things I I'd say the general idea of it is trying to um cause as much chaos as possible um with valid messages like messages that Lighthouse or any node has to process any consensus node um and so these would be like add a stations or blocks specifically where we're looking at the consensus information in a block not so much the payloads um and the idea here would be to induce reorgs as much as possible and this would be testing our Fork Choice implementation so deciding like which block everyone should vote on for the head um to try to like restore

stability um but then on top of that it's also really good for us to understand what our resource usage looks like in the worst case because a lot of our caching is based on like this is how we expect the chain to run most of the time with some margin of like this is how much RAM will dedicate to uh ensuring that in a bad case we can still run fast but then there's some limit to that where like we just can't store everything in memory um and so understanding those scenarios better is also really helpful from Hydra ohol so um Tom had a question I'm I'm not

entirely sure if this was targeted towards Amad or or you I didn't see it but uh I'm sure you can answer this in any case um so in the case of service um if you want to have as much impact on the um on the network itself um would someone be attacking the execution clients or the consensus clients um it's it's a good question I think maximal impact you probably would want to Target gu just in terms of like the usage um I think you could end up doing in

I would say equal amount of damage because on either side you can cause both a consensus split so if you can figure out that um something in the consensus layer is is different that would cause either like there notion of what is valid to be different then you can cause a consensus split um and also uh resource usage um for us like we don't have anything built in like the evm so I I personally don't understand I don't know what the um the impacts of of like understanding the doability of the EVR for

example um I'm more just focused on Research usage generally but I think the execution layer also has those types of problems yeah yeah sure um Marius uh has a question um so he's asking about how e clients store diffs between states um and he thinks that that's not possible on the consensus layer because of the states being able to change completely between blocks um so on the consensus layer do you think it would be possible to bound the changes between blocks um so let me

think I'm not I'm not sure about the ability to bound the changes but what I would say is like the risk of the CH the state changing completely is lower on the consensus layer because the state is only 100 megabytes and we are already holding multiple States in memory anyways um I also I'd have to think pretty hard about the scenario that would actually cause the entire state to change at once but yeah okay um one final question then um do you do you have any favorite uh Lighthouse vulnerability

that uh showed up throughout the years um okay let me think liability uh so I I feel like I'm more proud of having found a vulnerability in the the consensus specs and that was related to Fork Choice um and that was related to um a specific scenario with um proposer boost where you're adding an extra artificial waiting to whoever the block proposer is when trying to figure out what the head is um so I was proud of that one in terms of like it's pretty hard to find

Beck bugs because people think really deeply about the spec and then two just because work Choice itself is really difficult to reason that so I spent like I don't know eight hours just thinking through like specific scenarios um because Lighthouse was like not acting how it should when I was trying to run against the EF tests but yeah okay thank you very much Sean um I will hand over the word to M yes so thank you very much Mario ahed Sean for the Fantastic presentation I really learned here thank you Freddy for moderating in such a nice way and thank

you Nathan we will actually post the hopefully tomorrow the the recording and thank you everybody for attending and looking forward to seeing you in the next presentation thank you everybody for attending and thanks again Nathan for organizing and David thank you everybody bye Happy New Year