Highly Available Ethereum Nodes with N-version design

Transcript

[Music] [Applause] [Music] thanks well uh hello and thanks for the nice introduction so uh I'll be talking about this uh research paper that we published um last year it's about highly available blockchain notes with inmersion design but the Prototype that we made was entirely based on ethereum's execution client um so I guess I can skip this slide because of a very nice introduction so what's the problem that we are trying to solve when we say okay let's build a highly available uh piece of software highly available client and it's

that software fails all the time uh unfortunately we live in a not unfortunately fortunately we live in a world where software allows us to do very much uh interesting and very much easy things but you can see that software just fails all the time and you know it's uh it's a bit of a mess uh sometimes right and this is this happens a lot because of boxs in the the code or not even in your own code you know they can be boxs in your supply chain BS in your operating system Hardware failures like it goes up to the physical uh limit right where you cannot control that when and where your software will fail so blockchains of course and are

software and these are some extracts from a a pretty nice report done by uh theum Community about the the the reasons of why ethereum execution clients fail and you see that there is no real consensus here so most of the fails are bugs and random Corruptions and taking a step back so even blockchain infrastructure fails so you know we have seen uh large players of the blockchain infrastructure having large outages which is not nice for you know services that depend on on on on on on this infrastructure right so as a summary of

the problem we're trying to address software fails and you cannot be 100% sure that your software and in this case your blockchain uh nods or loction infrastructure will be working correctly 100% of the time so it is our uh it is our our goal to harden uh this infrastructure with a focus on availability all right so let's move on to this end version design that we that we propose so let's start with a normal approach of if you were uh like uh up developer right which you you want to run your own node to deploy your own uh

web Tria app right you you run your node you connect to it but you know then you upgrade the version of the node and then you have a a b in job and then your your service cannot work right so all right let's say that you are a little bit more cautious and then then you connect to a service that has several noes but these other noes that are not yours have let's say the same implementation let's say if it's ethereum they're all gu right and then there's a transaction that comes from the network that breaks all get clients and then again your your your your uh service your application doesn't work

right then where it comes this uh idea of end verion design right so it's a decade all idea proposed in the 70s by this person uh whose name I cannot pronounce and it says that uh you should design software systems in a way that you design only the spec and then you can you can implement this spec differently several versions several times but uh using different technology different te much like it happens with uh ethereum's uh execution layer and and consensus layer right so then what we propose is that you could run a sort of this some

sort of this type of nodes right right where you have uh sub nodes uh where these nodes have different implementations if there's the error that I mentioned earlier about upgrading it then your your your application will still have access to the to the chain and then if there's you know something some error with the other one then there's a higher probability that your your own application will have access to this chain right and the one of the one of the difference about this design is that it should go through a proxy or a broker of some sorts that has some design some some decision power and where to uh

redirect these calls or the the the the traffic from the external application all right so past the design so we want we we kind of know that uh ethereum's client diversity Works we've seen it working but um there is no academic data on how well does it works like we want numbers we want percentages we want we want hard data and in this case we implemented uh this prototype using the four major um execution clients of etherum and we deployed it uh sort of like this right so and these versions are a bit

outdated because we did this experiment last year so this we could do it again to see how much it has uh improved or or not so yeah but okay let's go to the experimental part of the of the talk so first we want to find out a baseline okay if we are saying that inversion design is the key to reliability then we have to decide we have to measure how different is the behavior of the different uh ethereum no the ethereum clients right so to do this we want to measure how they react to um

unstable execution environments and uh to do this we measure the the ethereum clients I mentioned before with 20 realistic fault injection models these fault injection models were based on operating system errors that we inject directly into the running processes and we only care about one metric the availability of the of their rpcs so we spawn an an application that will read uh from it will make requests through Json RPC to these clients that are being uh actively um injected with with operating system FS right a little bit on how we inject these folds so we put a layer of

uh this injector layer on top on on top of of the OS so that our process does not make requests directly to the OS to the operating system uh to read from dis or read from Network or read from memory so it goes through the injector and then the injector decides if this calls succeed or these calls fail and with with which error they will fail and with this we can simulate this Hardware um failures operating system box all right so this is what I just told you so this is the design for this first question that we have and we measure the availability all right so what's the

result of that so here's a table uh with lots of numbers that at plain side don't make much sense but uh if you look a bit closer so here in this red squares I've highlighted some unique behavior of different clients so this is an aggregation over all the over all the errors that were produced during the experiment and this is the count of how many times this error occurred so you you can see that for Bess here this error invalid character response uh in the that was uh present in the calling uh client to the to the to ethereum note uh appeared almost

300,000 times while in the others um in the other uh clients it it didn't appear and then we have some overlap of Behavior that's also true they are not perfectly uh you know distinct more numbers here uh this is an aggregation of um RPC we have in this column uh the RPC method name and here the numbers mean uh how much how available they were under uh under fault injection and each columns is uh it it corresponds either to a client so we have all the four clients and the three most impactful

fault injection models so remember that we have 20 fold injection models so these are the three most impactful so the same thing here we can see that there is uh there is uh different availability scores so this clients do not behave uh similarly they they they are different by Design another interesting finding here is that given default injection models the standard uh the the standard deviation uh over the different uh RPC calls is very low so if we if we just measure the ability of the availability

of one RPC to keep uh working so we you can we can generalize to the other uh RPC end points because the standard deviation is is very low so you can see that these columns are have roughly the same numbers in each column all right so to summarize that first part uh how different the eum clients react to unstable execution environments so a bit differently actually a lot differently and the important thing here is that these errors that we see are non-coincidental so that we can actually build this prototype that I mentioned in the beginning so it would be worth to

build second question that we want to address uh how much is the availability of aeral clients affected under unstable execution environments so here we deaggregate uh the data a little bit more so we want to see the data for each fold model for each client and I also want to uh qualify a little better what availability means so either so traditionally we say okay the RPC point point is available or or it's not but sometimes on fault injection the RPC point the RPC end point is available but it's returning bad data it's returning old data or even garbage right so this is what we call as degraded

availability another huge table um so here uh we can see that again the nodes the the client implementations um behave very differently so we can see here that get it's very good maintaining availability oh sorry uh for um most of the F injection models but actually when you take the most aggressive F injection models which are the last ones Bess is the one that has better availability so that's interesting another interesting finding is that even though Aron and get share a lot they have a lot in common the errors are still different in terms

of availability so different availability scores and then well nether mind it didn't perform so well in our test so it starts degrading uh a little bit fast uh with the first the first fold injection models uh so in summary uh these are the averages of the drops in availability that we saw uh when we injected this 20 fa models right uh yeah so we can say that availability the error is noticebly and but it depends right what kind of Errors you're injecting and which client you are injecting them all right so we go back to our

prototype to our end version prototype and we want to measure like is it is it worth it to to build this thing to have several nodes running at the same time as your you know as your entry point to reading data on the chain and we do the same right we we deploy this inversion Noe uh with the with the added fold injection models for each the fold injection modules for each uh client so in each of these clients we will be injecting simultaneously folds this operating system folds that will make Network call fail disc grds fails memory allocation fail right we also uh measure the impact on availability

like I said uh but we don't take this huge prototype and measure it uh what we do is like we are we first test with two simultaneous clients and then all possible combinations combinations of two clients like what happens if we use get and B get and nether mind B nether mind right all possible combinations and then we move on to three with the three implementations all three possible all and three possible implementations and then we move up to the full implementation which is using the for clients more tables more numbers more data and uh here what I can what I want to focus here is that there there is uh lots of

improvement so we can see that even from using two different uh models like already that you have an average of 94 uh% 94% available during using the same fults as we used for single clients and then you can see that for n equals 4 then the unavailability almost disappears so you still have a little bit of degraded availability but uh but you will you almost 100% of times you can access and can read data from the chain so so the question here is what is the cost of this so I don't so if I am

running my own nodes or if I am um a service provider running nodes like what how much does it cost me like it scales linearly right so it it's not that the cost is not trivial the cost is not trivial is lineal but it doesn't scale uh trivially as well so you can have combinations that are more efficient for Ram consumption or you can have combinations that are more efficient for disk consumption which is a problem right uh ethereum execution clients take lots and lots of dis um so to finish uh yeah compared to single implementation clients and with these specific fault injection models that we

tried we can take the availability from the highest performing ethereum execution client up to uh yeah so up to which was 91.1% to 98.4% which is quite a huge uh jump in availability on availability almost fully suppressed and the trade-offs are are kind of lineal so it as expected but they're not so that so easy to read all right so that was it so yeah etherum clients behave very differently they fail very differently

and you can use them to create your more resilient nodes uh with this like Leverage in this diversity this non-coincidental Behavior under uh fault under fault injection under unstable execution uh yeah so actually fun fact I run a lot of ter my execution loads so I don't I don't want to run this much notes again it was a lot of time and network consumed so 660 synchronized from scratch so it's it was quite a bit of work so uh a little bit of future work so we

like diversity in like software diversity at our team so one of the things that we're looking at right now is we only have four implementations right but maybe we can have more maybe if the even if the degree of diversity is a little bit smaller like what happens if you take one function of your of your of your program and diversify diversify that function so you could just technically have a different implementation that you can leverage to build a sort of system like this so we're looking into that this is uh yeah automatic and verifiably correct uh software diversification then another thing that we are looking into is um moving the client diversity incentives on chain so

right now we have some reliability uh incentives given you know slashing on the on the consensus layer like you don't want to uh we don't want to to to have a super majority but it's it's mostly about uh you know we should do this right but what if we incentivize this with onchain rewards something like slashing or some reward of some kind that if you run a minority client then you can actually earn uh a little bit more of stake or you know get slash less severely things like that all right so this was a talk about uh highly

available ethereum nodes in reality and uh with software diversity with client diversity so if you have any questions thank you thank you so much please a round of applause for cavier and his fantastic talk thank you so much right ladies and gentlemen any questions from the audience and by the way props sub to you supporting your friend from the front row I love that it's a good Community yes sir you'll have the microphone with you in a second thank you you also question I ask okay hi thanks for the talk um just a quick question in terms of the faults that you were intercepting at an OS level was it um only faults that were requiring uh

like a reread or a rewrite to memory or did you simulate any more catastrophic failures as communicated by the okay no yeah yeah yeah so uh how we selected the The Faults was a little bit more complicated than what I presented so first the the the thing we do is that we monitor all system calls and then we we kind of record the error rate of of how how often do they fail because they fail a lot so these rereads they happen a lot the network failures they happen a lot but they're handled well by the libraries or even by application code but sometimes they're not and this is like when we inject failures what we do is that we amplify this normal error

rate uh that we have and this is for all calls that happen uh for a specific client y thank you please right so uh in theault injection modules for example so I understand you intercept system calls and you return errors statistically for example the read example you showed but do you also have time based injections like for example read that succeeds but takes a random delay before doing so which in a threaded environment could result in a different causality graph or Pockets received in the wrong order uh yeah I I understand perfectly what you mean uh but no we didn't we

didn't test that so we only change the error code but it is a great F model like around with the the the concurrency of the threats is a very good F model like you can you can actually check if no if you have no Deadlocks if you have no simultaneous reads simultaneous wres things like that which is very good I should look into it thank you thank you great talk thank you thank you very much anyone else would like a question from the audience no okay I've got one mine's mine's much more high level I it's it's absolutely mind Ling what you've achieved and congratulations to you and your colleagues um uh at least in the Czech Republic um blockchain hasn't even

been registered at some of the technical universities um I'm curious when it comes to things like technological transfer working with the private sphere um whether you can make certain recommendations to concrete companies or concrete communities and how that kind of interaction between the academics and the and the entrepreneurs is basically coming along in this very kind of fresh new community uh yeah so I'd like to say that uh so we're very open to collaborations so yeah you can just contact me or the university directly or colleagues my supervisor he's very active in the in the crypto space and yeah we're very open to collaborations we don't move as fast maybe as you know as as startups or or you know companies that have lots of money but uh we're

there we want to we want to contribute and we want to contribute the way we know how uh which is with with hard data we know with facts that we can I spent spend you know two weeks three weeks looking at a table with a lot of numbers and try to uh get some insights from there and then that can be helpful for everyone the tough but rewarding world of Academia thanks to people like cavier we can take blockchain to the stratosphere thank you so much for your inspiring talk give him one more round of applause please and I'll see you very shortly with it another talk on this fantastic main stage see you soon