Analyzing Incentive-Mechanism Attacks with Reinforcement Learning

Transcript

okay hey everyone uh welcome from the ethereum bazaar uh my name is sham sridhar i'm a research intern at the robust incentives group which is a research team at the ethereum foundation and today i'll be talking to you all about some really fascinating and cool stuff like reinforcement learning proof of stake consensus and the ethereum blockchain and all of this is based on a project uh done in collaboration with my wonderful supervisor barnaby munno really grateful to be here in front of all of you and without any more delay let's let's just jump right into it oh here's some really cute unicorns okay so uh the first question we really need to ask ourselves is what is reinforcement learning or what exactly does this word mean uh so if you ask

famous researcher steve burton he actually defines this term as a framework for learning how to interact with an environment or with experience so this is actually very much inspired by nature uh this is how animals learn things they go through the trial and error method of doing something seeing the consequences and learning from it and and success subsequently uh improving it but what exactly this entire textbook definition really boils down to is this that the entire framework can be described as an interaction between an agent and an environment so in this interaction the environment is actually presented to the agent in the form of a state so for example if you want to train an agent how to play uh the game of chess uh this can simply be the position of each piece in the

chessboard next the agent gets to have some amount of influence over the environment or through an action the action in our text example can be anything like moving a piece on the game board as per predefined rules of chess and now before the status the next state is presented itself to the agent the agent also receives back what you get as a reward so this reward can be something like the number of opponent pieces the agent knocks out of the game or just about anything that returns a positive value to the agent every time it is closer to winning the game so essentially through this interaction the agent gets to now devise a policy for every given state action pair in order to maximize its rewards so this sounds simple enough right uh so what what exactly are this application so far

and and what are some of the really cool and amazing amazing things that you know rl has been able to achieve so far uh so if you look at some of the magical examples or the magical applications that we can talk about uh the first one we can basically think about is autonomous driving uh so several researchers across the globe have actually used this rl framework to develop self-driving cars uh where the agent actually makes several decisions uh such as the speed at which to go to the areas in which it can drive to really avoid any obstacle or to you know stay clear from uh you know the restricted areas and and so on just these are just to name a few uh but this framework has actually been uh very useful in actually developing some prototypes for these self-driving cars uh another common application is

automated trading uh where the agent gets to decide uh when to hold buy or sell various stocks bonds or just about any financial asset that you find in the market but what we really want to focus on and what the main application we are concerned about in our research is the third one that is to analyze attacks on blockchain incentive mechanisms in fact a team of researchers from different universities have actually employed this framework before to teach an agent uh the selfish mining strategy on bitcoin and proof of work ethereum uh so uh like i just mentioned right like what is this word blockchain incentive mechanisms so almost blockchain almost all blockchains have what you call incentive mechanisms which are in place to basically provide rewards for network

participants who play a crucial role in keeping the network up and running so in the case of proof-of-stake ethereum we have the ethereal specs which basically lays down all the rules of the protocol in order to ensure you know that minimal complexity uh is met liveliness is maintained that a large participation of validators is insured and just about a lot of these design objectives but essentially what would these network participants uh like what what these do with the network participants is that they are now incentivized to behave in a way that doesn't uh actually cause any harm to the system but in fact uh does good to the system but if you look at proof-of-stake ethereum itself uh which is quite uh which is still in this nascent stages it's not even a year old uh there are several deviations from the intended

protocol as we see here such as you know these low-cost attacks on ethereum 2.0 uh these ebb and flow attacks these decor flip-flop attacks and so on which actually makes sense for the validators to do since they get some positive reward out of this rather than just being rule abiding law-abiding validators uh so that's what we essentially need is a system that can one potentially help us figure out these more complicated deviations these more complicated attacks and number two provide a robust methodology for actually back testing the protocol every time that it is updated to see if these attacks are still possible or not with the updated protocol so if you look at the current system and see how caveats are identified there are roughly two steps which are involved

so the first one is intensive theoretical analysis which is done by researchers all over the world and once this analysis concludes with the result being that it is actually possible to harm the network in some way or the other which makes sense given the current protocol rules we have a protocol upgrade that actually fixes these caveats which basically kicks off however we believe that this pen and paper analysis has its own limits and we feel that a more systematic framework and it's a more systematic approach could actually help push this further and can help or push our cause further uh so naturally the next question we ask ourselves is if we were to integrate rl techniques right how uh would attack detection now look like and what would the resulting framework essentially be

so this is exactly what you're seeing in your screens here so if you look at proof-of-stake ethereum in general uh so all the role that the entire role of the validators as well as the beacon chain can basically be reduced to this framework of an interaction between an agent and the environment so the agent is essentially the validator in this case uh the environment is of course the beacon chain the various actions that the validator can do includes you know sending in proposals sending in attestations uh doing nothing uh when it's called on to propose or test or propose blocks but you know release it later and there are a bunch of different actions which the agent can do and if you look at the state so the state that the beacon chain can provide back to the agent can be anything such as the

current head of the chain uh who which validator is up next to you know uh propose or a test and the balances of all validators and essentially this anything that describes the state of the beacon chain uh and the rewards can be uh essentially anything like you know or the most obvious one is the earnings since that uh since we know that the protocol uh rewards uh participants for you know validating and proposing on time or it can also be you know the number of blocks free or if you want to see uh things from the lens of a person who does what who just wants to do anything to like harm the network no matter how much harm he causes onto himself and so there are various ways in which we can actually adopt this framework for the proof of stake ethereum we can chain and

this is exactly what we aim to do in our research uh so before moving on i think it would be important and relevant to introduce the beacon runner at the state so the beacon runner is basically an agent-based model and simulation environment of proof-of-stake ethereum which is being used to run various experiments to see how various situations would pan out according to the current rules of the ethereum proof-of-stake consensus layer so uh this is roughly our plan this is where we are this is this is a very oversimplified version of a road map and this is essentially uh you know the state at which we are right now uh so the first obvious step for us was to actually simulate the attack on beacon tunnel uh and the second step that we also completed is to basically create a very basic uh gym environment so jim is

this open ai framework which can be used to build uh custom reinforcement learning environments and so we actually built a very basic uh gym environment with the basic proof of stake uh concepts as well as a centralized client implementation of the in the entire beacon chain so the next step is to basically improve this model further uh decentralize the entire model have each validator have their own view of the chain and the final step to actually test whether or not this approach that we propose will work to you know number one detect attack so number two be used in back testing protocol upgrades is to actually train uh the agent with a pre-existing attack so we can see if the agent can actually successfully execute an attack that we know uh currently can work in the beacon chain and the attack

we have actually chosen for this is the attack malicious reorgs as described in noida 2021 so that's about it from me thank you very much for tuning in guys uh thank you very much for being here uh you can check out the more or more cool stuff that is done by the robust incentive group or in ethereum.github.ioslig and i hope you really enjoy the rest of econ thank you very much