ZK8: How to build a private DEX - Henry de Valence - Penumbra

Transcript

foreign who's going to talk about a topic very near and dear to my heart which is how to build a private decks harder than it seems anyways uh give it up for Henry [Applause] I've got front run um all right so uh ah great so yeah um my name is Henry uh I work on penumbra I'll be giving you this talk on

how to build a private decks um and uh to kind of start off um you know this is sort of uh look a little bit of a joke but also a little bit real um we have this somewhat sad State of Affairs which is if we imagine a Venn diagram um where one part is is change that people use and then the other part is chains with privacy then it's like almost two circles and the intersection is like basically the people at this conference right and I say this as somebody who spent like the last five years trying to build privacy infrastructure I don't think this is because of lack of trying it's just like a very hard problem and actually a lot

of the most um uh successful deployments of ZK that we've seen so far have actually been on the scalability side rather than the Privacy side and so I think that points to um some kind of issue of like we're just somehow not thinking about this right or something right um and for me uh and you know how we got started on on building penumbra the the thought was like okay well what if we tried to build just like one useful application so I personally am not smart enough to be like okay I'm just gonna sit down I'm going to come up with this like perfect idealized programming model for private programmability it's gonna like burst

out of my forehead and that'll just be like the perfect thing right so what if instead we just try to build sort of one thing that has some kind of fixed functionality figure out how we could make that useful and then after that see whether or not we could generalize anything and that's basically what this talk is going to be about it's about some kind of vague ideas of how you could generalize the approach that we've come up with into some kind of more more General programmability system so just from a kind of like product perspective how would you provide useful functionality on a private Ledger if you're going to build one application what would it be

um for us we kind of landed on trading as a as a use case to get started and the thought process there is that trading is in in one sense it's kind of this like minimum viable uh transaction a lot of the time people think of like oh like payments that's the simplest possible thing you could do on a blockchain but actually a payment is really quite complicated because whenever you have a transaction the counter flow of that payment is going to be recorded off chain so if you want people to use it now you have to sort of build and scale this whole parallel social infrastructure you know trading you can have a transaction that in itself is economically useful for

that person and trading is also really interesting because it's a an area where privacy thought of as control over the disclosure information is not just this kind of abstract principle or you know human right or whatever but it's something that has a very uh quantifiable near-term Financial value people don't want to get front run um and so you can actually sort of potentially move out of oh we're gonna like you know build this privacy product to we're going to build something that is better because it is private so that's kind of uh what we've what we've landed on so what we're building is this uh private proof of Stakeout one uh that has a cross chain shielded pool that can

record any type of asset and an integrated um private decks and so before getting into sort of what's the the interesting technical um idea or challenge uh the next thing that I want to try to convince you of is that useful blockchains revolve around public shared State this is actually really relevant for building a private system because if the goal is oh just like make everything private I think you end up actually not building a system that's very useful so for instance considering a market uh if you made the entire state of the market completely private you end up with something that nobody really wants to use because nobody has any idea like how much

liquidity there is with the trading volume what kind of prices are people getting instead you have to figure out like okay the the point of the blockchain is to be this kind of coordination fabric if you don't have any public shared State then all you're using it for is some kind of like messaging or settlement layer for some kind of off-chain coordination and that's basically like a kind of fancier form of payment channels which don't really seem to be that useful so how do we get then to being able to have private interaction with public shared State I think this is kind of the fundamental like big thing that we have

to figure out in order to build useful private applications um and to explain a little bit about why this is such a problem um we can kind of recap uh what may be familiar to some people in the room the the key differences between the state model of a transparent or a shielded chain so let's say we have this transparent chain like ethereum the the basic kind of mental picture is that we have this big Global mutable State we have transactions that we've come to consensus on and you know we're just applying them in sequence and each transaction is like locking the state doing stuff unlocking and then we do the next one and so on um in order to to make this private we

first have to move to a composable state so I guess this is kind of like rewinding backwards to to bitcoin but the idea is instead of having like one big Global State we'll have this tree of little State fragments and then each transaction is only going to have you know some determined set of inputs that it consumes and it composes those together to form new state fragments that get appended into the chain State and the reason that you want to do this is that you can replace all of the state fragments with commitments to the state fragments and then you can replace the actual sort of clear text transaction contents with a proof of a valid State transition but this diagram is like a little bit of a lie or it's not quite

capturing something important which is that what's actually happened is that we've you know successfully moved all of the user State off chain and replaced it with these commitments and the only on-chain thing is just this this proof pie um but that means that the execution has moved off chain and so this only works when there's no shared State because if for instance uh I want to interact with a DEX that say I want to trade on uniswap if I'm going to move the execution off chain then one of the inputs to my state transition that I'm supposed to be

supplying this kind of sealed verifiable proof of is going to be the state of that amm and the state is just going to change out from under me um and I also can only be touching my own State I can't interact with anybody else's shared state or anybody else's State fragments because I don't know what they are and I'm making ZK proofs so I have to have knowledge of the whole complete State transition so what this points us to is actually that we need some kind of better computational model for concurrent state um and and the big idea is uh if we think you know if we forget that we're

in blockchains right just imagine that we're doing systems programming and we want to build some kind of like really high performance like parallel concurrent system the way to do that is not to like take all your app State and like put like one big Global lock on it right um it's actually not even to you know have like finer locking like you could do that I guess but a much much better and and cleaner way to do concurrency is by having uh independent actors that are each controlling or responsible for one piece of state and then you have message passing between those and that's how you avoid the need for some kind of explicit data synchronization

so what would that look like in the context of a blockchain well we could get you know some kind of actor model for blockchains where instead of having each transaction uh acquired a global lock on the whole state and then do some synchronous function call we could say a transaction is going to pass a message to a contract then each contract can execute exactly once per block on input of all of the messages that were sent to it in that block and this allows the contract to do batch processing if it wants or you know it could do whatever kind of arbitrary application logic it it wanted so for instance if you had an auction you could

sort the messages by uh the the bid or something um and the user state is going to execute asynchronously off chain using ZK proofs so as a side effect that I won't get into in this talk this also in my opinion like mostly solves Mev um but the the interesting thing here is like okay what when I say we're going to do some kind of async execution of the user State off chain what does that look like so we're going to start instead of having um the ability to have like all of our inputs and outputs we know that we're trying to communicate with some contract asynchronously so we need to have some

kind of way to take our private inputs and then send a message out to the contract but at that point we can't continue executing we can't continue preparing this sort of pre-sealed State transition because we haven't got the result of the contract execution we haven't received a message back so what we're going to do instead is we're going to Mint to ourselves a an nft that will record the intermediate state of our execution so here I've got this like kind of little schematic diagram the circle is meant to represent like okay that's going to be like the state of the the amm or the decks or whatever and these uh Holo boxes are the outputs

that we know we're going to want to Mint later um and so the idea is that this state nft is is modeling that future computation and uh technically the way that you do that is you just miraculize all of the uh all of the execution State using a snark friendly hash and then you use that as the asset ID for this nft that means that later after we get a message back from the contract after you know in some later block maybe the next one um we'll we'll get this message from the contract and then we can consume the state nft and then open that mercolization of the uh intermediate execution state

show that we're modeling the correct future and then use the message from the contract to Mint the private outputs that we were supposed to get um so the the trade-off for this is that the execution is now potentially spread over over multiple blocks but you know on the other hand you can actually have private interaction with the the public shared state um so as an example of how you can you know that's a fairly General kind of abstract uh description but to see how this works uh concretely let's see how we can do some batch swaps on penumbra so this is looking at the the private State part of it so we have our our private input that some user wants to

contribute to uh at the batch swap for some block what they're going to do is actually encrypt verifiably encrypt their input amounts with a threshold key um that's controlled by the the validators that's going to be their message to the the contract effectively um they mend to themselves this private state nft um and now that the chain when it's processing these messages will have this batch of inputs the threshold encryption is actually going to be additively homorphic so they can sum up the ciphertexts compute an encryption of the batch total and decrypt that and now the only thing

that's revealed is the aggregate flow into this trading pair in that block at that point you can execute the Dex contract just once per block get the batch outputs and then that can be fed into uh this secondary swap claim transaction that consumes the the user's Swap nft and mints their private outputs but what's really cool about this is that because we're actually executing each uh uh each contract you know in this case it's a hard-coded decks but we're only executing it once per block and so that means that we can amortize the computational cost of whatever work we do over all of the transactions in the block because we know that we're only

going to have to do it once and so that lets us actually do a like a substantially like cooler and more sophisticated thing which is instead of having to do kind of middling like gas optimization of every single transaction that we're going to be interacting with this decks like a thousand times a block we can just group all the inputs of all the Assets in different trading pairs batch them all together decrypt them and now we have this kind of like global view of all of the trading intent in the block that we can just globally resolve uh using optimal Arbitrage like get all the prices in line we can do like multi-hop routing and all of that is happening only once per block so like if it takes a little longer to run it's

like not that big of a deal um at the end of that you get all the prices and now you can feed back into the uh the the second part of that um private State flow so this is pretty cool um we're you know still obviously just focused on the the one application to start but I think this is a kind of interesting um taught experiment of how we could build like a really in like useful asynchronous programming model for doing um interaction between private user State and public chain state the other thing that's pretty cool is

that actually this uh works so if you want to go and like use this the full kind of like cool globally optimal text part is is not implemented but the sort of interesting part of how do you interact with the the state model how do you have this like uh you know future in the shielded pool um you can go play with that at the moment if you go to guide.penember.zone there's instructions on how to play with the test net so cool foreign [Applause] are there any questions for Henry really everyone knows how to implement a

private decks already and it's uh perfectly good all right hey um thanks for the talk you mentioned that you're spitting out the execution of the contract over multiple blocks is it locked in for those multiple blocks or can you so the sequence anyway the the contract execution is once per block what is spread over multiple blocks is the the user state so for example in the uh in the swap application as soon as the user submits their or when a user submits a transaction with a swap in it as soon as they see that that transaction has been included in a block

they have control over this swap nft that already has the effect effectively the value of whatever whatever their outputs are it's just that they haven't sort of driven that to completion um so there's still this kind of atomic execution on the public chain state the part that takes potentially a you know a second phase is is sweeping those uh outputs from being represented as here's the you know receipt for your participation that you could redeem to actually redeeming it but either way the execution like has happened by the time that they uh see that they have control over that note uh yeah for someone to say this is super

cool thanks for sharing um I was curious like on the execution layer besides being able to support like this function call at the end of the block are there any other changes and then yeah first um I think yeah the main thing is the ability to have uh an end block phase um the other piece that is kind of a little bit you know TBD is if you weren't doing this as hard-coded functionality you have to think about like okay what does that trust model look like right like for for us because the Dex is kind of integrated functionality we can say like okay well

um here's a way that you can like burn these tokens mint this thing later you can like uh do a mint of your output tokens and prove that that's honest and that's all like fine because we're pretty sure that like those State transitions are correct whereas if you're going to allow like arbitrary user code then like how do you know like you you can't necessarily you need some kind of fault isolation and the the kind of interesting perspective there is like there's actually some kind of analogy lurking in the shadows between this and the kind of Cosmos maximalists like everything is an app chain that communicates over IBC except that instead of doing IBC between these

things it's like all the messaging is just done on one chain so it's sort of like app chains on one chain yeah I was asking because I was curious maybe like I know in your case it's really important to have like this like function called the end of the block but I think your program model like for example could work on ethereum and maybe you have some like relayer node that like calls some function to like do the end of the block transaction or maybe there's like multiple block latency or something like that but yeah just thought it was super cool you you also could um in principle you could kind of Multiplex this type of execution model over some other system in the way that you describe where people can make a transaction where the transaction is like registering the message or saving it in a queue or whatever and then

there's some kind of thing that's going to trigger the the end phase um if you want to have low latency I think that's kind of a challenge um and you might end up with like sort of weird issues where you're kind of like aliasing the time resolution um the other sort of philosophical point of of wanting to have this kind of batched actor model is the idea that um and this is kind of getting into the Mev Point um at least in my opinion a lot of Mev is the result of uh people designing mechanisms that operate at a higher time resolution than is actually provided by

the consensus mechanism and then because there's this like attempt to sort of have a higher sort of fidelity of time than is actually provided by consensus you get all this like kind of mechanical Arbitrage about the intra block ordering um and so the other piece of this approach is it's sort of saying like look you know you shouldn't build mechanisms that can operate at a higher time resolution than is actually provided by the underlying um consensus and so really you know you shouldn't even be doing any processing in the transaction handling at all like all of the processing should be in the end block Handler that's just that's just a philosophical

idea not an actual like concrete system design yeah idea like it doesn't actually work like that there's reasons are there any more questions um what uh proving system is the number using is a graph 16 or bulletproof space design um so this is kind of funny actually um as you can maybe tell from the talk the the emphasis that we've sort of had so far is like the hard part is figuring out the data flow and the data model um and so in order to be able to like iterate on that we've actually been

using a really cool proving system that's called uh protobufs in which you take all the witness data and you just like put it in a Proto and then you like don't look um and you know that sounds extremely stupid and in a way it is but also you know the the thing that's cool about it is that you end up with um uh a software object that has exactly the same interface that you're going to end up having when you have uh circuits but where you can like just change it you know in 30 seconds if you decide that you want to change your data modeling so for example um like in the in the coming weeks we're probably going to like redo the whole

data model so we have like one 28-bit note values to have better compatibility with sort of eth issues um and that type of change is like a lot easier to make uh if you don't actually have to go and like reprogram all your circuits so sometime this fall we'll then take those as a sort of like machine executable spec and then um turn them into grout 16 proofs which is probably the real answer that you were looking for well I have a question actually this is more of a philosophical point but do you think the same trick of kind of batching execution can actually be achieved on

different other like protocols that aren't just X's you know because it feels like it relies a lot on the fact that you can you know um kind of batch execution but is this all do you think it's more generally possible and we just need to do more research on it or so what I would say is that there's there's kind of two different uses of the word batching that are mixed together there um one sense is from the kind of specific decks application sense there's a an idea of like okay we're gonna batch all of these like trades into one trade and only execute the one trade but also from a just a general like system design perspective there is some kind of batch

execution in the sense that if we were to generalize this to arbitrary contracts each contract would be receiving a batch in the sense of like here's a list of all the messages um and then depending on the application they may or may not be able like it it's gonna vary how much work they're going to be able to overlap between processing of each individual items but in general like if the interface is that you get all of the messages then your application can choose doesn't make sense for me to batch can I like over can I reuse work

between different items how does like what is that for my specific application but if the interface that your contract has is like I'm getting all these things independently then you just can't do that at all um so I think in in every case it's like definitely not worse and probably better even if you can't necessarily get to the sort of you know so like hilariously optimal thing as on a on a batch swap where you can just execute one trade all right we have time for one more question if there are no more I think uh I will I'll ask for Anna to come on stage no I think no no more questions

really zero okay well thank you Henry for presenting how to privatize a decks