Your TEE Workload Is Attested, Now What? A Practical Threat Model for CVMs

Transcript

Um, all right. So, I will talk about T security. Uh, my name is Eager and I work at Flashbots. We build blocks in TE's. Um, and I'm both a part of web two security and also web free security. Uh, it's a team called Unrea team which does a lot of TR R&D in the company. Uh the goal is not to to to hate TE's in any way. It's just I'm sharing weaknesses and I'm uh this is just about security. So it probably will sound like a hate talk, but it's not. And yeah, I'm bullish on TE's. Uh they're great. And big shout out to Andromeda Fox who actually did a lot of um contribution to this talk. So I just realized that I have five minutes less than I thought.

So I'll skip recap. So we use Intel TDX specifically. Not all TE's are the same. and it has encrypted memory and a CPU module. Uh and it's basically the gist of Intel TDX. Uh the important thing for security is that in TDX everything inside the CPU is unencrypted. So all caches are running unencrypted and it's by design but it introduces a lot of um way to to to do channels. And then be mindful that network and storage they are u untrusted. So you should handle their security on your own. And the last thing in the introduction is attestation. So what it what it means is like it's what makes T special. Um it's

the idea that you can measure your workload, measure your firmware and uh there is a fused key that is in um like it it can be extracted from the TE and this key can be used to sign those things. So and verifier can verify if it's indeed a genuine workload that is running inside the TE. All right. So this TE test station it's also called called a quote. It's a little bit different but for the sake of this conversation is the same. So there are five things that it doesn't tell you. The first one is physical uh like who controls the machine and the quote doesn't tell anything. uh it only it has some clues which you can use to um to use to to get the physical information but um anyways next there is a TDX

module itself and it reports only its version numbers it doesn't report if it's updated or outdated or something like this then there is host VMM that can observe your T it can observe your CVM um it's uh stands for conf confidential VM uh so virtual machine that is running inside the T. Um, and at station also doesn't have anything uh to know about it. It only attests what's inside the T inside your CVM. Then there is a gray zone. Um, and the workload itself. So if you will put the malware inside your CVM, then it will be a tested malware and nothing more. Right?

Let's start with physical. There are two physical attacks. I will skip most of the info for the sake of time, but they are quite nasty and they they don't necessarily break the TT TDX or T encryption, but they can influence it. For example, the Drop uh it's particularly nasty. It can flip uh flip the bits in the specific region and usually we don't care basically because the bits are encrypted info is encrypted. But there's a debug bite uh debug bit that in inside t that uh specifically in tdx it enables the decryption of the memory. So host can see all everything inside the workload.

So those attacks they don't particularly break the encryption but they influence it. Uh and then there are supply chain attacks as well which uh we have not seen for TE specifically uh but we have seen them for hardware wallets and other um other hardware and then there is a vendor alliance. So this attestation scheme that I was explaining it relies on the secure facility in Intel's uh somewhere that um contains private keys that are used to verify that okay your your workload is was uh running inside the genuine TDX machine and it's not just something else. How to how do we solve some of those? So there is a thing called proof of cloud. uh it is used the PP ID and PP ID is just um secure not

not secure but uh it's just a line inside that station that is it's normally it doesn't change unless you do a factory reset uh to my understanding at least and you can come to the host and see the PP ID you can compare it and then host provider can have an API when you you you because you you see PP ID in that stationation right and you can uh go to provider and ask hey do you do you know this machine with this PP ID and if the provider will say yes I do know it I have it then it's fine and there is an alliance of uh of teams uh you can go to this website it's called proofofcloud.org or a G uh and they they they they did exactly it's like the realization vendor agnostic of this thing. So they go to data centers they

check PP ids on the machines and uh you can use this uh their API to query your um workload and see if this PP ID was uh marked by them as trusted or not. uh it's only about physical security and then there's a DCA which flashbots helped um they authored it and uh it tells about how to use measurements of the host itself how to bound it to to get the like a tested host not only a tested CVM but also a tested host um it and it also uh can be used to to do the physical security stuff and of course there is this platform ownership endorsement it comes directly from Intel but it only attests it's like the same things that has PP ID but it's

unchanged. It's called PID. Um very original Intel but uh yeah it's basically some some static info that is unchanged uh in host and if you're a hardware provider you can just use this to um to have this uh inventory and then expose it as an API for people to check if if you have this machine or not. Okay, this solves per part of problems uh but it doesn't solve for example vendor reliance and for this there are two solutions. One of which is puffs uh it's physical unclonable function and what it means is you can have theoretically a function that just inspects all like electric signal that goes through all your silicone and in

the end it will return the same value. It's like a hash function for hardware and we saw it in some specialized hardware but it's not there yet for general purpose CPUs. uh it's just not there yet. And there's also trustless TE's. So we do hope and we know some projects that sometimes in the future they will be an open-sourced version like open source hardware which you can just print print go to FAP and ask them to to get your hardware that will be open source completely and this will eliminate a reliance on Intel and other uh vendors. Okay. The next one is TDX module. um TDX attestation uh doesn't provide you the notion that it is up to date and it's the latest version of this

TDX module. It just provides you with some SVN. SVN stands for security version numbers. So it's like hey I'm running inside on this firmware with those five versions. Here are my my versions and it also gives you FMSPC and it's like a family identifier. So you can go to the Intel website uh and see if this family and this version numbers are okay. Um so those are status that you can get from Intel. It's like up to date needs mitigation out of date. Uh but it's not they they don't give you the uh the software that verifies it for you and that that goes uh there for you.

So we open sourced our TCB info monitor. It's a simple program that uh does exactly what it says. So you can put your family number and you will see how like if those measurements if sorry if those SVNs are secure or not up to date or not. Um also we did for the similar project uh for for the different project we generate a quote inside TDX and then we put it on chain. So this code is a little bit different. It goes onchain and checks um checks the same thing but for onchain attestations. uh you can use your model of charts to edit it as you as you like. Yeah, debug mode as I mentioned TTDX at stationation contains this bite that okay uh this this machine

this CVM is running a debug mode but you as a verifier is not like it's not enforced that you don't trust it. So you have to check it yourself. So yeah please check those those things like SVN uh and debug mode yourself. There are a lot of free letter words here in uh TE world and uh yeah I think those are doing this to us um on purpose. Next one is different like host and VMM are untrusted by design uh for all TE's and that's that's the that's the idea. So you you're protecting from from host but host can observe some things like caches, page faults and timings and use it to do some attacks on your T and for on your workload. Uh I will mention

specifically those couple of those. Uh the most nastiest one is TDX-ray. They combined a lot of you can find it online the research paper. They can combine a lot of non and unknown vulnerabilities to um so they were running a workload inside TDX and this workload was inference like LLM inference and they were able to restore the original uh prompt that user supplied. It relies on some static information that stays inside um TE, but it's still an ST attack. Um there's also T time. It's a little bit uh simpler. It is just um by comparing the timings to page access. Uh it it was able to steal some private RSA

keys. Um and then of course there are step execution class of attacks that is basically like debugging in software. you you can step one instruction one CPU instruction at a time and see what's changed like what this program was trying to do is it trying to access this line of memory or this line of memory and somehow so the important thing is that none none of those supply chain attacks were able to directly read from encrypted memory and I think this is really important that directly they can't read uh they just can use some side channels that's probably why they're going they're called side channels because it's it's not direct input um how to mitigate those. So that for the some of them it's enough just to use constant type functions and constant

type functions are functions in your software that take no matter what the input is it takes similar the same amount of time always to execute them and then oblivious RAM is a class of um mitigations and it's it's difficult to explain because they are all different there there's no specific implementation that is meant that that is understood by oblivious RAM But the idea is that no matter which memory line you access, physical access can be different. So if your program tries to access line ABC, it can be not necessarily line ABC, it can be something else. So it can be duplicated. It can be like load balancerish.

Um yeah, it's it's hard to explain, but anyways, uh you can read on those um somewhere online. They are expensive uh in terms of compute power, but still then you can patch your TDX and update your firmware. I believe those step um step execution attacks they were mostly mitigated by just updating your firmware. And then the last one is isolation. So no code tenency. If if your workload is running single on your machine, there will be lesser chance that it will be exploited by something else that is running on the machine and it's it's unfeasible in cloud providers but if you own the hardware then probably it's much easier. The next one uh of inputed of untrusted input is the network and network by default is

unencrypted. So you have to encrypt it by yourself and the idea is that you should terminate your TLS inside ZTD but it's only half of the solutions. The real solution is a tested TLS and um here is our implementation of ATLS solution. Um so ATLS is um for certificate authority you're relying on certificate authority to actually um say to you yeah this is the genuine the genuine uh server that you're trying to communicate uh because of the PI etc etc but in uh in TDX world you can just include you can generate a key keeper inside the TD inside the CVM and then expose u one public key through

attestation and that that can be enough for you to prove that yeah this is the genuine TD that you're trying to communicate. Um another one and this is this is I guess really important one is that encryption doesn't stop censor censorship. It doesn't protect anything from censorship. So for example if a blog builder decides that it doesn't like even IP addresses it can just filter them out or odd IP addresses. Anyways, it can I also delay the transactions from uh arriving to the builder or just do some strange uh unpleasant censorship and in this article we are um one my colleague from Andromeda he's explaining uh those

things and also tomorrow there will be an anonymous broadcast uh session it is done also by one of my colleagues and uh yeah we were R&D R&Ding some protection from censorship uh just join if If you're interested there are there is not only network in these that is untrusted there's also shared IO and there is there are block devices so you have to encrypt your uh disks of course otherwise hypervisor will be able just to read your information on your disk um entropy provider for your cryptography it's also it can be hardware supplied but it can be generated uh inside your kernel and uh by by some some techniques uh I will not go into details on this but I just want to say about the PCI uh

so for private inference for example it is important that the information frameware from your CPU to the video card is going uh encrypted so there's a protocol called called TDS I believe it's TDX and Intel specific uh but I'm not sure about this and it um it's basically an attested so it allows your CPU to attest your video card and create an encrypted uh attested channel uh inside so that all all hardware data will be encrypted and then T is the last thing here is T's do not guarantee livveness. So your host can just turn off your CVM and that's it. Um, of course it um it it doesn't break confidentiality but availability can be harmed. Right.

Uh speed running through the slides. Next uh the gray zone. Gray zone are some weird um notions that are pro probably not commonly known. probably commonly known I don't know but um so for example imagine the situation that you want to to authenticate your machine your workload to some some KMS that stores secrets right and how do you do it so you you you're like yeah I'm smart I'm running inside so I can I have measurements and measurements can prove that this is genuine machine and genuine workload but it just doesn't work because your an adversary can can spin up the same VM if your VM is open source first on the same machine and it will

have the same um measurements and but but a malicious workload potentially for example if they found a vulnerability and it's the same image and you don't know about this vulnerability they can exploit it to get those secrets and to leak them um we don't have like no I don't think anyone have solution other than uh introduce some secret that you will bring uh but I don't think that identity is possible purely on measurements um right now there is also is um an an opportunity for host to rewind your state. For example, the classic attack is you have 150 rupees and you spend 50 rupees, you're down to 100 rupees and you just restore your VM

from snapshot and boom, you're back to 150 rupees. Uh and the protection from it is uh monotonic counter. So just constantly imple incrementing counter that will be checked as well. The last thing here is um the host can supply ACPI tables. It's basically some driver info. So a host can say to your VM, hey here are some drivers for you to load lower level code and those drivers are uh they can they can contain as uh AML code and AML is just an executable. So this is a legit way from host to get RC inside your confidential VM and this can be scary.

But uh there is difference. So all those MLCPI are attested in RTMR like they are measured and put into RTMR zero. It's it's a part of it's a part of the measurement. So if you're running your own hardware and if you know all your stack you will pro probably safe because you know what you're running. But if you're running on a cloud provider, uh they tend to change uh RTMR0 all the time just because they can update some firmware that you don't care about and measurements. This specifically and RTMR0 can be changed. Um so how to protect from this attack? Um there's a edgeless uh systems patch. Uh it's a patch in the kernel. Here's a link to to

this patch. And what they did is they separated uh here. So they they introduced a kernel guard and it basically it allows access to the shared IO which ML code is okay to access. It doesn't have anything secret hopefully if yeah but if uh your kernel your your workload kernel will try to access something private then it will be denied and it's a small patch uh probably 100 lines but u yeah use it. Yeah I already mentioned that. So the last thing is workload is just normal web two stuff. Uh you can just attest your bad code. Yeah, do all of those. Do collect your sbombs. Uh scan those and use salsa to for provenence. Uh the one

specific thing that I want to highlight here is that over the last months there were enormous amount of CVS in kernel uh which we which is really tired of patching those and uh last week's uh yeah it was last week last week's they they were in they introduced the TCP stack vulnerability which which is which was used to hack a couple of hardware uh cloud providers. So yeah, just patch your kernels uh constantly no matter what because the fix for the last week's one was on like in inside the kernel for a couple of I believe days or weeks before it was public. Um yeah, your deploy chain was also important

obviously. Uh cryptography. Yeah. So yeah, this is this is a talk. Uh so use proof of cloud for physical protection. use uh TDX like check security version numbers uh on your your like even if you connect to someone else's workload on your own and then terminate ATLS inside your TD ACPI kernel patch is important do think about freshness and monotonic counters and then harden bills uh kernels and dependencies and yeah we've gone through all of those and just as the last slide that there is no 100% secure software and TDX model is just a combination of hardware and software and it's it's okay for it to be outdated sometimes. Uh yeah, because I know that people are complaining when a

new T vulnerability is out. People are complaining in Twitter a lot and say that yeah, we're all doomed. But yeah, just a software. Um thank you. You can use uh if you want to say something just X to communicate it's empty account, but yeah, you can just send me a message. Can we please get the questions up on the board? There's uh so far only one question. How do you balance risks between using uh cloud hosted these uh versus self-hosting hardware but having physical security >> uh I think those are different and for

example uh AWS nitra is mentioned here and it solves completely different question and completely different problem. um because if your if your whole stack inside AWS like AWS N is not really T in a in a normal way um because of the network and and etc. So I'd say you can try using both and there are ways to protect both of those. So there I don't think that there is a good solution because managing your hardware is more expensive and you have to do firmware updates by yourself and uh it's just uh yeah but for clouds uh there is a way of using them as well uh and probably using all of those that I mentioned it would be already I think

secure enough for you to trust those