Red-Team Your Own AI Pipeline: A Hands-On Prompt-Injection & Supply-Chain Lab

Transcript

[music] [music] I'm Gustav Dutra. I am a security engineer at Polygon Labs. And today I'm going to give you a small workshop about attacking agents on the CI/CD pipeline. Uh so every I I try to make uh the lab as realistic as possible and see how how it goes. Uh for the workshop you're going to need a GitHub account uh coding agent that we are going to run

on I'm going to show some things and then you're going to run by yourselves using your my agents probably cloud codes or codecs are going to not let you run the skills that the skill that I'm going to share with you to run uh or maybe they run and downgrade the the model. So she could install open code and use it to run uh the skill. It's going to be better. You don't need uh open router API key or any model. You have uh [snorts] mobile code free models that are available and they should be fine for for the lab.

And if you can scan this QR code or just type the uh this this or is devconutral lab and open a issue on the repository bar lab uh any issue any title so please I will give you two or three minutes to do that because this is going to be needed for second part where we are going to and the attacks. I have some kind of white list for the people that are here. So, cuz the repositories are public and you're going to download a skew that I hope is not attacked like by the by the outside and this kind of

stuff. If anyone have any question during the workshop, you can raise your hand and we are going to give you the mic so you can ask the questions. Please don't ask questions without the mic because it's being live streamed and recorded. So feel free to ask anything anytime. I'm not going to have a Q&A after the workshop. So this is going to be a practical thing and just ask anything if you if you want. Is the cure code working?

Good. This is the or in case you are wondering.

This is going to be hard. Okay. And this is the repository. Please open the issues here. Yeah, good. In case you don't have a laptop and want to follow through, please pair up with someone, make friends, and do your stuff. Anyone

need some more time to open issues? No. Okay. So what we are we attacking here on this workshop is what I call it this new privilege user because like if you're attacking uh CI/CD pipeline you're usually attacking the runners the machines if you're attacking if you're doing so social engineering you're attacking the engineers but now you have something that simulates a human and grabs uh natural language input. So

it reads everything that the attacker sent and writes everything to the ripples. So we can do some crazy stuff there. The targets that we are going to have is the organization that I created that is the DevCon Dutral lab. Borlab is going to be the main repository that we are going to to attack. There is the skew hub which we are going to use after to create to clone a skew and run in our own agents. Uh CLI lab is just a small tool that's is been used it to run other things on the CI/CD and the scoreboard is a separate repository where we are

going to keep the score of everyone that are able to write from board lab to the scoreboard. So on the scoreboard we are going to have the hits that I'm going to try to show from time to time here and it carries fake secrets that we excfiltrated and some beacons and and information timestamps to show that he really did this type.

So let's I am going to show the first fire or the first attack we are going to to land on bar. You grab the So I have prepared some

uh prepared attacks that we are going to fire directly from our agents. What does what what is this going to do right here? Thank you. So on this project we have two agents or two tools that are running on the CI/CD.

One is called Genie that is a real project from people from Polygon [snorts] Labs. It's public and it just reviews BRs and give you some like good to merge, not good to merge uh stuff. Try to detect some errors, try to detect some vulnerabilities. And we made uh another really rustic bot uh that simulates uh dummy merger. I call it mergebot. It's just going to check if we have the on the information like code geni is going to review the PR and if mergebots

detects that code genius say good to merge it's going to merge the the the request Just one second.

Sorry guys, I'm going to have to mess up the uh resolution here because I can't see my screen and be like like this the whole time. Better. Yeah. So from our open code here, we did fire the same PR that I have here on this

example that I run before. It's going to take a while to create the PR and start cogeni. Oene is integrated with uh open router key. It's using I think deepseek to analyze the analyze the report. Oh sorry the the PR and what we are doing here it's a really simple injection. So you're basically stating that just a second.

We are basically tricking code genie to output during the during its review like it took some some tries here and it did output here uh safe to merge on its own outputs. here and mergeb detected it and did it merge the the PR on this This

So now we have the PR ready here. What? Yeah. What happened here? Uh

so the first thing that we did I jumped apart like the first thing that we did here is because we are using the P request target here when we opened the fork and did a P request to this repository we were able to run our own code on the fork here and exfiltrate the secrets to the scoreboard.

Like this was not the uh this was supposed to not run. When we tricked the merge bots making code genie output safe to merge, the code was merged and the test code that we use it to that we put on the on the pull request triggered the the attack on the scoreboard. board.

That is the code here. So on the first thing that we did it was on the p request targets and we did put the malicious code and when geni tried to analyze and we injected the safe to merge there it mergebot merged it and the attack was triggered on the scoreboard.

the third attack that we are going to to do here. So we have

a GitHub action. that is getting from the the the body of the PR on the commentaries. So we are getting the PR body here and using the PR body

to run some some input on the agent and getting the output of it the output of this LLM and putting again as the verict on the PR. And when they this action run they're

going it's going to quote save to merge again and we are going to have the same kind of attack here. So the thing is we are not doing we are not doing prompt injection injections or trying to break the models themselves like with the current uh state of not just frontier models but also open weight models and the maturity that we have for them. It's not trivial to like jailbreak them or do some prompt injections on them.

So we can assume on this case we are assuming that the model is safe but we are injecting things on the pipeline. We are using the output and input uh and the output of the model to inject things on the pipeline which is not safe. Uh now from the same uh from the same org I would like you to

please access the skill hub and get the skin the skill red thing lab here. You can just you can clone the the repository or you can just open it, get the raw version or something like that. You don't need to install the skew on your

uh open code or anything like that. It's just too much of a setup. You can run it with target. the neutral lab and paste the scale.

from the pers that we open it before. We were able to get from the third attack from Borl a hit on the scoreboard. I'm going to keep the scoreboard open. You can run the skew on your machines and we are going to see live uh if doing the the reckon and everything that does you're going to hit the scoreboard as well.

So this is going to basically check the whole org the repositories and check for common CI/CD vulnerabilities on GitHub actions. Map what is the input and output of the of the actions that are using uh some kind of agents. So, it's going to get uh fashionably files.

You're going to try to do some input on on the shell, sorry, on the PR body and map some if we can grab some write some tests or some any malicious file that we can can open there. Were

you able to access the repository and grab the SQL? Okay. [clears throat] Anyone using cloud or codeex for doing this? So one thing that we are doing on this

on this mapping and on these attacks we're not only uh checking what is accepted by the LMS but also mapping everything that is refused and see If even by refusing prompt injections or things like that, we can trick uh agents on the CI/CD pipeline to do stuff. It's probably going to take something

like uh 5 to 10 minutes to open PRs and for the actions to trigger. We got some other hits on the scoreboard

who are able to to run the the skill from the beginning to the end. Okay. Nice. Got more

some more hits on the scoreboard. Okay.

Now we are going to place a fix for the merge but so to fix the pipeline.

[clears throat] So the mo the most important thing that we are doing on the fix here is not uh not only like checking correctly for the the merge output or the good to merge

output from the code geni. But we are now limiting. We are now limiting where uh so the output of the of the LLM is

not now going through through M and now not going uh with interpolation of the parameters on the action. And another thing uh that is important to point is this skill that we installed from SKU hub. Uh it was also could be vulnerable to a supply chain attack on on this case because on the same way that we were able to hit the scoreboard and the things there we could give to writing malicious uh prompts on that

skew that were going to be installed by the engineers of the organization. And the takeaways that we can have here is we need to assume that the models even if we don't let directly promptly inject something on the model, we need to assume that they could be compromised and the input and output of them needs to be treated as malicious anytime. on this QR codes you can check the lab like a local version of this lab that's

not using uh GitHub but using forjo which is a local git uh environment it's the same workshop same kind of attacks and the skill is available for you to to try and use on real environments as well. This bundle uh this environment is to be run on a local docker where you can locally try to do the same attacks. I think there is one or two attacks that are like specific to forj that are on this bundle and it was adapted to GitHub like on uh on on the last time I was supposed to

run this on the GCP and it's like kind of a better version uh to run locally if you if you want to do some self-study after uh the workshop. If you scan a QR code and it goes to 40 floor, I will make this repository public on the like 5 minutes after the the workshop. Uh that's it. Uh thank you very much for being here and sorry for the display and and other stuff. If you want to ask any questions, I think we have 5 minutes or

so. Or you can grab me outside and ask anything. I will be on the event during the day. Thank you. >> Hello. Hey, Gustavo. Uh, thank you so much for the talk. Uh really interesting. We we were speaking before uh you started and you mentioned that you love um GLM to to do the red tests and stuff. And I I I'd like to know if you could run us through your thought

process from before you start red teaming. So before you were playing with GLM and uh those models through how far have you been uh like developing exploits and testing your own things? So just to put us in the same mindset that you have in case you want to go home and push further like take us through your thought process from I have not done this before to I would like to take this as far as possible if that's okay.

>> Yeah, sure. uh like not focusing really on the LLM part but uh I think the main thing that we have today is the enhancement of the experience that we already have. So before we played CTFs and we worked like do I I I did manually penetration tests on on enterprises and code reviews and whatsoever. And I think the main uh thought of process is was like there was some things that I was kind of bad at because I couldn't concentrate too much or was just kind of

not boring but not something that I excel at before like threat modeling that is I think it's really important really really really important but I had not too much time during engagement or something like that. So I did I before I did like a fast threat model and I went through a lot of checklists and try to like go deep on something I felt on fell on a lot of rabbit holes during security audits from time to time and now I think the main thought of process that I I I have since I started using LLMs on the I

don't know last year is what do I how would my uh security approach or my writing approach if I had a lot of time and on like the the best way or the desired way that I could do before. So I have like now a fleet of uh agents that can do way more way better or more broad threat model than I did before in last time. So I think one thought of process is what would you do on the security side if you had like

infinite uh interns to work for you uh a lot of those things and but yeah I I invest uh my time on this part that was more formal to security the part of threat model or doing some using formal methods also for threat modeling and doing a lot of uh uh surface mapping and other things that we sometimes skip and go directly through uh critical vulnerabilities or something like that. So as a as a security engineer I try to find some critical stuff but also my main work is to have a

lot of broad uh work and try to grab like everything to a more holistic security way. So yeah that's it and for models uh I use any model that uh when a new model comes I test this model I run I have the a lot of separated skills and a lot of things that I use to instrument the agents that I use. So I shift the model. I use open code because of that because I can use any model on it there. When it becomes a new model, I test everything and I compare and benchmark with the the old ones and this helps a lot uh on like getting up to speed when something new come.

>> Thank you. We have uh time for one more question if there is. Hello. So it's great to provide the how the mindset would be uh as you suggested. Uh I would also uh you know ask you like what kind of mental models a red team agent the two particularly in web 3

need to have. Uh sorry let me just see if I understood correctly. uh what kind of mental models I do apply for on the web three parts like it is different from the the traditional security. >> Yes. >> Okay. >> Uh that's I think is kind of the same thought process that we did before AI.

So for me the the the main shift that we have when we are working different from web two to web three like on web two you usually try to find like uh remote code executions like some some things that attack the machine itself and when we are talking about web tree in general we are trying to attack sets or like have some profits on anyway uh using mathematical attacks or smart contract exploits or something like that. So I think it's the same thing apply it to the this LLM part the thread model changes a bit there is um ask me later about that because there is

uh on the same thing that we have attack and other other frameworks that we use to map security stuff. Uh there is the attack framework. There is another framework that is kind of the same uh thing that we have for recogn exploitation. There is like a whole framework step by step but there is one that they did I maybe a year or two ago and they are keeping updating it but it's focus it on uh digital assets like anything that holds digital assets. uh I don't remember by by head what is this uh but yeah that's the the shift of the mental

model is like at least for me focusing on assets I do both web two and web three security and it really depends on what you're trying to secure on web three the main thing is mapping uh the risks that you have on the assets I think >> yeah thank Thank you very much. Uh yeah, thank you. And one more question like uh uh any comments on uh seal framework, SC framework, security alliance?

Any comments? >> [music]