Ryan Kidd: Scaling AI Safety and Security | Intelligence at the Frontier 2026

Transcript

[music] >> Thank you so much, Trisha. Okay, so I'm going to talk about MATS, but I'm also going to talk about a broader problem, which is to say the talent bottleneck in AI safety and security. So, this is me. I did a PhD in physics. You can see me with some of my MATS team members. I did it back in Australia at UQ about 5 years ago, and since then I've been co-leading MATS, came over here for the pilot program, got involved, started running it from cohort two. Um I'm also co-founder and board member at LISA, the London Initiative for Safe AI, which is the largest AI safety hub in Europe. It's great, check it out. And I've given a bit away about 400k via Manifold, which is a wonderful re-granting platform for crowd-sourcing funding for cool, awesome projects.

Check that out, too. Okay, so what is the problem we're trying to solve? One, I think AGI, artificial general intelligence that can do most, you know, human tasks is pretty near. I think that it's dual-use fundamentally, and I think that making this safe in the broader context of the complex systems it interacts in is really hard. Okay, so when do we expect AGI? Metaculus says September 2032. That's pretty close, and notably the distribution is peaked, so the modal year is actually 2028. Right? And I think the sooner we get AGI, the more risky it is.

Two, AGI is dual-use. There's a lot of good things we can expect should we achieve this technology, such as better health and lifespan, democracy, perhaps promotion, and better information access for everyone, wealth and leisure if there's no more work. Also, probably much more powerful, scary weapons, worse addiction, more surveillance and enforcement in totalitarian dictatorships being empowered, and of course, permanent job loss and disempowerment. So, there's a lot to, you know, there's a lot to go off there. Also, I think it's really hard to make these systems safe and ultimately prosperous because the world is really complex, and we don't know the impact of these technologies on the world. We can

try and predict it, but that's hard. And then we deploy it, we try and do all these safety mitigation work in advance, we're always going to fall short. And so, it's also hard to tell in such a complex system as the world until it's too late what harm has been done. So, this needs a lot of work. Okay, so what is bottlenecking this whole thing of AI safety and security? Four things, very basically, talent, funding, ideas, and the will to deploy them. I'm going to skip the last one cuz that's not my forte, but I think the first three boil down to talent. Why? Because I think we have a funding overhang, and an ideas overhang in some ways. When I talk to the frontier AI safety people, they all say we have an ideas overhang. And even if we don't, we need the talent to come up with the

ideas. So, here's my proof of a funding overhang. There is so much money. The main non-profit donors like Covic and Giving Survival and Flourishing Fund, ACX Futures, they're pouring money. They're doubling or tripling their annual grant budgets this year. Wow, that's a lot of money. We also have all these for-profit VCs, Lionheart, Juniper, Safe AI Fund, Moonshot, 50 Years, etc. Look at all these great incubators as well. And then the Gates Foundation is entering the mix, funding Law Zero, the OpenAI Foundation, and when Anthropic IPOs, tens to hundreds of billions in philanthropic dollars are going to be there.

There's a lot of money. Also, the people have raised a bunch, too. Look at Goodfire, first AI safety unicorn. And of course, the market for for-profit AI insurance technologies is doubling annually, according to this report from Juniper Ventures. Also, the companies are growing two to three X per year. Frontier AI companies are scaling super fast, especially XAI. 6X per year is crazy. Their safety teams need to grow, too. If you look at the applicants to my program, MATS, both for fellows and mentors, these are growing about two X per year. So, the demand is clearly there from people who want to work in this space.

However, the actual deployed talent in frontier AI safety and security is only growing at 25% per year. This is really slow. This is not good. If we look at comparable science and engineering projects, I claim to aligning AI systems like the Manhattan Project, the Apollo Program, LIGO, Human Genome, they had a lot of people working on them. My own credence is something like 10 to 10,000 technical FTEs based on these, which will take us about 10 years at the current growth rate to reach. We might not have that long before AGI.

Another reference class, let's look at cybersecurity, rather regulatory industries. Something like 15% of SWEs are cybersecurity SWEs, or at least by by number, which would imply 10 to 20k technical FTEs at the frontier AI labs, more if there's more frontier AI labs that keep scaling. And then it'll take us somewhere like 10 years for that. And if we want to have the whole world more secure for the AI off-on sellers and so on, that's going to take us 25 years. So, here's where we are.

0.1% of people working on AI doing frontier AI safety and security. 0.1%. If we get to 1%, I would feel so much better about the world. If we get to 10%, that would then look like cybersecurity. That's a 10 to 100X scale-up. Now, notably, some people are doing AI safety and security work not captured by that 0.1%, but even that's probably only 10 times that, and it's it's not as high-priority work. That's a huge growth. How can we get there in in 3 to 6 years if we 2X each year? That's the goal. So, my mission is to solve the talent bottleneck in AI safety and security by giving people the skills, proof, and connections to start teams, to join existing teams, and to

increase the carrying capacity of the field. Okay, so what's bottlenecking AI safety talent? So, skills. People need mentorship from key mentors. They need like a really great peer feedback network, and they need high-quality reps, that many shots to build that research taste. It's hard. Two, they need proof. They need a portfolio. They need references, and they need credentials that they can prove cuz it's extremely competitive hiring market. And three, they need connections to collaborators, employers, and funders.

And MATS aims to provide all of these. What I think in the coming age of AI, these three skills will be most in demand. Research taste, this ineffable like what is a good hypothesis? When do I stop pursuing this line of inquiry? Stuff that language models can't currently do. Leveraging AIs, using them to do your homework, and three, coordinating with humans cuz we're always going to be trusting these human connections more. What about AI labor? Well, it can do some of the work, but not all. I think it will remain jagged in key ways for a while, and humans will be complementary, which means Jevons paradox applies. The more AI labor, the more humans you need to make use of that labor.

Two, the more the human oversight waterline is raised, the better chance we will have of inspecting AI output. Three, alignment MDPs, AI that does your homework for alignment, just might not work. They might be too dangerous and scary. So, then you need humans. So, if you don't build the field, we're screwed. And lastly, research taste might be particularly important for weird fields like alignment, whereas AI capabilities might just be like running a bunch of benchmarks, number go up. So, MATS runs uh two, soon to be three times a year. We're basically running these research bootcamps for 100 plus fellows in several locations in Berkeley and London, soon probably in DC. We have offices in these places. Uh we have

seminars, workshops, and awesome community, come visit us, uh and tons of research support staff. Here's our portfolio for the summer program that's going to be happening soon. We have mentors from all of these institutions, and we pair up our fellows with them, and you can see we have a whole spread of different types of research. We believe in a broad portfolio. So, so far impact over the last 4 years we've existed, 530 alumni, 80% of whom are are working in AI safety roles, uh 10% of whom founded organizations. Not all of them succeeded, but that's still cool. And over 180 publications uh on archive and other sources.

So, some examples, uh two two leads at Anthropic on the alignment science team of teams, that's Sam and Sarah. Uh several people at UK AC leading teams, Joseph and Alan. Um people leading teams at Apollo of AI Futures Project Redwood, Thomas Kwa, who set up the the meter timelines benchmark. Um yeah. Some cool people. You saw some today. Uh founders, tons of founders. So, Apollo Research was founded out of MATS, uh Timaeus, uh SL5 Task Force, Workshop Labs, Watertight AI, Theorem, tons of things.

Oop. I think I turned it off. Uh yeah, tons of cool research, like emergent misalignment if you've seen that. Um sparse autoencoders, activation steering, situational awareness evals, uh Soluble Magikarp. Lots of cool things came out of MATS. And of course, people seem to think we do a good job, 9.4 out of 10 from fellows, and 8.4 out of 10 from mentors. So, as a last kind of uh thing I want to say, what do I think needs founding? Well, tons of stuff, okay? So, we need auditors and underwriters to help inspect these technologies. These people can't all be based inside the frontier AI companies because you need to have some out external partners who can, you know, show that they're not necessarily compromised, right? And so, it's great

to have these these organizations that are inspecting and providing insurance mechanisms and audits. Um we need security and non-proliferation orgs. In some sense, there's some types of AI systems that we should definitely make open source. There's other kinds that like, well, you wouldn't give your kid a nuke, you know? Uh so, we need ways to like ensure that people's uh data is kept private in some cases. Defensive technology as well to make the world secure. So, making uh making it harder for AI systems to do attack vectors through bio-weapons or cyber-weapons or so on. Just making the world more safe and secure in general. Uh we're in information warfare. We need deep research startups in the non-profit and for-profit space. And we need more field building and offices like this place where research can flourish.

And just I'll give a little shout-out to This is a small fraction of the AI safety talent ecosystem, but uh there are some amazing organizations out there of which MATS is just one. And yeah, I just want to thank all of them for their work. Do I have time for questions? Can I get my mic on? Hello, hello. Ryan, that was really cool. I think um you know, so we were just talking backstage. And as the co-founder of Advanced AI Society running the association industry association for proof of control where our builders, our members literally offer the security measures that

enterprises need. Um so, I'm curious when you're talking to companies, to enterprises, do you see that there's an awareness that they have to hire these you know, these researchers that are coming out? Are you talking to CISOs and hearing from them? Or are they still kind of in a learning mode and they still don't totally understand like what they need who they need to hire yet? I actually don't talk to many CISOs or many people in the industry. The main people I talk to are like at the frontier AI companies and at like uh these kind of VCs and incubators that are trying to spin up more companies. Um I I have I have though talked to some people like at IBM and Cisco and stuff and they seem like aware. I think it's like there's a there's a most people in the in the uh

I'll say this like I think everybody should be trying to prepare for AGI. Doesn't matter what you work in. And I think in general people are not thinking about this. Um I'm not the expert on this. Uh I I do I think my partner is over here might have more to say about that. You can mob her. Um she works in in AI preparedness. Uh I I I would just say that like definitely at the the bigger, more um advanced frontier AI companies, they think about security and safety constantly. Yeah, we know that. That's for sure. But as AI becomes um as AI systems grow and infrastructures grow, uh we need to have more people, you know, the non-frontier labs to actually be concerned about this cuz you see, you know, regular enterprises like I don't know, like Clorox or you know, Walmart

deploying this stuff. And that's where it stuff is going to hit the around you. Or traditional, you know, finance like TradFi deploying this. Um like credit card companies, right? Letting agents loose. And so, we're going to have to We need these AI researchers and these security experts everywhere. Um and so, I have a I have another question then. I Do you remember the moment when Harvard um published this article? It was like so long ago when they were like data scientists had the sexiest job. Like and everybody wanted to become a data scientist. Um I feel like after see Wait, you know, you made such a convincing argument that we have a talent um constraint issue. That that's the bottleneck, right? And I have a prediction to make, which is that, you know, it's going to be AI researchers that are going to have the sexiest jobs.

And they are continually going to be needed. That it is actually an AGI-proof job. Would you agree with this, you know? So, I'd say like already like in terms of salaries, AI researchers have the sexiest or the most highly paid jobs, right? The people who can leverage AI better. Like you can't work at these AI companies unless you're leveraging AI. People don't write code there anymore. They just review code. Um in terms of like the sexiest job, I think safety and security is sexier because there's fewer people who know how to do it. You're working on something to benefit the world, not just your own pocket. Um and I yeah, I don't know. That seems pretty cool, man. Uh which is why we're here at California in the Commons, right? Um well, so what do you think is the next step in terms of if

you could make a call out to the audience right here and all those who are listening on the live stream. Um how do you want people to reach out to you? I know you gave a few options, but just to remind people what's How do you Who do you want to have talk to you right now? You know? Yeah, so I mean like I run MATS. Apply to MATS. It's great. We we do these we fellowships two to three times per year. We're hiring. >> You have to have a doctorate? Like what's the background? >> No, I mean 15% have PhDs, like 20% are undergrads. Like we'll take anyone who is great. You know, we just like we I think we have great uh assessment processes. And we we're super broad church. We have all sorts of kinds of research. Other than that, if you want

to found a cool startup, you can reach out to me. Uh ryankid.ai. That's kid of two D's. Um got some contact details there. I'd love to hear from people who want to found cool things. I love what you're doing in particular because it's not you're not gating it to where before to become a researcher, you have you know, I mean we're still in many these spaces in the academic, you have to have be attached to an academic research institution, you have to have a PhD, you have to have an advisor, and you have to create a whole committee. But in this is you're literally like if you have a great idea, we have a way to evaluate it and you just have to share that. Yeah, I like to think of MATS as operating on three pillars. That's uh uh non-profit impact, of course, academic

uh rigor in terms of the research results, but startup speed crucially. And so, we don't want to be like too compromised by institutional uh credentials or something. I still think there's a place for PhDs. There's a place for all these things, but you know, we we aim to be like impact first, which means not anchoring too much on proxies for impact that are bad. All right, cool. Well, we have a lot to talk about afterwards. So, thank you so much. I think this is really important this um what MATS is doing that you do not have to have a PhD. If you have a good idea for how to make AI safer, you've heard from three pri you know, great researchers from Evan, um from Paul Reiker, and from Quinson. Some of them had PhDs, some of them don't, some of them have startups, some of them are

currently researching. So, there's a long career ahead of this. And you thank you for creating a really great ecosystem and starting at the bottleneck. All right, thank you, [music] Ryan. >> [music]