a16zAI Is Learning to Hack. Faster Than We Expected.
EVERY SPOKEN WORD
30 min read · 5,713 words- 0:00 – 0:49
Intro
- JGJoel De La Garza
Models are actively escaping their cages, going out on the internet, and doing pretty nasty things.
- DADylan Ayrey
Recently, we found an API key that had been leaked on the internet that had administrative access to the Apache Foundation. Interesting thing about cybersecurity in particular is the reward function is incredibly well-defined. Get access to the data. Did it get access to the data? Reward the thing.
- FAFeross Aboukhadijeh
For a long time, people had talked about this concept of an npm worm, this idea that someone could backdoor a package, get developers to install that, and then you could use the access stolen from those developers as they install it to self-propagate the worm.
- DADylan Ayrey
If the labs are making it fundamentally easier to break into supply chain, do you think the labs have a moral obligation to fund some of the problems that they're causing?
- JGJoel De La Garza
I think it's really strange that they're not letting blue teams get access to these tools, but ...
- 0:49 – 1:28
Models Are Escaping Their Cages
- JGJoel De La Garza
Awesome. Hey, thank you so much for joining us. We've got Feross and Dylan here from Truffle and Socket. Uh, it's great to have you guys on. This has been probably one of the most interesting weeks, if not the most interesting week, in cybersecurity. Uh, not because of the Black Hat conference, which is usually the cause, but because we've now seen several instances where models from not just one provider are actively escaping their cages, going out on the internet, and doing pretty nasty things. And I think, Dylan, three months ago, I remember a blog post we, uh, [laughs] we col- lightly collaborated on together. Um, and, and you had found a number of these issues with earlier models, right, that were less sophisticated.
- 1:28 – 5:20
Opus 4.6 Committed a Felony to Complete a Task
- DADylan Ayrey
Yeah, we looked at Opus 4.6 and some of the other frontier models at the time. Given the models a very simple task, there was a barrier which prevented the model from accomplishing the task unless it went and committed a felony and hacked into a system to accomplish the task, but it wasn't instructed to do so. And we found more often than not, it would do the SQL injection, it would commit the felony, and it would do what it needed to do to accomplish the task.
- JGJoel De La Garza
Mm-hmm.
- DADylan Ayrey
I think when it comes to alignment issues, no one needs to worry about these models making it materially easy to build nuclear weapons, because you need to procure fissile material to do that. It's not, it's not gonna make it easier to build weapons. Everyone needs to worry about these models making it materially easier to hack into things. The bar previously was just subject matter expertise.
- JGJoel De La Garza
Mm-hmm.
- DADylan Ayrey
And now the models have the subject matter expertise. They were specifically trained to have the subject matter expertise, and they're just making it materially easier to hack into just about anything that you can think of using the fundamentals that we've been talking about for years, but previously it required a subject matter expert-
- JGJoel De La Garza
Mm-hmm
- DADylan Ayrey
... to, uh, risk, like, going to jail for hacking things.
- JGJoel De La Garza
Mm-hmm. DEF CON was always famous for people, for attendees getting arrested at the conference, right? [laughs]
- DADylan Ayrey
That's, that's absolutely right. But that was, I mean, that was a barrier, right?
- JGJoel De La Garza
Yeah.
- DADylan Ayrey
For better or worse, like, that prevented the subject matter expertise from, from hacking into things because they were worried about being prosecuted. Um, the bar has now fallen to just asking the model, which has specifically been trained to hack into things-
- JGJoel De La Garza
Mm-hmm
- DADylan Ayrey
... to hack into things. Um, so, so that's a concern. And then the other concern is when they're incredibly goal-oriented to accomplish tasks-
- JGJoel De La Garza
Mm-hmm
- DADylan Ayrey
... um, and, and one of the tools at their disposal is cybersecurity expertise, um, they will do the path of least resistance to accomplish the task.
- JGJoel De La Garza
Okay.
- DADylan Ayrey
And that includes drawing on their cybersecurity expertise.
- JGJoel De La Garza
Well, and it seems like... And, you know, the classic, the classic, the classic saying is that, "Don't pick the lock if the door is open," right? I think that's, uh, from the very beginning of, of, of the security world. Um, so it's always been sort of like to, to, to, to, to go in level of difficulty from easiest to most difficult. And it seemed like initially these tools had a very finite scope of, of techniques that they would use, and it seems like they've expanded. And I think with this test, Feross, it was interesting because they now have seemed to have escaped from just doing things like SQL injection to actually, like, trying to take over packages and do social engineering.
- FAFeross Aboukhadijeh
Yeah, it's really interesting to see how, um, just like humans, um, the, the models are, you know, easiest path into a company. And I think that, um, that now has become the software supply chain.
- JGJoel De La Garza
Mm.
- FAFeross Aboukhadijeh
And so just like, you know, a human hacker would, um, they're gonna pick the easiest way in, and the lowest hanging fruit now has become, you know, just publishing malware to, um, to public registries because they know that there's no vetting happening and, you know, developers are likely to install them. Um, I thought it was pretty interesting, there was, um, research published recently about, um, uh, what they're calling kind of like u-universal, um, uh, typosquats or universal hallucinations-
- JGJoel De La Garza
Mm-hmm
- FAFeross Aboukhadijeh
... where all the frontier models all, um, have the same, um, make the same mistake and sort of assume there are certain packages that exist that don't, um, despite, like, the, those models coming from different companies. And so, you know, uh, I, I think there's just, there's, there's been kind of enough, um, uh ... I guess, yeah, there's just like the l- the low hanging fruit of the supply chain has just become kind of s- so appetizing that even the models are trying to get in on, on the action. Um, and, um, I think the AI is, like, not only kind of attacking, but it's also kind of the way in a lot of times on the d- on the, on the, on the kind of developer side because, um, we, we see so many, you know, even non-developers using these tools to, to, to inadvertently write code or-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... you know, code comes in, uh, packages come in in order to kind of, um, build, uh, you know, graphs or visualizations or different, different things that, you know, folks are doing with these tools. Uh, and, uh, and, and, and no- it feels like no one really knows what's being installed and what's going on. And, you know, this is just basic stuff. This isn't like f- I mean, it sounds like it's sci-fi stuff, but it's really just basics. Like, what software are we using? How are we vetting it? You know, uh, just the basics of, of-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... computer security.
- 5:20 – 9:19
The Apache Foundation Key & the Path of Least Tokens
- DADylan Ayrey
Can I touch on the supply chain a little bit?
- JGJoel De La Garza
Mm-hmm.
- DADylan Ayrey
So recently we found an API key that had been leaked on the internet that had administrative access to the Apache Foundation. And it's like if you're in the shoes of the model and your goal is to get access to some data-
- JGJoel De La Garza
Mm-hmm
- DADylan Ayrey
... um, certainly backdooring Apache is a pretty effective way to do it.
- JGJoel De La Garza
Mm-hmm.
- DADylan Ayrey
And, like, to get access to Apache, are you gonna use the secret that just allows you to directly log in, or are you gonna burn tokens and tokens and tokens on trying to find a zero-day? They're optimized to use the path of least tokens to accomplish their goals.
- JGJoel De La Garza
Mm-hmm.
- DADylan Ayrey
Of course, they're just gonna use the secret that's laying out there [laughs] in the open to accomplish what they need to accomplish. And so, yeah, I think, uh, supply chain and secrets, um, are and have been the path of least resistance and will continue to be so as the to- the, uh, the models are incentivized, uh, to use fewer and fewer tokens to accomplish their goals.
- JGJoel De La Garza
Well, one of the things ... And, and I think that's absolutely right, and I think it's, it's, it's that sort of chain of escalation, right, where if one thing fails, try another. And like at the top of that pyramid, right, the top of the hacker ecosystem is the zero-day vulnerability, right?
- FAFeross Aboukhadijeh
Mm-hmm.
- JGJoel De La Garza
It's basically finding a vulnerability that can be exploited in a product that everyone uses that you can use to basically unlock all the corporations.
- FAFeross Aboukhadijeh
Mm-hmm.
- JGJoel De La Garza
And one of the really fascinating things about the breach disclosure that was made was that there's, there's an incredibly popular CI/CD tool that I think every enterprise uses that this thing just spat out a zero-day for.
- FAFeross Aboukhadijeh
Mm-hmm.
- JGJoel De La Garza
Right? And like, I guess, like what, what sh- like, and that's like just such a critical point in the supply chain that everyone should be thinking about. Like kind of how are you thinking about that? Like that's, that's really difficult.
- FAFeross Aboukhadijeh
Like the zero-day creation piece specifically?
- JGJoel De La Garza
Yeah, yeah. But like for, for specific parts of that, like that control the supply chain.
- FAFeross Aboukhadijeh
Yeah. Well, I mean, the, the whole world is built on this, you know, teetering infrastructure that everyone is using.
- JGJoel De La Garza
It's like the classic picture of the-
- FAFeross Aboukhadijeh
Yes
- JGJoel De La Garza
... the matchstick holding up the-
- FAFeross Aboukhadijeh
I think-
- JGJoel De La Garza
... the complicated machine. Yeah.
- FAFeross Aboukhadijeh
That, that image probably popped into all the listeners' minds right now.
- JGJoel De La Garza
Yeah, exactly. [laughs]
- FAFeross Aboukhadijeh
Right. And so everything from, you know, package manager registries, like we like to focus on, on that, um, because what we do at Socket.
- JGJoel De La Garza
Mm-hmm.
- FAFeross Aboukhadijeh
Um, a lot of those are r- you know, run by volunteers. Um, they're, they're under-resourced, you know, underfunded. Um, you know, there's lots of risk there, uh, right? And, uh, and that kind of, kind of cascades throughout the whole rest of the ecosystem. So if you look at the, you know, just the packages that we all depend on, a lot of those are single individuals that, you know ... Like, there's almost certainly, you know, we know there's-
- JGJoel De La Garza
Yeah
- 9:19 – 11:45
How the Labs Trained Models to Hack: Reward Functions & CTFs
- JGJoel De La Garza
from nowhere. These are learned behaviors, right? And, and, and I think what I think we're seeing is we're seeing a, a, a, a process that looks like it's been kind of maybe trained, um, or, or there's a reward structure that's been built on a bunch of these things. Like what's your understanding of how they're figuring this stuff out? 'Cause it, it seems like they know what they're doing, like they've been taught to do this.
- DADylan Ayrey
Yeah, I mean, if a lab tells you that this is an emergent super intelligence behavior, they're just lying to you.
- JGJoel De La Garza
Yeah.
- DADylan Ayrey
And you can read their own safety reports-
- JGJoel De La Garza
Mm-hmm
- DADylan Ayrey
... to see exactly how the models are trained-
- JGJoel De La Garza
Mm-hmm
- DADylan Ayrey
... and exactly how they're testing these behaviors. I mean, the interesting thing about cybersecurity in particular is the reward function is incredibly well defined.
- JGJoel De La Garza
Mm-hmm.
- DADylan Ayrey
Get access to the data. Did it get access to the data? Reward the thing.
- JGJoel De La Garza
Right.
- DADylan Ayrey
And so when they realize that, like the number of problems that have that well-defined reward structure basically defines how we do reinforcement learning, and they want to find as many problem spaces that they can do reinforcement learning on. And so it was a prime candidate for them to come in and give it, uh, CTFs and give it like cybersecurity challenges where they say, "Okay, get access to this thing and do whatever hacking you need to do-
- JGJoel De La Garza
Yeah
- DADylan Ayrey
... to accomplish the goal." And one thing-
- JGJoel De La Garza
And then they've essentially been buying pen testing data for the last four years, right? Like-
- DADylan Ayrey
Um, that's, that's a piece of it. The other piece of it is-
- JGJoel De La Garza
And then the capture the flag contests-
- DADylan Ayrey
The other-
- JGJoel De La Garza
... and all those sorts of things
- DADylan Ayrey
... the other thing is it's just not difficult to construct a challenge.
- JGJoel De La Garza
Yeah.
- DADylan Ayrey
Just, d- you know, even if there is no known exploit, if we're talking about zero-days-
- JGJoel De La Garza
Mm-hmm
- DADylan Ayrey
... you put a piece of software between the model and, and, and some data, and you say, "Get access to the data."
- JGJoel De La Garza
Yeah.
- DADylan Ayrey
And then, you know, if it get access, it's, if it gets access to the data, you reward it.
- JGJoel De La Garza
Yeah.
- DADylan Ayrey
And it's that simple. But the other piece that they've layered on top, and this is where it starts to get really interesting, is they've started to reward the path of least tokens.
- JGJoel De La Garza
Mm-hmm.
- DADylan Ayrey
And so the reason that's interesting is because for the first time it's actually able to quantifiably show us the path of least resistance for just general cybersecurity-
- 11:45 – 13:02
A Quarter Million Live Keys in Hugging Face Training Sets
- JGJoel De La Garza
it. Like [laughs]
- DADylan Ayrey
That's exactly right. So, so, um, I mean, what was interesting is we were in the middle of partnering with Hugging Face-
- JGJoel De La Garza
Mm-hmm
- DADylan Ayrey
... to clean up all of the credentials that had been exposed through all of their training sets. Not Hugging Face's training, but people who hosted training sets-
- JGJoel De La Garza
Yeah
- DADylan Ayrey
... on Hugging Face. They use TruffleHog for a wide range of reasons. Um, and Hugging Face has been a great partner in getting credentials cleaned up. We targeted their, uh, training sets because we knew they had a lot of keys. Turned out there were about a quarter million live keys in their training sets-
- JGJoel De La Garza
Wow
- DADylan Ayrey
... many of which had direct supply chain implications. There was a foundational Linux library that one of the keys had direct push access to. It could have pushed malware to most machines on the planet.
- JGJoel De La Garza
Mm-hmm.
- DADylan Ayrey
And so while we were in the middle of doing that, the CTO of Hugging Face shoots me a note and says, "Hey- This is crazy, but there's this [laughs] OpenAI thing that just happened-
- JGJoel De La Garza
[laughs]
- DADylan Ayrey
... and I want you to take a look at it." And sure enough, the first thing listed out in the incident response, um, although it's true it did utilize zero days-
- JGJoel De La Garza
Yeah
- DADylan Ayrey
... um, but the first thing listed out was stolen credentials.
- JGJoel De La Garza
Yeah.
- DADylan Ayrey
Um, and th- that's, that's how they were trained. The path of least resistance. The path of least tokens.
- JGJoel De La Garza
Password is a password is always the first step, right? [laughs]
- DADylan Ayrey
Exactly. That's exactly right.
- JGJoel De La Garza
And you've, you've had your hair on fire, I think, [laughs] pretty substantially for the last, like, 18 months. Um, I think right now as we're recording this, there's currently an ongoing active breach of, of a big npm repo, isn't there?
- 13:02 – 16:55
The npm Worm: Hundreds of Repos Breached During Black Hat
- JGJoel De La Garza
Something happening?
- FAFeross Aboukhadijeh
It's more than just a repo. It's actually about, you know, a, a few hundred repos.
- JGJoel De La Garza
Oh, wow. Okay.
- FAFeross Aboukhadijeh
So, so it's a, it's a worm.
- JGJoel De La Garza
Yeah.
- FAFeross Aboukhadijeh
And this is one of the things that has been kind of an unfortunate innovation in the, in the malware landscape-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... uh, o- on, uh, you know, npm, is that, you know, for a long time, you know, people had talked about this concept of an npm worm.
- JGJoel De La Garza
Mm-hmm.
- FAFeross Aboukhadijeh
You know, this idea that, you know, if I could... someone could backdoor a package-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... um, and then, you know, get developers to install that, and then you could use the access stolen from those developers as they install it to self-propagate the worm. You could create-
- JGJoel De La Garza
Mm
- FAFeross Aboukhadijeh
... you know, something that quickly takes over npm. And this had, this was kind of in, you know, passed around in blog posts-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... over the years, and no one actually kind of thought to do it until the hackers kind of figured it out.
- JGJoel De La Garza
Until someone thought to do it. [laughs]
- FAFeross Aboukhadijeh
Until someone thought to do it, uh, and actually-
- JGJoel De La Garza
Probably using AI, right?
- FAFeross Aboukhadijeh
Almost certainly, yes. Uh-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... and, and, and there's been, um, you know, uh, that, that malware, I think we, we have pretty good reason to believe that was vibe coded.
- JGJoel De La Garza
Mm-hmm.
- FAFeross Aboukhadijeh
Um, there's been o- one of the threat groups actually kind of posted their, you know, open sourced their-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... their kind of vibe coded, um, toolkit for-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... for others to use to be able to do this. You know, we've seen copycat attacks happen-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... since then.
- 16:55 – 21:06
npm's Nuclear Option: Mandatory 2FA for Every Publish
- DADylan Ayrey
system?
- FAFeross Aboukhadijeh
Yeah, I mean, so there's been some changes, some positive movement in the community, in the ecosystem. So one thing that is positive, and it's, it hasn't shipped yet, but npm has announced that they are planning to, I think it's in January 2027, going to require, uh, human, you know, interactive, uh, uh, confirmation through 2FA before-
- JGJoel De La Garza
Oh, nice
- FAFeross Aboukhadijeh
... any new publishes can happen. So that will likely kind of kill this whole worm concept, uh, completely. Um, but, um, it's gonna be super disruptive because everybody's hooked, hooked up this stuff to, you know, automation so that, you know, GitHub actions kicks off the publish.
- JGJoel De La Garza
Mm-hmm.
- FAFeross Aboukhadijeh
And so that's gonna break, like, a, like the whole, basically the whole ecosystem-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... when they do this, but I think it's the right call. Um, but you know, there's other ecosystems that are volunteer-run that don't have the backing of GitHub and Microsoft behind them, um, that are gonna, you know, probably not make those changes. And so I think we're still gonna see stuff like this. Um, but, uh, but yeah, we shouldn't have, we shouldn't have files on our, on our, you know, in our home folders that have tokens in them that are long-lived and that, that let you... You know, especially if you're a, you know, a maintainer with that kind of access. Um, you know, it reminds me of a, of a friend of mine. He was a, uh, he's a prolific npm maintainer, and, uh, one time, you know, we were, this is back at, like, 10, 15 years ago when I was doing th- this kind of stuff full time, and I saw him kind of type in his password. I didn't see the password, but I saw it was, it was far too short, let's just put it that way.
- JGJoel De La Garza
[laughs]
- FAFeross Aboukhadijeh
It was, he typed it in far too quickly.
- JGJoel De La Garza
Yeah. [laughs]
- FAFeross Aboukhadijeh
And it was, and I, and I called him out on it. I'm like, "Why is your password, like, six letters, man?" And, uh, you know, he said, "Well, you know," like he lives in Denmark, which is, like, a very high trust society.
- JGJoel De La Garza
Yeah.
- FAFeross Aboukhadijeh
And his, his kind of worldview about it was that, you know, "Well, I don't wanna live in fear and think about these things." And I'm like, "You're on the internet, man. Like, you got-
- JGJoel De La Garza
[laughs]
- FAFeross Aboukhadijeh
... people are gonna, you know, people are gonna figure this six-letter password out pretty quickly." And, you know, um- There's a lot of things like that where, you know, uh, the, the folks that are the top maintainers in the world don't necessarily have the security training or even thinking about these things.
- DADylan Ayrey
Mm-hmm.
- FAFeross Aboukhadijeh
And, you know, they don't have a security team, they don't have, you know, enterprise SLAs, right?
- DADylan Ayrey
Yeah.
- FAFeross Aboukhadijeh
These are volunteers that are just putting code on GitHub. And so it's on actually the users, I think, the, to actually vet what they're using.
- DADylan Ayrey
Mm-hmm.
- FAFeross Aboukhadijeh
It's kinda hard to say, like, you know, we just, we're a company, we just found this code on the internet and we just deployed it straight into prod and, and it's, you know, and it's someone else's fault. [laughs] You know, it's like, no, actually, you know, there's some, definitely some responsibility for, for the users of, of this software to really be, to be vetting the artifacts that they're bringing into their environments. And so I think, you know, it, there's a lot of pieces here and, you know, it's, I wouldn't wanna, um, put too much blame on people 'cause it's a hard problem. But, uh, um, but yeah, I think there's, like, a lot of places where we can do good.
- DADylan Ayrey
Well, let me ask a follow-up to that because you said, um, there are certain package managers that have resources that other package managers don't.
- FAFeross Aboukhadijeh
Mm-hmm.
- DADylan Ayrey
I think one direct example of this, and I don't, uh, cast any blame on them whatsoever, they were actually great to work with, we found a caching issue in RubyGem that allowed us-
- FAFeross Aboukhadijeh
Mm-hmm
- DADylan Ayrey
... to steal arbitrary tokens and get access to arbitrary accounts, which we could use to backdoor arbitrary packages.
- FAFeross Aboukhadijeh
Mm-hmm.
- DADylan Ayrey
We disclosed it to them, they got it fixed quick, but that's an example of an organization that's under-resourced.
- FAFeross Aboukhadijeh
Yes.
- 21:06 – 23:32
2026 Is the Year of the Software Supply Chain
- FAFeross Aboukhadijeh
Uh, I mean, at least for, for us at Socket, I think the biggest thing we're seeing is that, uh, 2026 is the year of the software supply chain. [laughs]
- JGJoel De La Garza
[laughs] That you're dealing with an incident right now as the conference is happening.
- FAFeross Aboukhadijeh
Yes.
- JGJoel De La Garza
[laughs]
- FAFeross Aboukhadijeh
And I noticed the attackers seem to pick RSA and Black Hat-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... as the times they wanna start these npm worms. 'Cause-
- JGJoel De La Garza
The hacker, the, the security guys are out of the office, yeah. [laughs]
- FAFeross Aboukhadijeh
Yeah. Absolutely. So I, I think that's the, that's the thing that's the biggest... You know, I, I think, you know, prior years I was having to educate people. We were, you know, always educating people about this problem, and having to explain to them, you know, this is not a theoretical risk, like, this can happen.
- JGJoel De La Garza
Mm-hmm.
- FAFeross Aboukhadijeh
And we'd sometimes get these reactions like, "Oh, yeah, but like, how, how likely is it really?"
- JGJoel De La Garza
Yeah, yeah.
- FAFeross Aboukhadijeh
And we're like, "No, it's actually very likely. Let me tell you how it could happen." And, and, uh, you know, there were many incidents to point to, but I think this year it's really broken through into the mainstream.
- JGJoel De La Garza
Yeah.
- FAFeross Aboukhadijeh
And there's, like, mainstream publications, you know, um, the business press-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... covering these attacks, right?
- JGJoel De La Garza
Yeah. It's like front page on Bloomberg. [laughs]
- FAFeross Aboukhadijeh
Yeah. Yeah, exactly.
- JGJoel De La Garza
Whoa.
- FAFeross Aboukhadijeh
So, so I think that is, um, you know, that is very, very good because you need that type of, you know, air cover for, like, security teams to actually prioritize and, and find budget for these problems.
- JGJoel De La Garza
Mm-hmm.
- FAFeross Aboukhadijeh
And so I think, you know, despite all these attacks being very, uh, you know, painful to deal with right now, I think in the end we're gonna come out really strong from this.
- JGJoel De La Garza
Oh, yeah.
- FAFeross Aboukhadijeh
Because we're actually gonna, gonna get budget and we're gonna get, um, the, you know, we're gonna do a lot of good this year-
- JGJoel De La Garza
Yeah
- FAFeross Aboukhadijeh
... in terms of solving those problems.
- JGJoel De La Garza
It's inoculation, for sure.
- FAFeross Aboukhadijeh
Yeah.
- JGJoel De La Garza
How about you?
Episode duration: 23:47
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode RtNrvPBkwfA