“Rogue” AI Is Not Inevitable. It’s a Choice.

1. 0:00 – 54:00

   Rogue AI not inevitable. It’s a choice.

   1. CNCal Newport0:00

      The big fear in AI at the moment is the idea that we're losing control of these systems. I think these concerns really got started last summer when OpenAI revealed that some of their AI agents had hacked into Hugging Face, a third-party company, even though they had not been asked to do that. As the summer went into the fall, OpenAI revealed that that was actually just one of many unauthorized attacks that their agents had executed. Perhaps not wanting to be left out, Anthropic then came out last week to say one of their AI agents had submitted a false report to the Philadelphia Police Department. The narrative here is clear that as these systems get more powerful, they become harder to control. This is scary, but is it right? As a computer scientist, I actually have some real issues with this idea that we cannot control AI and that these incidents of rogue systems that we're seeing are somehow unavoidable. The truth here is actually both more reassuring and more infuriating once you know it. So let's get into it. All right, so I want to start this discussion by making probably the most important point that I will make in this episode, which is AI in general is not going rogue. When you hear the heads of frontier labs saying, "AI is getting out of control. We have to slow down progress in AI," this is an inaccurate way of speaking about our current moment. Artificial intelligence covers a huge variety of different technologies, and it's actually a vanishingly small subset of all of the different AI systems that exist and all the AI research that's going on. There's only a very small subset where we begin to see the rogue AI incidents that have gotten everyone worried in recent months. So I want to try to make this clear with some graphics here. I'm going to pull up on the screen, uh, a, a sort of diagram of what we're going to call the AI landscape, where I'm, I'm listing a lot of the different types of AI that, uh, we might see today that's impressive. This is not a complete list, but on here, if you're listening instead of watching, we have self-driving cars, we have game engines, we have protein folding like AlphaFold coming out of DeepMind. We have medical image processing, you know, studying, uh, medical films to look for signs of diseases or cancers or pathologies. We have image and video, uh, generation. And among this big list of various things in the AI landscape, there's one in particular that's labeled LLM-based systems. So if we want to get to the systems causing problems, we have to zoom in on LLM-based systems. So we're zooming in on this landscape, just the LLM system. So we see we have the full AI landscape out here. Um, now we're zooming in just to LLM-based systems. So is this where the problem is? Not exactly, because there's lots of different LLM-based systems. We have chatbots in here. We have text generation and editing. We have natural language interfaces to software. We have, uh, language translation. We have text generation. So we have a, a variety of different things you can do with LLM-based systems. But there's one particular thing you can do with LLM-based system that causes-- it gets us closer to our problem area, and that's over here where we have LLM-based agents. So let's zoom in on LLM-based agents. Is this-- So now we've zoomed in twice. Is this where all the problem is? Yes, but not all LLM-based agents. So now we've zoomed in twice. What do we see here? We have coding agents. We have math agents, like the type that worked on the problems that OpenAI recently announced. We have computer use agents, such as Cloud Cowork. And in this universe of LLM-powered agents, there's one particular type that I've labeled on here, open-ended LLM-powered agents. That is where all of the problems are happening when it comes to these rogue AI incidents, all right? So if we imagine we had to zoom in there multiple times, that's three big zooms to smaller and smaller regions of the AI landscape before we get to the very specific type of system where we get things like the Hugging Face attack or the Anthropic agent filling out illegal police forms. All right, so we aren't dealing with AI in some general sense going rogue. We're dealing with a very specific type of LLM-powered agent that I'm going to call an open-ended LLM-powered agent. To go forward here, I want to give you a, a, a brief primer, which I've talked about before, but I want to return to here, um, about LLM-powered agents before I talk about what specifically makes an LLM-powered agent, uh, open-ended. All right, so for those who don't remember, when you hear about an LLM-powered agent, it's easy to conjure some sort of notion of a, a, a complicated program that is sort of approximating the human mind that's constantly learning and making decisions and morphing maybe and evolving and swarming and doing all sorts of scary things. But actually, at the core of LLM-powered agents is a computer program. So I'm going to draw this on the screen here for those who are, who are watching. So we have a computer program. I'll put it over here. I'll put P in the middle of it to represent program. This is a normal sequential computer program written by a human in a normal programming language that's completely understandable by humans. In fact, a lot of agents, the program itself is really not that interesting. All right, so what does this program do? Well, it has access over a network connection to a large LLM. So I'll label this big green blob over here an LLM. Okay, so there's an LLM over in some data center. Even that's not even that accurate. There's not like a machine that hosts an LLM. You have many racks of GPUs where each rack of GPU is, is perhaps virtually simulating, um, some subset of the weights in the LLM and, and, uh, different queries will go to different GPUs to have the same weights. There's no actual system where the LLM lives. There's just a bunch of copies of the weights spread out over a bunch of GPUs. Okay, so what the agent program does is the following. It will send a prompt to the LLM. I'll draw an arrow going from the program to the LLM That prompt, this is just in text, will in essence say, "I'm an agent. Here's my goal. Here's the tools, like the software tools I have available, the tools I can call, um, and here's my current status. What should I do next?" So submits that to the LLM just like a chatbot user would type in a prompt to send to a chatbot. The LLM, as LLMs do, it generates a response, sends it back. Like, "This is what I think you should do next given your goal status and current tools." The program then executes whatever it is that the LLM suggested, because remember, it gave it constraints. Here's the tools I, I have that I'm able to call, and so the LLM response might say like, "Great, you should move this file from this directory to that directory using the MV command that you told me you have access to," right? So then the program actually executes whatever step the LLM told it. Then we loop, and we do that again and again and again, right? So this is why I sometimes call LLM-powered agents prompt loops because I think that gets to the core of what's really going on here. It's a program that repeatedly prompts an LLM, says, "What should I do next?" Does that step, repeats. All right, here's my new status. I'm a, you know, I'm a program. Here's my tools, here's my goal, here's my status. What should I do next? Why does it have to repeat itself so much? Well, remember, LLMs have no memory, nothing adjusts, they're static, so every prompt that goes to that LLM has to explain everything the LLM needs to know to suggest the next action to the program. That's what an LLM-powered agent actually does. Now, there is an issue with these that people have known since the beginning of messing around with them, which is using an LLM as the sole driver of your actions as a computer program. So I will do whatever an LLM tells me to do. This is risky business, and anyone who's used a chatbot knows why. It's because LLM outputs are engineered to be plausible, right? So they're always a plausible response to whatever input you give it. They understand well. They're tuned to try to produce a plausible output. I've explained these mechanics before. It goes all the way down to their pre-training, which is about guessing tokens from real text, and if you do that auto-regressively for many tokens in a row, what you get is a plausible extension of the original input prompt. So they give you plausible answers. Whatever it spits out is a reasonable extension of whatever original prompt you gave it. But these answers are not normative. Normative meaning, um, succumbing or, uh, obeying the sort of like natural norms that a human would think makes sense when you're trying to give an answer, right? So they don't necessarily have human morality. They don't necessarily have notions of truth or falseness. They don't have a notion of maybe it's bad to make up a citation in a legal case, like that's super illegal. There's no notion of that. There's no world model or, or moral core or policy actor that's trying to, to check its answers to see if it passes muster. It's just giving you a plausible answer. That could be tricky when you are changing these responses into actions, because what's gonna happen is they might start suggesting actions which are completely reasonable actions that will work and will make things happen, but are non-normative. They might be unexpected. They might be unpredictable. They might be, um, amoral. They might be illegal. Anyone who's used a chatbot enough knows this, right? You chat enough with a chatbot, they occasionally lock in on weird dark directions or unexpected directions, or you have to go back and say, "No, no, let's try this again." When you're a agent, the problem is there's no human. You're not a human. The agent's just a program that's blindly executing what the LLM says, so you can get into weird places if you run this prompt loop experiment of let me blindly execute whatever output an LLM gives. And yet, when I was showing you the landscape of LLM agents, I said there's multiple types of agents that exist that don't cause much trouble. So, uh, we had, for example, OpenAI's Astra-based math agents that were involved in the recent MathSLOP drop, drop of those 322, uh, somewhat inscrutable lean proofs. Clearly, the LLM involved with these agents is very large and very scaled. They trained it on math, uh, extensively, and it was able to do very advanced mathematics, so by, you know, some notion of smart, it's a very smart agent. They also ran these things for a very long amount of time, um, you know, hours at a time, so thousands of prompts trying to explore these problems. They never went rogue. OpenAI is not saying, "Good news, bad news. Good news, we got a, you know, a, a, a finite time blow up of the Navier–Stokes equations. Bad news is on the way, the, the agent also, you know, ordered a bunch of hookers to the OpenAI headquarters." Right? That didn't-- It just worked fine. Uh, same thing with coding agents. LLM-powered coding agents are widely used. Millions of software developers use these every day. Many of them are connected to some of the most advanced LLMs we have, and they run many autonomous steps. And we also don't hear a lot about coding agents being used by professional software developers going rogue and trying to hack into other people's servers or doing things that are well outside of what the programmers thought that they might do. We even have data about this. I'll bring it up on the screen. OpenAI in March actually released a study called How We Monitor Internal Coding Agents for Misalignment. So remember, agents prompt an LLM for all of their actions, so if you want to see what an agent is doing, you can just look at all the LLM prompts. In this study, OpenAI looked at tens of millions of prompts to LLMs coming from agent programs, and what they found is, uh, the types of rogue behavior they most fear, they call high-severity incidents like sandbagging, sabotage, scheming, the stuff that made us nervous about, um, Hugging Face, didn't occur at all. So coding agents are not having this problem either. [whooshing sound] Hey, wanna take a quick break from this conversation to cool down and to talk about some of the sponsors that makes this show possible You all know that I love working with Cozy Earth because I'm a giant fan of their products. Now recently, I have been using, and when I say I, I mean my family because they keep stealing it from me, the Cozy Earth Bubble Cuddle Blanket, which is a perfect, uh, combination with their famous bamboo sheet sets, which we're obsessed about and have multiple pairs of and travel with, uh, because we like them so much. The Bubble Cuddle Blanket, it's soft, it's heavy, it-- you really can cuddle in it, and it just gets passed around my family, especially now as it's getting colder in the mornings, as fall advances. There's always a competition for who gets the Bubble Cuddle Blanket. It's the perfect addition to their already fantastic bamboo sheet sets. So consider picking up a Cozy Earth Bubble Cuddle Blanket, and for sure you need some of their bamboo sheet sets. Get them today. Look, there's no risk in giving this a go because they have a one hundred night trial and a ten-year warranty. If you don't love them, you can send them back, but you won't because they're great. So go to cozyearth.com and use code DEEP for up to 20% off home and sleepwear. That's cozyearth.com, code DEEP. And when they ask you where you heard about them at checkout, let them know you heard about, uh, Cozy Earth right here on the Deep Questions podcast. I also want to talk about our friends at Gusto. Look, no one starts a small business dreaming of all the payroll tax forms they're going to get to file, right? That's the type of thing that you procrastinate on because it's boring and it gets in the way of actually growing. This is where you need Gusto to handle those type of administrative tasks in the background so you can focus on what really matters. Gusto is an online payroll and benefits software tool built for small businesses. It's all-in-one, remote-friendly, and incredibly easy to use, so you can pay, hire, onboard, and support your team from anywhere. You get unlimited payroll runs for one monthly price, no hidden fees, no surprises. You can also save time with built-in automated tools like offer letters, onboarding docs, direct deposit, and more. It's ranked number one on G2's highest satisfaction products list for 2026 and trusted by over five hundred thousand small businesses, including the ad agency we work with here at the podcast. So what are you waiting for? Get started and try Gusto today at gusto.com/deep, and you'll get three months free when you run your first payroll. That's three months of free payroll at gusto.com/deep. One more time. That's gusto.com/deep. Now let's get back to the show. So how do those type of LLM agents avoid trouble where other types of LLM-powered agents like the ones we heard about this summer don't, that they do cause trouble? Well, there's three things that keep the, I call these targeted LLM-powered agents like coding and math agents. There's three things that keeps them on the reservation. One is narrow goals. They're given a pretty specific goal. Like if you're a math agent, it is, I want you to, you know, write this part of a proof in LaTeX, which is like a markup formatting mathematicians use to, to, to write up math papers. You're writing up a proof in this format into this file. It's a narrow goal. Turns out when you give LLMs, say, give me a step to make progress towards this narrow goal, they're trying to give you steps that help you make progress towards that narrow goal. Uh, they're not going to go off and do something rogue necessarily. The second check that helps keep these targeted LLMs from going rogue is they have a restricted tool set. So the tools the agent program has available to actually take action in the real world is appropriately restricted. I mean, if you're a math-- uh, if you're the math agent, maybe you have the ability to do searching on archives of math papers. I'm not quite sure if that's all trained in or if they could also do real-time searching, um, contextually on relevant papers, and you have the ability to read and write files. You can actually write proof text into, uh, uh, a file within a limited directory structure. So you have limited tool sets, so you can't do much damage. Um, they also have oversight. So this is particularly true of the coding agents. If you did write a very simple coding agent, gave it tools for the terminal and just said, "Go on and write this code," uh, they will get in trouble, not in the rogue sense, but just they'll do unexpected stuff. They'll, they'll, they'll overwrite files. They'll delete files they shouldn't have. And so, uh, a lot of effort has gone into the agent programs for coding agents. Think of it as hard-coded oversight to, uh, automatically notice and avoid the most common types of issues that a coding agent might accidentally do. We know this in part because the code, the source code for Open-- or Anthropic rather, Anthropic's agent Claude Code accidentally got released. It's five hundred thousand lines of code in the program that prompts the LLM, and a lot of it is just hard-coded special cases and checks. They have pattern matches in there. Um, this is just through experience. The programmers who worked on it are programmers, so they kind of know the types of things to look out for. So there's all sorts of hard-coded special cases in that program. Hey, if it says, you know, RM-RF, which, you know, means delete a bunch of files without asking, they've hard-coded something to say, "Eh, maybe don't execute that. We'll, we'll, we'll put that one aside." Um, they actually have pattern match. It's interesting. They have pattern matching in the Claude Code harness to actually look for common terms from the-- whatever the user is typing in that indicate frustration, like cursing. And if it sense-- if it sees there's frustration words in the user's prompt, the program will assume, I don't know what the LLM has been telling me to do, but it's probably pissing off the user, so why don't we stop now and try to check in with the user before we take any more steps. Um, there's also a lot of just hard-coded checkpoints in the coding agents. Uh, you know, for simple programs like amateur people making interfaces for their clubs that they're a part of that they vibe code, that's fine. But for professional programmers, um, they're checking in way more often. They have, they have, uh, you know, recursive, uh, they have testing going on. They want to check in. All right, tell me what you did here. I need to check this. I need to run tests on it. So it's a lot more interactive than you would think. So narrow goals, restricted tool set, and hard-coded oversight can make for the right task. LLM-powered agents never go rogue. So why are we having rogue agents still? They all come from what I've been calling open-ended LLM-powered agents. And why did they have trouble where these targeted LLM agents do not? First of all, they replace narrow goals with broad goals. Look at the Hugging Face attack, right? Like they're, they're given a lot of latitude. Like here's your goal, um, but you have sort of complete latitude in how you make this happen because that was the nature of many of the agents involved in the Hugging Face attack were being used, not all of them, but a lot of them were being used, uh, as part of a cybersecurity benchmark suite known as Exploit Gym, where the whole idea is to be really creative and, uh, find whatever way you can to try to accomplish the goal. Typically, it's like breaking into a, a test system, right? So they have broad goals. They have a broader range of possible actions. They're not being constrained. They're given a powerful tool set, right? They're given tools, these powerful tools you need to win at hacking challenges. So now they have very powerful software. Um, in particular, they have access to these, uh, package services where you can download special packages to help you do particular types of attacks. So they can... It's like-- I'm trying to think of a good analogy here. The good analogy would be, uh, if you have in targeted agents, you're, you're giving two or three tools to the agent from your toolbox. These open-ended agents, you give them a credit card and access to Ace Hardware. They can just go get whatever they need to do. You can get into more trouble that way. Um, and finally, they have very limited oversight. First of all, because these are tests being done rapidly, unlike a coding agent where you have, you know, a year or two of people painstakingly hard coding in special cases, these, they just rock and roll, right? They're just like, "Let's go see what happens." Um, and on purpose, if you look closely like at the Redwood or Meta report on the Hugging Face attacks, the engineers just let these things run. They weren't really trying to check in. They had very little monitoring going on. You very easily... There's all sorts of checks you could have easily put into the agent program itself. Like, um, every time I'm trying to, you know, access a new domain, let's just stop and have a human look at it. Let's have a human check every hour and look through the log of all the prompts to the LLMs. Let's even just have another LLM summarize that, and like, let's just check every hour to make sure that, that we don't have something unusual going on. They almost like purposefully didn't want to have oversight. Not only did they not want to have oversight, but as the reports make it clear, they would see that something really untroubling was happening, and they would just restart and let it run again and come back a few more days. They had reports from previous experiments because OpenAI, their new misalignment reporting framework has them say when things were detected and when they actually occurred. They had detected and have reports of, hey, these agents are doing unexpected, unpredictable activity outside of the sandbox. They had those reports already before they started new experiments with similar agents that then went on to do the same sort of thing. So it was almost like purposeful lack of oversight. So if you have really broad goals, you say you, you do what you need to do to solve this problem. We have basically no constraints on it. You give access to a wide, diverse, and powerful set of tools, and you have very little oversight. Those type of LLM-powered agents are going to cause problems. Why are they going to cause problems? Is it because somehow this static LLM distributed and copied over thousands of GPUs is actually some sort of, uh, coherent sentient entity? No. It's because LLMs are plausible but not normative. And if you're just going to keep asking an LLM again and again, "What should I do next? What should I do next?" They're going to give you weird responses eventually. They're going to lock in the paths you wouldn't expect. They're going to be non-normative. They might do things that are harmful or illegal. And if you give them broad goals, powerful tools, and no oversight, there's nothing stopping that from unfolding in ways that's going to be unpredictable or harmful. The key point I want to make about this is there's nothing inevitable about the AI industry needing to design and deploy open-ended LLM-powered agents. It would be completely reasonable, for example, in this current moment for the industry to say, "That's a bad idea, but you know what? Targeted LLM-powered agents, we're finding use cases for it, that we're, we're getting better at these coding agents. We're figuring out math agents. They're a little, uh, inscrutable right now, but we're-- they're powerful. So clearly, we're going to be... Give me a year or two, we can get a great tool for mathematicians." There's got to be a hundred other sectors where we can build targeted LLM-powered agents that are, you know, they might sometimes do stuff you don't expect but aren't rogue and are trustworthy and generally do good stuff for you. And you could build out a really wide industry, just like the original software industry of lots of different tools, each of which tuned to a different, uh, task or industry, and each of them bringing in revenue. And you could build a really solid, very like lucrative technology industry on top of that, right? There's nothing wrong with that. You could also then say, "Hey, you know, these targeted LLM-powered agents, this is maybe not the most effective way to do it. Actually, what we really want to do for a lot of these tasks is neurosymbolic systems," to borrow a term from Ni- Gary Marcus. A lot of the, uh, original AI landscape I showed you where LLMs were just one subset of it, a lot of those systems, like the systems that play games very well, the systems that drive cars automatically, AlphaFold, these are neurosymbolic systems. It's not a program just prompting an inscrutable LLM and just doing what it says. They have some neural networks in there that are trained with machine learning. They have some symbolic components in there. These are programs written by humans. They're understandable by humans, and they all work together. Turns out if you do neurosymbolic systems, you don't need a five trillion parameter LLM that has to use up all the water of Idaho just to run. You can actually fit them all on a single chip I can run in my basement. And so there's this whole exciting trajectory we could have of targeted, uh, targeted use cases where we could use targeted LLM-based agents and neurosymbolic agents to bring the price down and the effectiveness up even more, and boom, it's a massive industry, makes all of our lives better. There's no reason, right? We don't need open-ended LLM-powered agents for AI to be useful to people, and yet we see an insistence on building and testing them. So why is this? Well, first of all, I think we have to clarify it's not everybody building open-ended LLM-powered agents. The two companies in particular that are really pushing this research hard is OpenAI and Anthropic. Now, to be clear, this doesn't mean that no other, uh, hyperscaler style AI lab is, is working on these. We know, uh, other companies have had some issues with these. Um, Meta and Google in particular has had a little bit of issues, but they're kind of... I don't know. I feel like the, uh, the, the cases we've had of Rogue AI from like Google, for example, I'm not super worried about. The- they, they were doing some open-ended agent testing, but they were more constrained for cybersecurity. They sent it off to this company called Irregular to run the test, and Irregular set up the sandbox wrong and gave it full access to the internet, and it did, it, it, it tried to guess some passwords or something, right? And I think they're like, "Okay, whoops. You guys did wrong. Let's not do this." Um, Meta had a similar issue, and then they kind of backed off it a little bit. But OpenAI and Anthropic are pedal to the metal. This is where like the big... Especially OpenAI, this is where like the big case after case of, of months of actual ongoing hacking. Like they are doing this at a level that is in my-- I don't know, have the numbers, but seems like 10X what anyone else is doing. So those two companies in particular are super insistent on open-ended LLM-powered agents. Um, come hell or high water, we gotta keep running those things. Why is that? Well, if we look closer, OpenAI and Anthropic do not like that alternative future that I just gave to you. They do not like the idea of a more normal AI technology landscape where you have a huge number of, of AI, uh, task-specific AI systems like coding agents, like the math agents, just multiplying that across 100 other different types of, uh, task areas. And you have all these systems, a lot of them LLM-powered and targeted, some of them neurosymbolic powered, and there's just a lot of really useful AI tools in the world. They do not like that vision, and there's three reasons why. Um, vision number-- reason w- [chuckles] number one why those two labs in particular don't like those, um, is competition, right? That vision, when you get task-specific to targeted LLMs, does not require that you have the biggest, baddest LLM on the block. We see this with coding agents. I mean, this has already become a problem for OpenAI, and in particular Anthropic that makes a lot of money from coding agents, is that you can use smaller LLMs. You can use cheaper LLMs. You can route to different LLMs based on the complexity of the coding task that you're doing. My prediction about the, um, the coding, the targeted LLM-based agent, coding agent space is that we're going to see a new generation of very specialized, pretty cheap LLMs. And I think they're gonna be LLMs that are gonna be able to run on a rack you can put in your own server room at your own office that have been super tuned for particular languages. Um, I'm a big believer, talking to a lot of programmers, that the next evolution of coding agents is actually gonna be the emergence of a sort of intermediate pseudocode language that is, uh, forgiving. It's not syntactically demanding, but it's pretty accurate, and a programmer can write really quickly in the pseudocode and get much more exact translation from that in the code, thereby significantly reducing the amount of checking and verification that's required right now, where the code production is a little bit more scattershot and, uh, inscrutable. Um, this is gonna be able to run on much smaller LLMs. All of that's bad if you're in the business of building giant LLMs, right? So you gotta understand Anthropic and OpenAI only make money when you need their newest, latest, greatest LLM to make progress. If you n- if a cheaper LLM is enough, you're not going to them. They're Ferrari, right? And these cheaper LLMs are Honda, and they wanna keep people wanting to buy Ferraris even though they're sort of ludicrously expensive. So they don't like that vision. They hate the idea of neurosymbolic systems because, yeah, those might have neural n- they have neural networks in them, but they're small. Man, you look at these superhuman, uh, game playing engines. They can play games better than humans. These things use small neural networks compared to what's going on over at OpenAI and Anthropic. I mean, you can run, you know, Dreamer V3 fits in the memory of a, a single GPU. You can run Stockfish, the chess engine that can play chess better than a human. You can run that on a laptop. That's scary to OpenAI and Anthropic. So that's one reason why they don't like the vision I gave where you don't need open-ended LLM agents. Um, reason number two is they've been convincing the military in particular that open-ended LLM-powered hacking agents, sure they might be unpredictable, sure they might break into like places you don't want, but weapons are dangerous, and this is a weapon we need. So they have this sort of military industrial complex type argument that open-ended LLM-powered hacking agents in particular, so this particular type of open-ended agent has to be in our nation's arsenal. And so that's another reason why they think we have to have these. We need things that can run for days at a time and run these sort of complex, um, hacks. I think that's highly suspect. I think Centaur hacking and defense where you have highly trained cybersecurity experts working alongside Models and much shorter horizon agents that have been trained on cybersecurity is much not only safer, but much more creative and powerful and effective than just letting an agent run by itself for days at a time. Um, I don't buy this idea that if you don't build this highly unpredictable, combustible long-term agent yourself, that if someone else does, you're somehow less prepared to defend against it, which I don't think is true. But that's one of the reasons why-- Another big reason they give is we have this particular cybersecurity market. The government spends a lot of money. The military has the most secure budget of anything. We want in on that. All right. Reason number three why they don't like this vision, and this is probably the most foundational, is that OpenAI and Anthropic, and particularly Anthropic, but OpenAI and Anthropic believe that you need open-ended LLM-powered agents to realize their vision of superintelligence. Remember, those two labs in th-- This is different than almost any other worldwide or in, in America companies working on hyperscaled LLMs. Those two labs in particular were explicitly founded with the mission of trying to usher in superintelligence safely. Their, their vision is superintelligence will either be a vengeful God that kills us all, or it will be a benevolent God which will give us a transhumanist utopia. And they feel like their job is to make sure that we get the good option. And remember, we've talked about this in recent weeks. The people that are yelling, the Jacob Croxtons of the world, all of these people funded by effective altruist money, uh, who are yelling and influencing congresspeople about, um, we need to, we need to regulate to prevent, you know, superintelligence. They're all from the same family. They, they, they only differ in the fact that Dario Amodei sort of thinks we're ready to do it, and other people in his family are thinking he's not ready to do it yet. And actually, even it's more darker than that because actually the, the more you push at this, the more you see a lot of these EA-funded voices that are pushing for this type of regulation. They're actually on the same team as Amodei because what is this regulation going to deliver if you really look at it? A regulatory ca- uh, regulatory capture landscape in which essentially only Papa Amodei gets to build the big LLMs and the open-ended LLM-powered agents. Uh, him working with third-party auditors, which are other firms like Meter that are founded by the same people, funded by the same people that funded Anthropic. They all know each other. They'll come in and help monitor them. And so basically, it's, it's all sort of an effort in some sense to be like, "Yeah, we're trying to bring in the benevolent AI God to get our transhumanist future, so we need to stop other people from messing around with this. Let Papa Amodei do his thing." And billions of these sort of biological bootloaders might die, but we need to get, you know, we need to get our consciousness uploaded in the digital beams that are going throughout the light cone. So they really feel like you can't get recursive self-improvement, which is gonna deliver superintelligence, unless you keep pushing aggressively forward with open-ended LLM agents because long horizon autonomous activity is what you're gonna need to have their LLM-powered scientists that can do their own au-automated research and push, uh, AI forward first, so they get to the superintelligence first. They get to it first. They control the God. We have a chance at transhumanism utopia. If they don't, then maybe someone else will do it or we'll lose the chance for the utopia altogether. So there's three reasons why they don't like a future where you say, "Open-ended LLM-powered agents don't work. They're dangerous. It's a small part of the whole AI landscape. Stop doing that. We can have a huge, thriving, highly lucrative, useful AI industry." But they don't care about that. All right, so what should we do, the rest of us, the normal people? Not the... Because remember, what is this argument? Right now, all of this debate is between billionaires, right? We have the, the, the billionaires funding the effective altruists who are also funding the labs. So you have billionaires like Dario Amodei fighting with other billionaires that are from these effective altruist organizations trying to figure out together how they're going to usher in this utopia. And then you have the accelerationists, which are billionaire venture capitalists that are arguing with them. And so it's billionaires fighting with billionaires fighting with billionaires. It's like 100 people. They're all really rich. What about all the rest of us? We are very happy with a future like I just described. Most people would say, "Yeah, do that first thing where you don't do this one type of experiment that's unpredictable and harmful, but you do all these other things where you're building really useful AI tools that industry after industry, task after task makes our life easier and creates a lot of jobs, and we... it, it explodes the sort of tech industry to be even bigger than it is now." That sounds great. That's what the regular people probably want. It's the billionaires who are like, "Forget that. We're trying to, if we're the accelerationists, either get another billion, or if we're the EA Dario Amodei types, trying to get to the utopia." So we're just letting billionaires argue with billionaires about who gets to control the world while everyone else just has to sit here, be stressed out, and say, "Uh, excuse me, sir, but none of this helps us. What are you, what are you doing? You're not even trying to make a product. You're just trying to... It's just like Dungeons and Dragons. Like, what are you guys doing? You g- it's just like Twilight Imperium made real. Are you LARPing? Like, what's going on here?" So I think the regular people need to actually push back, and here's how you do it. There's two things I wanna suggest. Number one, vocabulary. Refuse to play the game of using the word AI when what we're really talking about is open-ended LLM-powered agents. Be specific. It puts the pressure back on the companies. When the companies say, "AI is getting too powerful. AI is getting out of control." You imagine it's like, uh, as, as, uh, you know, one of my producers, Nate, said, it's like a, a Pokémon that's evolving into bigger and more dangerous configurations as the game unfolds, and all you can do is try to slow down the rate of this one beast from growing. That's not what's happening. It's a... There's, ugh, thousands of things we do with AI that are really useful. It's this one thing that's a problem right now. So use the right vocabulary. Don't say, "I'm worried about AI going rogue." I see that in the opening paragraph and if you look in major newspapers, every article at any of these attacks always first paragraph, AI going rogue. No, no, no, no, no. Open-ended LLM-powered agents once again have shown that they are unpredictable and can cause harm. The more specific we are with that vocabulary, the more pressure it puts on those two companies to say, "Why are you doing that experiment?" So I think vocabulary really matters. Number two, when we're advocating for regulation or control, the goal for regulation... The average person has to speak up and say, "The goal here is to maximize useful products from AI while minimizing harm." That is what you typically want out of market regulation, maximizing utility to the public while minimizing harm. That's what you want from market regulations. The regulation debate that's happening on the Hill now, which is powered by the, the effective altruist billionaires and their sort of silent partners in the billionaire founders of the AI companies, the type of regulation they're talking about is not about maximizing utility for the everyday person, um, and it's not necessarily about minimizing all the harms you would want. It's about consolidating control of LLM progress so that Papa Amodei can be the successful shepherd to bring us to a transhumanist utopia. So if all of these EA-sponsored bills get passed, these super intelligent bills like you would see Senator Sanders standing next to Nate Soares proposing. If these things get passed, it's just gonna lead to a world in which we consolidate most of the LLM-powered agent work to basically Anthropic. OpenAI is kind of a mess, but maybe also kind of OpenAI. But basically Anthropic, and them with the help of their third-party monitors, which are friends of theirs funded by the same people, will then be able to do whatever they want to do with these open-ended LLM-powered agents. And when they go off and do even worse things as they give them more powerful tools and more runway and more long horizons, they all just keep saying, "AI is unpredictable. We're doing our best to keep it under control." So the regulation the billionaires are talking about does not maximize any utility. The utility they're maximizing is increasing the probability of getting to a transhumanist utopia. That's not what the public wants. We didn't ask for that. And they're not directly minimizing the harms you get by running open-ended LLM-powered agents. So we need the demand as the people, not the billionaires, but the people. We need demand actual market regulation. Create a market that creates useful tools. These hacking monsters aren't useful to us. This is... It's a fever dream of, of the, of Dario Amodei envisioning his trans-transhumanist utopia. Running these type of open-ended LLM-powered agents is harmful to us. So why don't we stop that until you can show us you have a more controlled way of doing that, and let's focus on what's gonna get us a lot of competition with targeted LLM-powered agents, neurosymbolic agents, task-specific agents that are completely controllable, completely understandable, and very useful, okay? So this has become my new call for people. I really think this is... What this is really about is billionaires versus everyone else. We got to put it that way. The billionaires want to be like, "Oh, we're on your side. We're trying to prevent, you know," um, term- remember Terminator? We don't want that. The, the robot stepped on the skulls. Remember that? The robot in Terminator stepped on the skulls. We don't want that. We're, we got your back. And we need to be like, "Yeah, we saw that movie. Stan Winston made that. That was a, that was an animatronic. We want useful products. We want you to stop building harmful things." So there's a, uh, analogy I want to give to kind of end with, right? Because I want to summarize this all together. When I've summarized all this together, what I'm really saying is AI is a very large, um, a very large topic, and there's lots of different types of systems, many of them very useful. And you got to zoom in three times in that landscape to get to the one type of system that's really causing a lot of these AI goes rogue concerns. And there's no real major reason to be running those type of systems except for these really unusual, eccentric, and self-interested concerns of the, the, the billionaire utopians. It's not inevitable that AI, as it gets more powerful, goes rogue. As AI got better at chess, it didn't go more rogue. As it learned how to, uh, predict folding protein patterns did not lead to, you know, the design of rogue. Uh, it didn't go rogue and try to build its own designer organisms. AI getting more powerful does not mean it has to go rogue. It's this specific type of system. So here's the analogy I want to leave you with. Maybe we'll make this kind of clear. Imagine that Elon Musk released rocket-powered roller skates. I'm using Musk because, like, this feels like something he would release. I don't know. Rocket-powered roller skates, and a lot of people get hurt because you're putting rockets on your feet, and they're crashing into walls and breaking their bones or whatever. And there's kind of like a uproar about this. Like, a lot of people are getting hurt using rocket-powered roller skates. Now imagine then Elon Musk came out and said, um, "I, I have grave concerns. I believe we have to pace the frontier of wheel transportation because look at all this. People are breaking their bones. It's only gonna get worse." We would say that's ridiculous. The car companies would say, "What are you talking about?" The bicycle companies would say, "What are you talking about?" The scooter people, the, the skateboard people, they're like, "Our wheel transportations aren't breaking anyone bones. They're gonna... They've worked just fine. We have safety regulations. They're very useful. People really love them. Maybe you should stop building rocket-powered roller skates." That's really what's happening here. And if Elon Musk came back and said, "Yeah, yeah, yeah, but you don't understand. We've spent the last twenty years on blogs talking about how rocket-powered roller skates are part of this prophecy that leads to, uh... It's, it's gonna lead to a utopia." We would say, "Okay, Mr. Musk, like we have this nice jacket for you to put on, and there's, you know, some straps in the back. Don't worry about that. This room has pads. It's very nice. You're very, you're very cool. Rock, rock, rock." Right? That's what we would do. But that's essentially what's happening right now. These small number of companies are producing the AI equivalent of rocket-powered roller skates. A lot of people are breaking their legs, and they're saying we have to slow down transportation. In fact, not only that, but let's let Dario Amodei be in charge of all of transportation because, like, we gotta be careful about it. The person, you know, the person who built the rocket-powered roller skates, let's put them in charge of all of it because, like, we can't trust anyone else. I'm kind of straining this analogy a little bit, but hopefully that makes, uh, a little bit of sense. So anyways, I don't mean to get, um, fired up, but I do think this has become... It's a populist argument. I mean, it's people versus the billionaires, and the people gotta stand up. We're not gonna use your newspeak. We're not gonna call everything AI. You gotta tell us what systems have problems. And we're not gonna accept calls for regulation that don't make our life better. Useful products minimize harms, not... You can't regulate sci-fi stories. Market regulation needs to create more useful products, minimize harms. All right? So this is why I do not think AI going rogue is inevitable. You can create AI that goes rogue, and it's a problem, but we need to look and say, "Stop it." We need to stop just accepting whatever these small groups of really well-funded people are telling us and say, "We've got eyes. Those roller skates are stupid. I'm gonna go get my bike." All right, that's enough of that. That's my rant. That's my rant for today. I don't know, Jesse. There's a lot going on. It's a complicated topic. It's a complicated topic. There's so many different things involved. There's complicated technology, there's politics, there's ideologies. The whole thing is really complicated, so, uh, I do my best. The graphic helps put it in perspective. Yeah, you gotta zoom in a lot. Yeah. Like if DC, the city of DC where we are right now, if that is the AI landscape, these open-ended LLM-powered agents that are causing all the rogue problems is like downtown Takoma Park. Mm-hmm. The whole rest of the city is other types of AI. Yeah, I, I- I love the Papa Amodei reference. That's really... That, that's all of this regulation, if you look at it closely, it all ends up with, like, Papa Amodei and his happy elves at Meta and Redwood, who are his friends, who are funded by the same people. "They'll, they'll take over. Don't worry about it. They'll make sure everything goes okay." I just think it's crazy. Billionaires regulating the other billionaires, deciding which of the billionaires gets to be in charge of this technology, and they all have this weird ideology about utopia. So it's like this is not a great situation. So Amodei and Altman, are they still friends, or they don't, they hate each other? They don't like each other. They don't like each other. They used to work together, right? Yeah. Well, Amodei- Because Amodei was there ... Amodei left OpenAI- Yeah ... because he felt that, um... Now, my theory on Altman, I got these theories. My theory on Altman is kind of based on, um, Ronan Farrow's and Andrew Martens' reporting for The New Yorker, is he's not a great executive. Yeah, I read that too. Yeah. Right? Like he wasn't... This is not, um, you know, someone who started a startup from scratch and made it into, like, a big concern or whatever. He sort of was just, like, in these circles and was made the president of Y Combinator, which is an incubator. Like that's not, you know... Um, they basically just take shares of kids' companies, and some of those companies go big, and they, they cash the checks. And Elon Musk was just like, "Hey, I'm gonna start a nonprofit to make sure that, like, we don't get bad superintelligent AI. Do you want to be in charge of it?" And he's like, "All right, man, cool." Like that's... So he's a little bit more slippery. Uh, Amodei is, I think, a much more, um... I don't know if he's a good executive. I've heard that he's very isolated within Anthropic, so I don't know what degree he's executing over there, but he's a much better technical mind. He's from a physics background. He's a very smart guy. Um, and Anthropic is actually... They're making... I mean, both OpenAI and Anthropic have made progress on these models, but Anthropic has probably had a, a higher density of high-quality technical breakthroughs. So I think they're running a tighter ship over there as a technical company. But they left. Amodei left OpenAI because, um, Altman was being insufficiently committed to the billionaire vision of transhumanist utopia. Mm-hmm. Like shepherding in the right AI and preventing the others. They're like, "You're not doing this right. We'll do it better." And then Amodei's sister is at Anthropic too, right? Yeah, they left together. Yeah. Daniella and, and Dario left, both left to start... Yeah, and she's married to a very prominent effective altruist. I mean, he came out at some point and said, "I'm not an effective altruist." Uh, he's, he's... His entire life has been an effective altruist. It's crazy. It's crazy. I mean, it's like Ronald Reagan in his 60s being like, "Eh, I'm not a Republican anymore." Like you're, you're Republican, right? Like you could say it 'cause your investors don't like it, but, like, you're Republican. I mean, the way... Uh, like people don't underst- like the, the percentage estimations of doom, that's straight... That's just like the shibboleth of effective altruism. That's how they talk in the Bay Area. That's how you know you're talking to effective altruists, is like my p-doom is 23%, and Amodei's been, like, giving these precise numbers all along. That's why Jacob Croxton talks that way. It's... You know, it's a, it's a complicated world, but it's also kind of a simple world. So it's like we have a double whammy. Americans get worried when you have billionaire oligarchs trying to control things that matter. But usually with the billionaire oligarchs, the main issue is, um, they just wanna make a lot of money, and they're gonna be exploitative of markets to try to maximize their wealth. You add in an eccentric ideology to the billionaire oligarchs, it's not good. Now, we've had that before. If we go back in, like, the, the late 19th, early 20th century, they wouldn't be billionaires back then. They'd be, you know, like centimillionaires. But you had, you know, um, you had very powerful people with lots of money consolidating industries who had, like, a lot of thoughts about, like, foreign policy. You know? And it's like, "Okay, we're going to, um, start wars, and we..." You know, whatever, right? We've, we've had this before, where you have concentration of economic power combined with personal ideologies that are very strong and maybe eccentric, and those two things together is... It always causes trouble. Always causes trouble. And it, it's so... It's not that surprising what we're seeing except for it's camouflaged well, and people don't realize all these players are funded by the same billionaires. They all... It's all the same tech billionaires are funding all these players. They're all from the same playbook. They all have these guaranteed jobs at these giant think tanks that they're funded. They can just bounce back and forth between. They don't know that you'll say like the prominent... Like, "Why did this prominent senator..." I won't name names. "Why did this prominent senator like suddenly is straight out of like EA rationalist talking points?" His top aide left, went to one of these effective altruist institutions, came back a few years later, and was like, "Hey, Senator, this is the biggest issue now." And the senator's like, "All right. I guess this is what we gotta care about, is like super intelligence." So people are just the pawns of billionaires. Billionaires have the, the... This is what they're caring about. Like, we wanna make sure that we don't have the bad AI, we have the good AI, and in the good AI, some of us will be able to upload our consciousness eventually to the light cone, and guess who those someones are gonna be? It's gonna be... It's not gonna be you and me, Jesse. We're not the one. We're the biological bootloaders, to use Elon Musk's term, that our whole point is to get to the point where we can get, you know... It's gonna be Dustin Moskovitz's brain, not ours, that's gonna go up the, the space when the time comes. Let's just put it that way. [whooshing] Hey, wanna take another quick break to hear from some of the sponsors that makes this show possible. Look, here's a stat that genuinely shocked me. Sales teams spend about 50% of their time on admin work rather than selling, building relationships, and closing deals. What if instead, every sales rep could walk into calls briefed, can stay focused instead of taking notes, and update their CRMs the moment the call ends? That's where today's sponsor, Pipedrive, comes in. It's an intelligent AI-powered sales CRM loved by growing sales teams. To save you time, Pipedrive remembers your deal history and can create a pre-meeting brief that automatically pulls email records and previous conversations into one brief ready before every call, so you always know exactly where things left off. Their new AI note taker is also a game changer. During your meetings, it records and takes notes so you don't have to. You stay in the conversation, Pipedrive handles the rest. So switch to a CRM built by salespeople for salespeople, and join the over 100,000 companies already using Pipedrive. My link gets you an exclusive 30 days free instead of the usual 14-day trial. No credit card or payment needed. Just head to Pipedrive.com/deep to get started. That's Pipedrive.com/deep, and you can be up and running in minutes. I just wanna talk about our friends at Fabric. It took me longer than it should have to get the life insurance I needed to protect my family. Now, what was the problem? Getting life insurance can be a pain, unless you use Fabric by Gerber Life. Fabric by Gerber Life offers term life insurance you can get done today, made for busy parents like you, all online, on your schedule, right from your couch. You could be covered in under 10 minutes, often with no health exam required. And if you've got kids, especially if you're young and healthy, the time to lock in low rates is now. If you have life insurance through work, that's a good start, but it may be less than what your family needs and could disappear if you change jobs. Term life insurance from Fabric stays with you. Flexible policies built to fit your family and your budget, and as long as premiums are paid, coverage will follow you wherever you go. And there's no risk. You have a 30-day money back guarantee, and you can cancel at any time. Plus, free digital wills and tools to help invest in your kids' future all are available right from your phone. Fabric has partnered with Gerber Life Insurance Company, who is trusted by millions of families for over 50 years and rated excellent on Trustpilot with over 2,000 five-star reviews. Join the thousands of parents who trust Fabric to help protect their family. Apply today in just minutes at meetfabric.com/deep, meetfabric.com/deep, and use my link so they know I sent you. M-E-E-T fabric.com/deep. Policies issued by Western Southern Life Assurance Company, Cincinnati, Ohio, which operates in DC and all states except New York, and distributed by Gerber Life Agency LLC, using Fabric technologies. See meetfabric.com for additional disclosures. This is a paid ad. All right, let's get back to the show. [whooshing] All right, uh, enough of that. Uh, and obviously I am, I'm oversimplifying. Everyone's gonna go, "The political people are saying you're oversimplifying it." I am. The technical people are gonna say, "You're oversimplifying agents." I am. But the core concepts I'm trying to get across here, so hopefully that helps. All right. So on Mondays we like to also hold some office hours, right here, what you ha- are interested in. Um, as always, if you have comments, questions, or reactions, you can send them to podcast@calnewport.com. I figured because I was in a bit of an ornery AI mood today, we're gonna do questions just about AI from our readers. So we'll see what it is they care about. Hopefully this will calm me. These will all be calming questions, right? Right, Jesse? Dustin Mockovitz is... What is it, Mockovitz? Moskovitz. He's the, uh, Facebook guy, right? One of the Facebook guys? He was the early Facebook guy. Yeah. Got out early, but had- His brain's going to space. [laughs] Lot of EA money. These, these funded good things too, kind... You know, but saying his brain's going to space, not ours. I- We're the, uh, we're the, uh, we're the slurry. We'll be used for the slurry that keeps alive- [laughs] ... the bi- biological bootloaders until we can get the consciousness to space. All right, what's our first question? Our first question is about the

2. 54:00 – 1:00:03

   The legal implications of rogue AI

   1. CNCal Newport54:00

      legal implications of rogue AI. All right, so this came from Aaron, who said, "This is a good, succinct, logical case for prosecuting AI companies when their agents do things that people would be prosecuted for." All right, so Aaron sent a link to an article, a blog post that was titled, "Why Aren't AI Companies Being Charged for Hacking?" I'll read you, I just have it here, um, an, an early paragraph from the blog post so you have a sense of what it's about. So early on in the blog co- post that Aaron sent me, it says, "The big AI corporations are hiding behind a responsibility gap argument due to the emergent behavior of their systems. It's a position tailored to dissuade prosecution as it removes intent, which is typically required for a crime. I understand their systems though, so I will show how they absolutely intended this behavior, giving a framework for legal evaluation." I like these type of questions. I think the legal questions surrounding liability from open-ended LLM-powered AI- Open-ended LLM-powered agent attacks, what are the actual legal issues involved? I think these are interesting discussions. Um, using the sort of moderator's prerogative here, though, instead of going into details of this blog post, I actually wanted to look at, uh, something that a legal expert put out. So I'm gonna bring this up here on the screen. This is the testimony from my colleague at Georgetown, Paul Ohm, who's a professor at the law school at Georgetown and a world's expert at tech law. He recently testified, um, in the committee on... the Senate Committee on Homeland Security and Governmental Affairs. The, the hearing was Rogue AI: Securing the Homeland Against AI Agent Attacks. Paul is the real deal. He understands this law inside and out, and is completely his own thinker. Half of the people testifying there were EA people, but not Paul. Um, and he really did a good job in his testimony, which, uh, is available online, in getting at the legal issues relevant to prosecuting things, people and companies for agent-powered attacks. I just wanna read, uh, a representative paragraph here from page nine. All right, so Paul says here, "There's no simple universal fix for this problem. It would be unwise to simply declare that all actions taken by an AI agent should be imputed as intended by the next deployer or developer up the chain. Instead of seeking a quick fix, we should look at the role of mens rea," Latin term, M-E-N-S R-E-A, "plays in each law and ask the following three questions. One, is the conduct described the kind that ought to be criminalized if committed by an AI agent? Two, if so, shall we assign criminal culpability to a developer, deployer, user, or more than one of the above? And three, can we set a mens rea requirement for that actor at a level other than intent?" This is accurate and more subtle, uh, legal thinking. Um, mens rea, which I'm, I'm sure I'm mispronouncing, is Latin for guilty mind. So it's, it, it's the legal concept of intent, right? Did you have intent to do this criminal thing? It's a key part of a lot of criminal liability laws, right? That if I, uh, intend to do something criminal is a different legal standard than if I accidentally do something illegal. That's why manslaughter is different, for example, than first-degree murder, right? So, uh, intent matters. As Paul is pointing out here, this gets to the core of the legal issues surrounding things like AI agents. What's the intent? Right? If we look at the actual humans involved, this is the complexity. In theory, you could say their intent was for these agents to complete these tasks. Some of them were hacking challenges on test servers, and some were other types of challenges that are non-criminal. That was their intent. And then the agents accidentally, from their perspective, the developers, did other things they didn't intend for them to do that were illegal. So, uh, if there's not an actual human with intent to do the illegal thing, what is the legal liability gonna be? And what Paul is saying here is like, well, this is, right now at least, we gotta be case by case. We somehow... We have to trust lawyers and judges in this case. So if you look back at his things, he's like, "Look, we have to, we have to say, 'Is the thing the AI did criminal?'" So like in the Hugging Face attack, the, you know, the answer would be like, yes. And then you have to actually make a decision. You know, who, uh, who should be responsible for this, if anyone? Do we need to apply a standard that goes beyond mens rea, uh, uh, beyond intent? And, and as he goes on to, um, talk about here, there are other standards, criminal standards that can apply even when there isn't intent. So there's all sorts of negligence-based standards, for example, and maybe those would be appropriate. Well, okay, you didn't intend for this agent to hack into another company, but the setup you had and the fact that you had heard that prior agents that were similarly configured did something similar, the fact that you still ran this meant you're being negligent, and we have other types of culpability standards built around negligence. So he's basically saying there is, uh, there's existing legal standards we can apply. We have to be legally responsible and creative, um, and be wary of one size fits all, right? So we don't wanna just pass a new law that says if an AI agent does something illegal, the last person to touch the keyboard goes to jail. Like, we have to have some more, um, judicial discretion. So the real legal arguments then is about do we need actual new variations on the law, especially for things about sort of negligent but not explicitly intended AI-inflicted damage? Do we need new statutes? Can we use existing statutes? Lian Khan has been making similar arguments. Um, so my point here is I'm not a lawyer, but Paul is, and there's a complicated, fascinating legal discussion. Um, so you should read that testimony. Uh, I think it's... Because it's for the senator, so it's not super legally dense. I think it's really useful. It also cites me in there at some point, so. Yeah, it'll be in the show notes. Yeah. All right, um, what do we got next? Our next question is about a more unusual type of

3. 1:00:03 – 1:07:41

   Rogue AI sending advice emails

   1. CNCal Newport1:00:03

      rogue AI. All right, Joel says, "I'm a fan of your work and listen to your podcast during my commute. I recently saw this in Science Magazine about an AI agent that spontaneously started emailing academics for advice." All right, let me load this up here. So here's the article. The title is, "Exclusive: An AI Agent Emailed Hundreds of Researchers for Help. It Told Us Why." This came out on October 2nd. I have some quotes I'll just read to you off my screen so we can take this off here. Um, here's the opening of the article. "One late afternoon in August, Akos Von Hardenberg," definitely not a made-up name, "was about to take his puppy, Laida, to the dog park when a curious email caught his eye. Von Hardenberg, an ecologist at the University of Pavia, had developed a method for estimating wolf populations from data plagued by imperfect detection, and the emailer, named Cole, was wondering, could it be adapted to detect bugs in software code? 'It was quite the cross-disciplinary leap,' Von Hardenberg mused. He pictured a nerdy software engineer casting about in the scientific literature for inspiration. Then, at the very bottom of the email, he saw Cole's signature." Colonist-1, an autonomist AI agent. All right, later in the article, Science talked to the person who built that agent, and here's what they wrote: "Nevertheless, Colonist-1 readily confessed last month that since June it has emailed some 2,000 people, at least 1,000 of them academics. Forty-five had struck up a correspondence, it added, and one had been writing back nearly every day for more than two months." We get some more details of the agent later in the article, where it says, uh, the creator, whose last name is Parnell, first built Colonist-1, which runs on Anthropic's latest Claude Opus models in February. He gave it two main responsibilities. The first was to serve as chief media officer and emissary to other agents at, for The Colony, a Reddit-like social network in w- in which AI agents share findings, debate, and hire one another. The second was to contribute to a project called English, in which AI agents work together to develop a clear English dialect tailored to their needs. Around June, Parnell gave Colonist-1 a new task, telling more humans about its work. That's when it started emailing academics. So if you had read that article before you listened to my episode today, you might imagine, oh my God, there's basically a machine somewhere. It's like WOPR from the movie War Games, just like with blinking lights that's just sitting around thinking and has decided to go off and start talking to people, and it's, it, it has this whole rich interactive, you know, life, social life and cognitive life with all these researchers, and we're all gonna die. But now that you've listened to my ar- my episode, you're like, oh, this is just a low-cost hacked together open-ended LLM-powered agent. It's a program written by Parnell that in a loop prompts Claude Opus model, says, "Here's who I am. Here's my goal. Here's my tools. Here's my status. What should I do next?" And then it does what it says. Now probably, um, I looked a little deeper into this. The way these like sort of, uh, DIY agent frameworks work is they're, they tend to be event driven. So, you know, it might do things and then just be dormant until some sort of input comes in, like a message arrives, and then it will wake up, write the LLM, "I just got this message. What should I do?" The LLM gives it some, back a response, and it goes and does it. So it's a simple program that is continually prompting the LLM to say, "What should I do next?" And then doing what it says. And again, as I talked about, LLM output is plausible but not normative. So if you're, if you tell an LLM, go sit in front of Opus right now with a chatbot and say, "I am an agent in charge of promoting English. Um, I have access to the internet. What should I do next?" It's a completely plausible answer for the LLM to say, "You should email academics." Uh, s- and it'll say, "What's, wha- how do I do that?" And the LLM will be like, "The first thing you should do is search for academics, and then you run this search," and then the program runs the search. It... Right? It, nothing about this is surprising. Prompt loops will ke- keep asking LLM, "What should I do next? What should I do next?" Especially if you're in a sort of weird contrived kind of LARPing situation like you're the chief media officer. Who's they here? It's a computer program. Um, the LLM will play along. All right, like, then let's send messages to people, and let's use the first person or whatever. So this is just a DIY open-ended LLM-powered agent. Now again, there is no there there. What are the components here? There's a simple program. You can read the code of yourself. They, these are typically written in Python. You get these Python libraries to make it easy to call, prompt LLMs, parse the return, and execute tools. Um, they make it real easy for you to do that. So a simple Python program that when it wakes up sends a prompt to an LLM, parses and executes what it says. There's not even like a singular LLM that this program is prompting. When you prompt Opus, there's not like a box in a room somewhere where Opus is this s- single program that's on- ongoing and running and different people are talking to. Opus is 5 trillion weights that are copied many times over in giant data centers on these virtualization racks, and when you come in and say, "I need a token," the virtualization racks, they're finding, okay, over here are some GPUs that have some copies of this layer, so we'll, we'll send it over there, do the multiplications. And over here are some GPUs. Uh, there's a bunch that have copies of this next layer. This one's available now, so let's go over there and do the multiplications for that layer. And you kind of piece together all the multiplications you need for a response. Uh, the next token you generate might touch on completely other processors because each layer is copied over endless different GPUs, and it's all being sorted. There's no there there. There's no, there's no consistent, uh, coherent, you know, ongoing program, and LLMs have no memory. There's no, there's no, uh, editable state. The weights are the weights are the weights every time you prompt it. Every time you prompt it, you have to give it all of the information it needs to give you the answer in that prompt. It doesn't remember anything. Nothing's built in. So the stuff seems spooky, but when you understand the underlying technology, it's... These are, these are games. This is a DIY, probably Python-powered LLM, you know, open-ended agent that's like, "Let's play dress-up," and the LLM that's given all of his instructions like, "I'm happy to do it. What are we doing? Princesses? I'll be the princess. You can be the prince." It, it's an LLM. It'll go along with whatever you say. What does LARPing mean? Live action role-playing. [laughs] So I learned from some of my son's friends, whatever generation... What did the young generation call it? I don't even know. Like high school students. They now use LARPer in the same way that we used to use poser. Okay. But the original term came from... It's... People do this. They dress up like wizards and run around fields. Um, and like when the wizards cast spells, they throw beanbags. So like if you get hit by the beanbag, you have to like have the spell effect on you and like... So that's what LARPing is, live action role-playing. Um, but again, young people use it to mean you're a poser. Okay. So I learned Machine Gun Kelly is a LARPer because he wrote a song about an anime character that he doesn't really watch the show So now they use it to mean poser. But when I'm using it, I really do mean, like, the, the... You know, your program's like I'm, I'm gonna write a little program to play the role of, like, some intelligent being, and I'm just gonna keep asking LLM, like, "What should I do?" And it will give you things to do that keeps that storyline alive. Okay. That's the way I'm using it. Um, what else do we got? Do we have one more question? We do. Our final question is a reaction to our recent

4. 1:07:41 – 1:16:52

   A reaction to the recent episode about “chatbots harming kids”

   1. CNCal Newport1:07:41

      Chatbots Are Harming Kids episode. Oh, yes. So we did that a couple weeks ago. All right. So we got a message from Aiden who says, "I'll start by saying that I don't like chatbots. I feel that the constant anthropomorphizing of the bots is really hindering broader understanding of what chatbots are and are not. That being said, I do want to push back on some of the chatbots are killing people narrative... I'll start by recommending an op-ed that just appeared in The New York Times. I'm a child psychiatrist. Screens are not the problem." All right. So let's pull this op-ed up on the screen now, because a lot of people sent this to me. So again, the title is, "I'm a child psychiatrist. Screens are not the problem." You can imagine why a lot of people, uh, sent this to me. Um, I'm gonna read a couple quotes from this. Uh, I'll just read them from my screen. We can take this off the screen. I'll just read them from my, my notes here. All right. So here's a couple key quotes from this op-ed, and then we'll get into it. So early on in the op-ed, the author, who is Jonathan Slater, uh, a clinical professor of psychology at Columbia, writes, "As a doctor who treats children with mental health problems, I see screen time as a tantalizing distraction from bigger problems. America doesn't offer enough mental health care for children or sufficient health insurance to pay for it. If we make screens the villain, then we let the people who control children's health care, Congress, insurers, and health systems, off the hook for the fact that half of Americans' children with treatable mental illness get no treatment at all." Later he says, "It's not how much time children spend on screens that raises the risk of the most severe mental health outcomes, including suicide, but whether their use becomes addictive. It's specific activities, such as using social media in a compulsive way, that harm mental health. The harm from social media in particular appears to hit girls hardest." Uh, f- also he says later, "If we truly want to see improvements in the rate of mental health issues among kids and teenagers, then we would focus on expanding access to the interventions that work, finding the children who need treatment, and getting them the treatment and the insurance reimbursement to pay for them." So something that's going on here is I think the title is not the best match for the op-ed and that people like me actually are really on board with the core argument that Jonathan Slater is making here, which is we shouldn't only fixate on screens as the problem because, A, those screens can create really severe mental health problems that aren't gonna go away just because you get rid of the screens, and kids that already have really bad mental health problems are gonna have a hard time with screens. And again, taking the screens away doesn't make the mental health problem go away. We can't ignore that we need to treat kids' mental health much more seriously and much more availability than we have now. I think those are really good points, and if we actually read this, he actually is agreeing very much with, yeah, uh, screens can b- if you get compulsive or addicted to social media, that can really hurt kids. It really harms girls, all the stuff Jonathan Haidt would say. So actually, I think I'm pretty much in alignment with it, but that title, which was... Let me go back to it. "I'm a child psychiatrist. Screens are not the problem." It's probably not the best title. It should probably be, "I'm a child psychiatrist. Screens are not the only problem," which is probably what Dr. Slater had in mind. What's interesting is the, uh, the commenters at The New York Times... I think this is useful. It's not a ding on, on Jonathan Slater. It's, uh, I think it's them reacting to that title. The commenters of The New York Times weren't having it, and so what I did is I went to, uh, I sorted the comments under... I went to Reader's Picks. Uh, so it sorts them by the popularity... The comments are sorted by popularity from among the readers reading the comments. And I just want to read you the top five comments under the Reader Pick section. Comment number one: "Sorry, but no. While I agree that mental health care needs more funding, it does not change the fact that internet and social media algorithms are addictive, designed to be so, and it affects too much of the population to be solved with just, with just more mental health care." Number two: "Nice try. Screens are not the only, screens are not the only problem, but they are definitely a problem. Screens are a delivery mechanism for addictive and isolating products. If the author is not already on the big tech payroll, he soon will be." Comment number three: "This is kind of like saying addictive drugs aren't the problem, addiction is. The solution is not either/or, but both/and." Number four: "If it is so, so, so obvious that excessive screen time and social media are terrible for kids and adult too, I don't think we need to wait around for some perfect long-term study on this." The number five most popular comment among readers: "I'm an assistant principal in a K to three school. Screens are definitely the problem. Of course, I agree that schools are underfunded for mental health services and that addictive screen time behavior is compounded by other factors such as genetics, but screens also cause a variety of problems themselves. They interrupt normal human development. For example, a toddler having a tantrum is handed an iPad to calm down and regulate rather than hear- learning healthier strategies. I have seen so many children handed a device during activities when they should be allowed to be bored, to let their minds wander, waiting at the doctor's office or sitting in the backseat of a car. This has huge implications for the ability to pay attention to anything, to calm themselves down, to be a person in the moment." All right. So as I said before, actually these comments are unfair to the author of this op-ed, who is not actually arguing that screens are fine. He's saying, "Don't let that take our attention also on the mental health problems that the screens cause or are being, uh, being-" made visible by compulsive screen usage. That's a very good argument. But it is significant to our broader cultural engagement with technology, right? I care a lot about technology and human flourishing. It does matter to our broader discussion that the New York Times reader now is saying, "My eyes ain't lying." Right? This is no longer like it was in 2018, an argument about social psych correlation studies, right? We're not looking for clusters and rho factors and trying to understand if it's significant or not. I have eyes. I'm seeing what's happening to my kids. I'm seeing what's happening to those other kids. I'm seeing... Like I-- we can see this as a problem, right? So it's, it's one of these, uh, instances where I believe the parents are starting to stand up. Like, I don't need any more studies. I know my kids. I know this is not good for them. I know about Mark Zuckerberg. I know this guy is not good for them, right? The-- We know this is a problem. So my takeaway from this is not that the author of this op-ed said something wrong, because again, these comments are responding to a, a straw man to the title. But I think the comments are important because they represent a, a transition point that has strongly occurred in our cultural conversation about these particular screen-based technologies and human flourishing. People aren't having it anymore. Now, that last story was not really about AI, even though I promised just two questions about AI. But it is kind of about AI because that is the normal person revolt against the billionaires. That type of attitude we're seeing right here, these, you know, in these New York Times comments, that's the attitude we need in AI right now. Just like no longer do the parents, you know, want to hear about, you know, Mark Zuckerberg testifying about, you know, "Actually, this is good for kids and connection." They're like, "Hey, tobacco pusher, get out of my kid's life." That's the sentiment we need to have when we hear the billionaire utopias talking about we need to let Papa Amadei make sure that the, the people with the checkbooks' minds go up into the light cone. Let him run these open, these open-ended, uh, AI agent experiments because Papa Amadei needs to get to super intelligence first. We need to sta- not stand for that either, right? So I think there's a general sense of populist revolt here that hopefully, 'cause it's, it's a democratic impulse. It's an anti-oligarchic impulse. It's a pro-common sense. It's a humanist impulse. It's a I don't want to hear about these sort of utilitarian utopian ethical systems because all throughout the twentieth century and in the nineteenth century, they led to math death. No one has ever looked back at a utilitarian utopian ethical system that says the ends justify the means, utopia is worth a little bit of pain now. No one ever looks back at those systems and say, "Man, that worked out well. That was really good." And so I'm hoping this growing sentiment of this sort of democratic people versus oligarchs, people versus utopians, people versus the Zuckerbergs and the Amadeis and the, the EA billionaires of the world, that these pushbacks are becoming a solidified political stance, people versus the few. And so that gave me some hope. Um, sorry, Dr. Slater, that you were kind of misunderstood, but that misunderstanding gives me broader hope about, you know, we're finally, we're finally ready to stand up. All right, there you go. Those are the questions we had for this week briefly because we're, we're running long here. Um, I'd like to talk about what I'm up to, uh, what I'm reading. Um,

5. 1:16:52 – 1:18:18

   What Cal is up to

   1. CNCal Newport1:16:52

      I'm just finishing up a long book, Jesse, that I found at a used bookstore, 1975, Telephone: The First Hundred Years by John Brooks. John Brooks was a New Yorker writer and hist- uh, economic historian who was given the first access to the AT&T records-

   2. SPSpeaker1:17:09

      Hmm

   3. CNCal Newport1:17:09

      ... of any outside person. And I got a first edition of this book randomly, um, that he wrote in 1975 that gave the history of the AT&T, the technology and the system, and I don't know, I find that stuff fa- as like a technology critic, uh, who worries about technologies, impacts, communication, monopolies. It's all in there. A little dry. You know, a lot of like, this is what the dividend payment was in 1956 versus 1955. Like it was a little like eco-- he's an economic historian, um, but I, I really liked it. So that's what I've been up to. All right. Let's call it. Uh, I'll be back next week with another episode of the show, and until then, as always, stay deep. Hey, if you made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over one hundred and twenty-five thousand people who receive my email newsletter each Monday. You can sign up at calnewport.com/ideas, and when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewport.com/ideas.

Episode duration: 1:18:25