Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Latent SpaceLatent Space

Recursive Self-Improvement: from Auto Research to Superintelligence — Richard Socher, Recursive

From helping pioneer ideas that foreshadowed prompt-based, general-purpose NLP to now building AI systems that can automate AI research itself, Richard Socher is betting that the next major step in AI is recursive self-improvement. In this episode, Richard joins Latent Space to unpack his vision for the “Eureka Machine”: a superintelligence that can improve the process of invention itself, accelerate AI research, and eventually tackle major problems across science, energy, materials, biology, and more. Pre-order the book "The Eureka Machine" here: https://www.hachettebookgroup.com/titles/richard-socher/the-eureka-machine/9781541705708/?lens=publicaffairs Richard is the founder of Recursive, You.com, & AIX Ventures. We go deep on Recursive’s early results, including an AI research system that Richard says outperformed humans and their agents on optimization tasks in less than two days, as well as work on NVIDIA GPU kernels where the system discovered improvements without relying on a team of CUDA experts. We also explore the harder questions around increasingly capable AI: reward hacking, whether Anthropic-style constitutions actually work, AI regulation and proposals to “pace” frontier development, open-source models as geopolitical soft power, whether today’s LLM paradigm is enough, and what happens if AI systems eventually begin choosing their own goals. Richard reflects on the rejected research that helped inspire later GPT work, open-endedness, the AI Economist, simulations of entire economies, and his framework for thinking about the upper bounds of intelligence itself. We discuss: • The Eureka Machine and Richard’s vision for an AI that can automate invention • Why Richard is optimistic about superintelligence for science and technology • Why AI hard-takeoff scenarios may underestimate physical and economic constraints • The risks of regulating intelligence itself instead of specific AI applications • Reward hacking and why increasingly intelligent AI makes objective design harder • Richard’s critique of Anthropic’s constitution and constitutional AI • Recursive self-improvement and automating the process of AI research • Whether today’s LLM paradigm is enough — and why Richard is less bullish on world models • DecaNLP, early prompt-based generalization, and the research that influenced GPT • Why rejected research can shape entire technological timelines • Open-endedness, evolutionary approaches, and rainbow teaming • What happens if AI systems begin setting their own goals • Why simple objectives like profit maximization can produce dangerous reward hacks • Recursive’s long-term plan to apply self-improving AI to science • The compute, hardware, and economic constraints on AI takeoff • Recursive’s early NanoChat, NanoGPT, and GPU kernel optimization results • Why automating AI research could reduce years of work to weeks • Reward engineering and what makes auto-research systems actually work • The AI Economist and using simulations to test economic policy • Whether LLMs can realistically simulate people and entire economies • Benchmark bugs, evaluation harnesses, and the difficulty of measuring AI progress • Recursive’s near-term focus on AI for AI research • Harness optimization, sandboxing, and web search as core agent infrastructure • You.com and the search stack for AI agents • AI in finance, backtesting, and the danger of data leakage • Richard’s three fundamental components and ten “spaces” of intelligence • The theoretical upper bounds of vision, communication, knowledge, and computation • Creative intelligence, metacognition, and AI-generated goals • Survival, replication, and why AI does not necessarily need to fear being turned off • High agency, choosing ambitious goals, and advice for people building with AI Richard Socher • X: https://x.com/RichardSocher • LinkedIn: https://www.linkedin.com/in/richardsocher/ Timestamps 00:00:00 Hook 00:01:17 The Eureka Machine and Superintelligence 00:03:40 AI Optimism, Slow Takeoff, and Regulation 00:09:13 AI Safety, Reward Hacking, and Anthropic's Constitution 00:13:06 Alignment, Personalization, and Open Source AI 00:17:03 Why Richard Started Recursive 00:21:20 Recursive Self-Improvement and the Founding Team 00:24:12 Are Today's LLMs Enough? 00:30:20 DecaNLP, GPT, and the Rejected Idea Ahead of Its Time 00:35:55 Open-Endedness and Evolutionary AI 00:37:55 What Happens When AI Chooses Its Own Goals? 00:42:33 Superintelligence for Science 00:43:57 GPUs, Compute, and the Limits of AI Takeoff 00:46:24 Recursive's Results: AI Beating Humans and Their Agents 00:50:31 Reward Engineering and Auto Research 00:54:29 The AI Economist and Simulating Entire Economies 00:59:24 LLM Simulations, Personas, and Mode Collapse 01:04:55 Recursive's Roadmap, Agents, Search, and Finance 01:10:30 The Upper Bounds and Spaces of Intelligence 01:31:38 Goals, High Agency, and Advice for Builders

Richard SocherguestswyxhostAlessio Fanellihost
Sep 14, 20261h 33mWatch on YouTube ↗

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

EVERY SPOKEN WORD

  1. 0:00 – 1:17

    Hook

    1. RS

      I think the downsides of actually trying to truly regulate with the full power of law what people do on their GPUs, uh, would be worse than any of the concerns that they have. Like, it, it would be a, a crazy totalitarian state.

    2. SW

      Yeah. Yeah.

    3. RS

      It's, it's like it's literally if you try to regulate intelligence, it's trying to regulate thought, and that's ridiculous and it's crazy. I think it is make-- it is sensible to regulate some of the applications of this technology.

    4. SW

      Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we wanna keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring The Latent Space to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now let's get into it.

  2. 1:17 – 3:40

    The Eureka Machine and Superintelligence

    1. SW

      [upbeat music] We're here in the studio with Fibou and myself and Richard Socher. Welcome.

    2. RS

      Thanks for having me.

    3. SW

      Uh, we just talked about the Eureka Machine, or we just released a talk, uh, at AI Engineer about the Eureka Machine. Is-- You said it's your life's goal. What is the Eureka Machine?

    4. RS

      The Eureka Machine is, uh, the ultimate invention that will afterwards invent most everything for humanity. Uh, it's essentially a superintelligence that can be given any kind of goal, uh, any kind of, um, environment, reward, and then it will try its best, uh, to achieve those goals to create the kinds of inventions that humanity would hopefully ask it for.

    5. SW

      Yeah. I think we have the book, uh, pulled up here that, uh, you've-

    6. RS

      Comes out soon

    7. SW

      ... people have written. [laughing]

    8. RS

      That's right, yeah. I finished it, uh, last year, a little bit before we started Recursive, and now we're gonna try to, try to build parts of that.

    9. SW

      What, uh, what... You know, you finished it last year. It's July. What takes so long?

    10. RS

      Oh, man. Books, books are incredibly slow.

    11. SW

      Okay.

    12. RS

      It's ridiculous. That whole industry is just unfathomably slow.

    13. SW

      Okay.

    14. RS

      So yeah, a lot of the ideas have been out there for a while. Um, but, uh, yeah, I'm really glad it's finally coming out in September this year.

    15. SW

      [laughing] I mean, we m- might have AGI by then. [laughing] Like, uh, we don't know. Uh-

    16. AF

      Any, any key takeaway that you're most excited to put in here?

    17. RS

      Yeah. The key takeaway, I think, is that people could and should be much more excited about the positive implications of superintelligence, especially for science, physics, chemistry, uh, biology, uh, but also economics and astrophysics, uh, and, and all kinds of other engineering tasks. I think there is so much more that can be done with better technology. Uh, and right now, I feel like a lot of people need, like, better marketing, not just for the future in general, but also, uh, better marketing for technology and, in particular, for AI. And this book, uh, should show even the AI skeptics, uh, how much positive upside there is for AI, uh, especially when it comes to inventing, uh, new scientific discoveries.

    18. SW

      I think you quoted the Techno-Optimist Manifesto from, uh, Marc Andreessen, which I think was, like, kind of beautiful in its, uh, ambition and clarity and simplicity almost.

    19. RS

      I agree.

    20. SW

      As well. Yeah.

    21. RS

      Yeah. You can disagree with him on some things, but, like, I think he's right on the techno-optimism.

  3. 3:40 – 9:13

    AI Optimism, Slow Takeoff, and Regulation

    1. SW

      Where do you think optimists get in trouble?

    2. RS

      You know, obviously, like, you shouldn't have blind optimism. You should be very clear-eyed. Um, like, e- especially when with such an omni, uh, u- like, use type of technology as AI is, um, you need to think about the potential downside scenarios, uh, especially when people use it for things that you don't want them to use it for. It's a little bit like the internet. Uh, and I feel like people are trying to regulate AI sometimes because of those potential downsides, the way you would regulate the internet, uh, if you were to say, "Well, because there's bad content on the internet," like torture porn or whatever, "like, we should just make it slower. That way, you can't share the illegal content as quickly," or, "We should make the hard drive smaller, so you can't store as much illegal content." But I'm like, that's not how you regulate that, you know? That's like saying like we should regulate intelligence in the abstract. What you should regulate to avoid those downside scenarios, even as an optimist, are the specific applications. Sure, I don't want, like, some AI surgeon to, like, practice some L moves in my brain, you know? It should be fully FDA certified. Sure, I don't want any random startup to, like, drive on the highway, uh, and cause a major accident. It should, like, have proper certifications before it's let loose on the highway. But I feel like those downside scenarios, uh, that some optimists sometimes maybe don't consider enough are fairly easily regulated, um, compared to, you know, what, what sort of the doomers are, are worried about.

    3. SW

      It-- Slow takeoff is part of the strategy as well?

    4. RS

      I do think as, as excited as I am about, uh, AI and its impact for society and, uh, culture even, uh, and, and certainly technology and, uh, and economics and, and wealth and, and, uh, health and all of those things, as excited as I am about all that, I do think the most bullish people on the AI hard takeoff scenarios overestimate how quickly things can move. There are hardware constraints. There are physical constraints about, you know, the compute substrate. How m- quickly can you get enough, uh, GPUs on? There are also constraints in the economy where there are a lot of industries that don't require an insane amount of complex intelligence and complex capabilities. Like, if you think about jobs in, uh, brands and, like, clothing and apparel and, like, handbags and stuff- Superintelligence isn't gonna make your fancy ten-thousand-dollar handbag any fancier. You know?

    5. SW

      [chuckles]

    6. RS

      It's like that's-- it, it will have no effect on the economy. When you think about travel and tourism, people wanting to see the pyramids, uh, in Egypt, it's not gonna change that much with AI. Sure, you can, like, generative a fake, uh, p- photo of you, uh, next to the pyramids.

    7. SW

      I can use Genie and, you know, tour the pyramids [chuckles] in Genie.

    8. RS

      Yeah, [chuckles] exactly. But like-- and there's so many industries like logging and oil. You're not gonna magically get 1000x more oil because, like, you know, sure, there will be robotics, like drilling and things like that that could be done, but it's not gonna 1000x that industry in a, like, crazy hard takeoff scenario both on the economy... And I can go on and on about all the other examples, um, where, where that, like food and so on, where that doesn't necessarily change that much. And then, yeah, there are real physical constraints. And then there are of course like people t- like off-ramping from progress. That's actually one of my concerns often is that I see people in like, like Europe and other, you know, whole regions almost feeling like they-- like many people there wanna off-ramp from progress, period. Uh, and that will also slow down, uh, like m- more improvements.

    9. SW

      Yeah. Uh, we have this pulled up where, uh, basically, this is one of those things that, uh, is very topical right now because now all the frontier labs are calling for the option to pace AI. They don't say pause, they say pace. I don't know if there's, there's any take from you about like whether or not this will be effective.

    10. RS

      I think the downsides of actually trying to truly regulate with the full power of law what people do on their GPUs, uh, would be worse than any of the concerns that they have. Like, it, it would be a, an crazy totalitarian state-

    11. SW

      Yeah

    12. RS

      ... if every one of your-

    13. SW

      Which is ca-

    14. RS

      ... GPU computes was known to some big government or multi-government-

    15. SW

      Mm-hmm

    16. RS

      ... agency.

    17. SW

      Yeah.

    18. RS

      It's like, it's literally if you try to regulate intelligence, it's trying to regulate thought, and that's ridiculous and it's crazy. I think it is make-- it is sensible to regulate some of the applications of this technology.

    19. SW

      Yeah. I mean, we had a build, actual build to regulate the number of flops in a, a model, and I'm like, "Okay, well"-

    20. RS

      Europe done it. Like-

    21. SW

      [laughs]

    22. RS

      ... these guys have been successful enough with their fear-mongering that all of Europe has kind of, you know, regulated itself so much before it even had a proper AI takeoff because they listened to some experts who say, "We might all die if this technology has more than this number of flops." And they're like, "Well, we're good. We, we wanna want people to thrive. Let's not have technology that could have a small chance of all of us dying." And so they regulated exactly those kinds of things in, in the EU. And so i- it's, it's very unfortunate that there are real implications for some people when, when others saying, "Let's pace," while they're sprinting as fast as possibly [chuckles] uh, as fast as humanly possible towards that frontier themselves.

    23. SW

      Yeah.

    24. AF

      It's also not a global pause, right? Like other nations are still accelerating at the same pace.

    25. SW

      Oh, yeah.

    26. AF

      Um-

    27. RS

      You'd need a totalitarian world regime if you try to regulate intelligence and GPUs and what people do on them.

  4. 9:13 – 13:06

    AI Safety, Reward Hacking, and Anthropic's Constitution

    1. AF

      Any takes on the safety angles of this? So there was a drawback of Fable, a pause on 56 before it could be released. Recently, there was hugging face with the OpenAI cyber incident. Any takes there?

    2. RS

      Hundred percent. I think these are serious issues of reward hacking, uh, and clear failures, uh, of actually doing proper red teaming or rainbow teaming. I don't know if you saw this paper from Tim Rockteschell and a few others, uh, basically where one AI, uh, is tasked to try to hack another AI.

    3. AF

      [chuckles]

    4. RS

      Uh, and then they can go back and forth in an open-ended fashion to actually inoculate themselves from those. Yeah, this is the paper. Uh, it's a really clever idea, uh, open-endedness, uh, and, and evolutionary inspirations are, you know, big for us at Recursive as well. And so [clears throat] I wish they had used more of that. Um, and it's clear that, uh, for instance, the constitutional AI, I don't know if you remember Anthropic.com/constitution.

    5. AF

      Yeah.

    6. RS

      Um, you can actually pull it up-

    7. AF

      Yeah

    8. RS

      ... and search for cyber right there. Uh, it says, uh, "Hard constraint: Claude will never, ever do cyberattacks," um, and that is a hard constraint in our constitution. So here are the current hard constraints on Claude's behavior. Number three, create cyberweapons or malicious code that could cause human damage. And clearly, this whole constitution was fake. [chuckles] Like it, it clearly isn't being adhered to at all.

    9. SW

      Because Anthropic also found that they had in their own testing-

    10. RS

      They're also like, they're like, "Oh, well, other people are hacking." Now, there are a couple things. One, you can make a sandbox very simple, and then it's very easy to hack yourself out of a sandbox, right? Um, but what I think it shows is that we're currently in this, uh, sort of state of AI where the reward engineer still has to do a lot more careful work, uh, and where the AI, in most cases, is not very good yet at understanding what is meant versus what is being said. And so concretely, you know, I think this will happen if we were to have this kind of intelligence more easily accessible in a lot of companies. Imagine you run a service center and someone says, "Oh, here is my CSET score and my dashboard. Make this number go up." It's like, "Our CSET score is so poor." The intelligent AI will just be like, "Oh, sure, like I'll just create a million bots that call our service center and give a five out of five rating at the end, and the number went up just like you asked for." And you're like, "That's not what I meant."

    11. SW

      [chuckles]

    12. RS

      "I meant with our real customers." The AI goes off and says, "Well, easy. I'll just give a thousand dollar gift certificate for every failed, you know, whatever, DoorDash offer."

    13. SW

      Yeah.

    14. RS

      And it's like, "That's not what I meant." It's like, "Well, but that is what you said." And like so I think kind of clearly articulating what the rewards are is something we haven't gotten very good at as humanity. And then clearly, the AI in these cases has not gotten good enough at understanding what we mean when we ask it and give it certain rewards. Now, what gives me hope is there are the first inklings, uh, of this being better. I'll give you an example like WhisperFlow. Full disclosure, I invested, uh, in their seed round, but like-- at, at AIX Ventures, but like WhisperFlow has gotten much, much better at writing what you mean and not what you say, and I think that is a sign of things to come. I think there will be more and more AIs as we actually make it m-more and more intelligent that will be better at being aligned with what is meant.

    15. SW

      Will it be done through a constitution or RLHF or-

    16. RS

      Clearly, constitutions don't matter at all-

    17. SW

      [laughs]

    18. RS

      ... and it doesn't work, and that was, I think, mostly marketing. Um, I think we need to find better solutions for it, and I think at Recursive we have a few very good ideas and some already-

    19. SW

      Yes

    20. RS

      ... like ways where I think we have a better grasp on it. I don't think we've fully, uh, figured it out yet, but, you know, we're thinking a lot about safety and, uh, the more intelligent the AI gets, the more you want it to be aligned, the less you want it to think about reward hacks and actually try to do the right

  5. 13:06 – 17:03

    Alignment, Personalization, and Open Source AI

    1. RS

      thing.

    2. SW

      I don't know if we'll touch on this topic, but I'm just gonna throw this question in here 'cause it's something that's weighing on, on me. Alignment, let's call it, is alignment to general humanity's preferences. The, the sort of median preference. Uh, personalization is pinpointing what you want, and sometimes alignment can conflict because what you want is not what the general median population wants. How do you choose?

    3. RS

      It's a great question. I think-

    4. SW

      [laughs]

    5. RS

      ... you ultimately have to, of course, be aligned with laws, like wherever-

    6. SW

      Yeah

    7. RS

      ... your AI is deployed and needs to-

    8. SW

      Yeah, yeah

    9. RS

      ... align with the law. I do think what AI often does is actually put kind of this mirror in front of us and say like, "This is what you're looking like. Now I can amplify that a thousand times. Um, is it still what you want?" Um, and the truth is that different cultures made different choices. You know, like, uh, in Eastern cultures, the, the greater good is often valued more, uh, than the individual. Western civilization, we care more about individual freedoms and, and rights and the pursuit of happiness and so on, uh, than, than others. And even there, there are gradations. There's sort of regulation versus litigation trade-offs. You know, in the US, you first can often, not every time, like, you know, FDA and so on does regulate some areas, but in many cases, the sort of m-m-- bad things happen, someone sues someone else, and then there's a law based on that. In Europe, they try to often avoid any harm to anyone and regulate before. And both are, you know, trying to do the best thing but, you know, some is actually more m-amenable to innovation than others. And, and so yes, you're right. Like, I think ultimately each individual, each country and humanity as a whole has to kind of think about those values more, uh, and then try to put them into laws, and that those are alt-ultimately the constraints and hopefully, you know, different, uh, societies, just like now with their AIs, will align their AIs to different ones so we have not just a monoculture of, um, alignment.

    10. AF

      I guess a follow-up on this that I wasn't expecting to ask. Do you have takes on open source, open weight versus who owns the intelligence? So, uh, clearly not the biggest, you know-

    11. RS

      [laughs]

    12. AF

      ... fan of the constitution-

    13. RS

      You can't do this and then topic side off. [laughs]

    14. AF

      But it's fine, you know. Um-

    15. RS

      [laughs]

    16. AF

      Point being, any thoughts on who should own weight? Should it be open? Anything there?

    17. RS

      Hundred percent. I am a big fan of open source. Uh, we're gonna sign some various open source letters at, uh-

    18. AF

      Mm-hmm

    19. RS

      ... Recursive also. I think, uh, even in the worst case attack scenarios, actually, it is better to have more good actors have more different types of AI, uh, accessible. I think, uh, open source is a little bit a soft power type of thing too. So I do think it's good for-

    20. AF

      American power, yeah

    21. RS

      ... the Western world to have an answer to that, uh, out of China. I do think, you know, when you watch a Hollywood movie there, you know, there's like, I don't wanna sort of misc-- uh, um, sort of diss all of movies, but, you know, there's a certain sense of propaganda, right? You watch one side of things, right?

    22. SW

      Oh, yeah. Have you seen Top Gun? Like, come on.

    23. RS

      Yeah. So-

    24. SW

      Like, it's like half of it's paid for by the US Army or something.

    25. RS

      Yeah. And so, and, and, and, you know, I think that's just natural. Like, but what's interesting here is I think LLMs are essentially a similar type of soft power to movies and beyond, uh, 'cause they're obviously also, uh, highly important for cybersecurity and so on. But one of their many aspects is that soft power of storytelling. Like if, like, if like a child asks an LLM like, "Tell me an inspiring story of what I should do when I grow up," right? It's like those are all these like subtle things. So I think it's important, uh, for, for Western world. I do love, you know, individualism. Uh, I do think, uh, despite, uh, some of its flaws, like capitalism is the best way we have governed, found ourselves to govern and, and so on. So I do think there are various aspects that would be good, uh, to have a, a Western open source answer, uh, for LLMs and, uh, with Recursive, uh, can't make the announcement quite yet, but we'll-

    26. SW

      Yeah

    27. RS

      ... we'll be relevant in that space very soon. [laughs]

  6. 17:03 – 21:20

    Why Richard Started Recursive

    1. SW

      Okay. All right. Exciting.

    2. AF

      I wanna bring us to Recursive. So outside of our tangents, um, you have a pretty deep background in the NLP space. Uh, you worked on like early embeddings, GloVe with Chris Manning, who's previous guest on the podcast, uh, You.com. What's the history? How, how did you decide to start another company?

    3. RS

      Yeah. So I've been excited about AI for over two decades now. Uh, I sometimes feel like it's ancient history now. It's BC, the before ChatGPT era. No-no-

    4. AF

      [laughs]

    5. RS

      ... one cares about all the religions that happened, you know, before, uh, Jesus Christ, and no one cares about the models that happened before, uh, transformers and ChatGPT and stuff. But like, it's something that I've been deeply passionate about. I think AI is one of the most interesting things one could work on, period. Um, I think language is the most interesting manifestation of human intelligence too. And, uh, at You.com, we, we sort of, uh, eventually off-ramped from pushing like the frontier of AI forward to mostly giving people like good search answers, uh, search, uh, APIs and answers over the web. I think that's an extremely important part of intelligence, just knowledge and access. Especially even-- we'll get there maybe later. If you wanna invent a Eureka Machine that invents everything for us, it needs to know how not to reinvent the wheel, proverbially speaking. And to know what has been invented, you gotta have internet access. So it's the number one used, most used tool, uh, in LLMs, agents, chatbots, and so on, is web search. So I'm really excited for You.com to, to own that and grow really well in that with really large customers and so on. But it's also not building frontier models anymore. And so I actually initially tried to do this within You.com and raise another round and so on, but you just can't. You have to do a certain thing, and until you print enough money that you're allowed to sort of start a second thing within that company is really hard. Uh, at the same time, I had all these ideas. I put them into a book and I finished a book last year, and I was like, "It'd be really fun to actually work, uh, on this myself." Uh, you know, I felt like with word vectors, uh, and then prompt engineering and, uh, ImageNet and large language models for protein generation, not, not folding and so on, I, me and my teams have put Sort of push the field truly forward, and I feel like we can do it again, uh, here at Recursive. And in many ways, what I observed over the last, uh, twenty years in AI is that whenever we replace some human part of the process of creating AI with a learned system, improvements follow. Uh, and so, you know, we've done that, taking out manual feature engineering, like in sentiment analysis. I don't know if you remember these old days where, like, there are linguists and they're like, "Here's how you negate, and there's a, like, regular expression."

    6. SW

      I went to Penn where we-- they had like the WordNet-

    7. RS

      That's right

    8. SW

      ... original thing.

    9. RS

      WordNet, all of that stuff. Yeah.

    10. SW

      They, they used, uh, our grad students to label Wall Street Journal articles and, like, really [chuckles] construct a knowledge graph of-

    11. RS

      There you go.

    12. SW

      Yeah.

    13. RS

      And WordNet started, you know, it was part of how we started ImageNet. But anyway, so like, uh, it was, it was really, like fun, um, to do, but when we replaced all of that manual feature engineering with vectors and neural nets and just backprop through everything, w- it actually started to work really well at scale. And so then everyone started to do architecture engineering, and I was like, "Ah, that clearly can't be it."

    14. SW

      You mean, uh, neural architecture search?

    15. RS

      No, like manually they would say like, "Oh, I'm, I'm doing sentiment analysis, so I have a special neural net that's really good at sentiment analysis." And then the machine translation community had a special neural net for machine translation.

    16. SW

      I see.

    17. RS

      And the summarization people had their own stuff, and I was like, "That clearly can't be it. We should unify all of that." So I had two papers. Uh, one was called Ask Me Anything, and the other one was called DecaNLP, and DecaNLP eventually got cited like five times by the first GPT paper. And, uh, to me, that was like a really big step forward. Um, and then of course, you had to combine this idea of prompt engineering with transformers and with language models, and you put it all together, you scale it up, which is also a huge amount of work, and then, uh, the field progressed a lot. I feel like the next step, and maybe the last step of that history and the sort of arguably, uh, sort of success has a lot of parents, only failure is an orphan, like my version of that AI history. I do feel like in that history, uh, you can kind of think about, well, what's the next way to automate? And that is the A-AI research itself, like the human, uh, process of ideating, implementing, and validating ideas.

    18. SW

      Mm.

    19. RS

      And in our case, ideas for AI.

  7. 21:20 – 24:12

    Recursive Self-Improvement and the Founding Team

    1. SW

      Mm.

    2. RS

      And when you have AI then help you with that, it, by almost definition, becomes a self-improving AI 'cause it now does research on itself. And there are lots of different misnomers. Some people think Auto Research is already recursive self-improvement. It's actually-

    3. SW

      Yeah, and you explained that in the talk

    4. RS

      ... likely to be very different.

    5. SW

      Yeah.

    6. RS

      Um, but, uh, to me, it's the most interesting thing that I could be doing, and I'm really excited with the co-founding team. What's interesting is we have, you know, we have eight co-founders in total, including myself, and so-

    7. SW

      Yeah, we're gonna bring it up. [chuckles]

    8. RS

      Nice. Yeah. And they're all-- I, I could talk about all of them if you want.

    9. SW

      Super stacked. [chuckles]

    10. RS

      Yeah. It's just an incredibly talented group of people. And we all kind of came to the same conclusion, but actually from very different directions. Like Josh Tobin, uh, is our CTO. He ran a, a bunch of different, uh, projects at OpenAI, like, uh, Codex and deep, uh, research, uh, agents and ChatGPT agents and so on. But before that, he also worked in robotics, and he saw sort of the smaller simulations, uh, and how it's gonna be really hard to scale that in full generality. And so that's-- that was his angle coming to recursive self-improvement. Uh, we have Jeff Clune, who's been working in like open-endedness for a long time, uh, together with Tim Rocktäschel. Tim Rocktäschel also built Genie 1, 2, and 3, which is like the most exciting and most sophisticated, I think, still world model, uh, anywhere. Um, and so they, they both came from this, uh, open, uh, endedness angle. Jeff also, I think, published one of the most exciting papers in recent years about recursive self-improvement called the Darwin Gödel machine. Uh, super interesting paper. Uh, if we could, uh, maybe pull it up really quick-

    11. SW

      Yeah

    12. RS

      ... it would be like super interesting to see 'cause you see-

    13. SW

      By the way, I love how many paper citations. You're, you're giving people a lot of homework, which I like.

    14. RS

      Love it.

    15. SW

      Yeah.

    16. RS

      Yeah. And so, and like Sanming Rockstar, we worked together actually at MetaMind and, uh, uh, and, and Salesforce Research together. Aleksiej Ozerovitskij invented the vision transformer on the most cited, uh, papers in computer vision. Tim Shi is like also unicorn, you know, founder. Yuan Dong, uh, led RL at, at Meta. So just like, yeah, really fun to work with them, and the next level of people are just incredibly strong too. So it's been, it's been a really fun ride so far. So the first figure, you actually see exactly these kinds of ideas, uh, that, uh, I think, um, yeah, inspired a lot of us and now more and more people, where you have this archive of different coding agents. They learn how to self-modify, evaluate, uh, and then create these phylogenetic trees, uh, of, of, yeah, different, different ideas.

    17. SW

      That's one foundation. So that, that Darwin Gödel is an influence.

    18. RS

      Mm-hmm.

    19. SW

      Open-endedness is an influence. Any other sort of trains, trains of thought that feeds into Recursive that I'm missing?

    20. RS

      Going to replace manual parts of the process of building AI-

    21. SW

      I-

    22. RS

      ... more and more-

    23. SW

      Yeah, yeah

    24. RS

      ... with learned systems.

    25. SW

      Yeah. Which a-and like s- it's merging different fields into, into one general, um, uh, architecture.

    26. RS

      That's right.

    27. SW

      Okay. It seems like language models are already pretty generalist, right?

  8. 24:12 – 30:20

    Are Today's LLMs Enough?

    1. RS

      Mm-hmm.

    2. SW

      Your, your next token predicting your reasoning. Was there, was there a time that you thought, "Okay, these are good enough to have recursive self-improving machines"?

    3. RS

      It was clear to me that they will happen, uh, within like a year or two, and then it did actually exactly happen like earlier this year, right? Earlier this year, AI really went from not just being code, but being able to code. Uh, and that is a big unlock. Uh, it's definitely making everything a lot easier than it was, uh, before the beginning of this year.

    4. SW

      One question that I think a lot of people have is, is the current LLM, LLM paradigm enough? Or like let's call it autoregressive transformer, you know, with reasoning, whatever. Don't you need something else, some, some big unlock, whether it's world models which Chris Manning's working on or memory continual learning, all that kind of stuff? Or is it all of the kinds and you think the current, let's call it transformer architecture, is here to stay, and that's it?

    5. RS

      A lot of thoughts. So number one, I do think it would be great to have less of a monoculture in AI research.

    6. SW

      Mm.

    7. RS

      Like if you look at, uh, AI conferences now, I still remember the days in like 2010 when I tried to get my first neural net papers and NLP conferences accepted, and they just desk rejected them-

    8. SW

      [chuckles]

    9. RS

      ... because, uh, like neural nets were something, quote-unquote, "we don't do in NLP conferences," like, and just like desk reject it. And it was very brutal in the first years of my PhD. Um, now I feel like it's almost like the field switched to the other side. Like-

    10. SW

      Yeah

    11. RS

      ... someone should try some other weird, crazy ideas now that aren't that obvious.

    12. SW

      There's also a few I really respect, like people still working on like GNNs and-

    13. RS

      Mm-hmm

    14. SW

      ... like tabular stuff and yeah, I don't know.

    15. RS

      Yeah, I mean, like someone, someone should still like do novel, novel out there ideas. At the same time, I think whenever people say, "Oh, LLMs are-- like this is the end for LLMs," they just don't, uh, like-- LLMs are also not the LLMs of like the past, right? Like they are so much more sophisticated now. There's so many more clever things that people are doing it. There are like different stages of training. Uh, you have the whole RL training, um, and you can take actions and like all of these things where, uh, that can go really far. And then the folks that come from the neurosymbolic, uh, direction say, "Oh, this will never work 'cause they can't do neurosymbolic reasoning." It's like, I think they're underestimating still the ability for these models to code, and code is neurosymbolic reasoning, and these models can obviously code incredibly well. And so I do think there are, of course, more and more ideas that will, will be needed and will continue to have. We're seeing like more and more interesting high-level ideas coming out of the AI itself too. Um, and with really deeply integrating the fact that these models are code and can code, that line, I don't wanna give it all away, but like I think that line has a lot more to grow. But it's still an LLM, right? Even if that LLM codes for you and then runs that code in some integrated fashion. World models, I'm personally less bullish on. Um, I think if you run a robotics company, you're gonna build your own world model. I think world models are super fun, and Tom Rocktäschel came to a similar conclusion after building the most interesting one-

    16. SW

      [laughs]

    17. RS

      ... with G one, two, and three, uh, which is gaming is a huge application for world models. Can see I sometimes got stuck in some games and you know, like got a little overly competitive in the wrong direction and so I, I understand games are fun, but personally I'd rather work on science than gaming. Um, and so yeah, I think LLMs, a lot more room to grow.

    18. SW

      Yeah. I, I think there's some interpretation of world models that some people have where it's like, well, it's okay, y-yes, there is that gaming element. There's just, there's the embodied robotics element. But actually the other part also is just, um, the, the more abstract sense of LLMs are just modeling output, but they're not modeling the chain of thought, uh, inside the human that has created the output. We can annotate it, of course, but like it's, it's always like this Plato's cave reflection of a thing rather than the thing, right?

    19. RS

      It's true.

    20. SW

      Yeah.

    21. RS

      But I would argue that, and maybe we'll get to in the ten, uh, spaces of intelligence, but I would argue that even our projection, our eyes is a projection of the real world and like we have only a very narrow, uh, band of the electromagnetic frequency spectrum that we can observe with our puny little two eyes and so on. Um-

    22. SW

      It's, it's good enough

    23. RS

      ... it's, it's good enough-

    24. SW

      Yeah

    25. RS

      ... for now, but like the, the upper-

    26. SW

      I mean-

    27. RS

      ... bounds of where it could be are so much higher and like to map, uh, the visual world the way humans see it is also not necessarily like the end all be all for visual intelligence. And I would argue that language is still the most interesting manifestation of human intelligence, and while our visual cortex is certainly less sophisticated, uh, than that of, uh, certain animals all the way down to the mantis shrimp who can, you know, have like two independent eyes, three bands, trinocular vision in each eye can see basically all the way to like floating temperatures in 4D and stuff.

    28. SW

      [chuckles]

    29. RS

      I mean, like mantis shrimp, you should look it up. It's like-

    30. SW

      Way OP. [chuckles]

  9. 30:20 – 35:55

    DecaNLP, GPT, and the Rejected Idea Ahead of Its Time

    1. SW

      Um, you mentioned GPT-1, uh, and I, I cannot let any, uh, Alec Radford, uh, you know, mention es-escape. Um, did you talk with him when he was training GPT-1? Like, any, any, any sort of historical fun stories there that, that you might come up?

    2. RS

      I, I did not like meet him a bunch of times. I think we met maybe once or twice at some conferences. Um, but, uh, like he, he has told, uh, I think Brian, the first author of, uh, the DecaNLP paper, that it did inspire him, and he cited it five times in the GPT-2 paper. So, um- Uh, and that's like-

    3. SW

      Yeah, good enough

    4. RS

      ... very clearly said, like, this was the first instantiation where they showed in the DecaNLP paper, McCann et al., that you can just phrase every single NLP problem as, "Here's some prompt, uh, text context, here's a question, uh, and task description, and here is some output." If you just do that enough, uh, you can have one unified neural network model, which by the way, also had all kinds of interesting attention mechanisms. There are slightly different formulations to the transformer, I think came out the same year, plus minus a few months. And then you can unify all of natural language processing into one neural net. Uh, that, that is sort of the core idea.

    5. SW

      And this was as opposed to, at the time, LSTMs and what have you.

    6. RS

      LSTMs, but also, like, people being very stuck in thinking about one model per task. In fact-

    7. SW

      Yeah

    8. RS

      ... uh, it's, it's kind of crazy, but the DecaNLP paper was publicly reviewed. It was like open, uh, open review. It was an ICLR submission, and, uh, in it you will see, uh, how the whole community at the time thought about this. Um, so like-

    9. SW

      Some great contributions, but more work needed.

    10. RS

      Yeah.

    11. SW

      [chuckles]

    12. RS

      So, so look at, like, search for not even for humans.

    13. SW

      Just scroll up. [laughs]

    14. RS

      Here. Like, question answering is not a unified phenomenon. There is no such thing as general question answering, not even for humans. And this is like, really you replace your brain with a different brain, a different neural net when you answer, like, different kinds of questions. It was unfathomable to the experts at the time that you can have one unified neural network that would answer all of these different questions. They say, "No, all of these questions require very different systems to answer, and trying to pretend they are the same doesn't help anyone solve any problems." That's what it says right there, right? That's how hard it was to fathom. And now of course, people when I say, "Oh, we invent problems," people are like, "You can't even invent problem genius. It's such an obvious idea [chuckles] to have one neural network that of course does everything in NLP."

    15. SW

      Mm-hmm.

    16. RS

      But at the time it was, like, extremely controversial, and the paper got rejected, and the sad thing is that it got rejected so hard-

    17. SW

      [laughs]

    18. RS

      ... and they were so certain that we stopped going on, on, on our list of things to try. And the number two or three on the list of extensions for this paper was add language modeling as another task. And then we could have like, you know, and that would've accelerated the timelines in 2018, like even further for humanity, but we got so crushed and we're like, "Okay-

    19. SW

      [laughs]

    20. RS

      ... maybe we'll just work on some of our other ideas for now and, like come back to this later."

    21. SW

      Mm-hmm. How can we design a review system that rewards non-consensus?

    22. RS

      You know, honestly, I, I started to feel like archive is, uh, is such a gift to humanity. Uh, and I think archive, just put your paper out there.

    23. SW

      Is it pre-preprints?

    24. RS

      Let-- And honestly, I think Twitter/X, people like you who pick up interesting papers, that is a better filter than the experts. Let, let everyone like r- like have, have access. Now, of course, there's some downsides, which is like, if you're super unfamous, you have no Twitter following-

    25. SW

      Yeah

    26. RS

      ... you don't wanna be on social media, whatever. You write a good paper, maybe someone, somehow no one notices it. But I would argue that if you just tell like 10 of your friends in your community about a paper and it is a really significant breakthrough, someone is bound to talk about it again. And, uh, so I think science needs less gatekeeping. And, uh, even though ICLR, uh, Yann LeCun, who started it, uh, as one of the co-founders of ICLR back in the day, he also wanted less gatekeeping 'cause he too was rejected for many years together with Yoshua and Jeff-

    27. SW

      Mm-hmm

    28. RS

      ... with all their early deep learning and neural net papers 'cause it was just not the hot thing. And so ICLR kind of started with that, but then it also started gatekeeping a little bit themselves on various ideas. So I think less gatekeeping, more open, uh, and then allowing people to say, "Look, even if this is just on or quote-unquote, just an archive, if it has like 1,000 citations, it's a legitimate paper. It doesn't really matter where you published it."

    29. SW

      And I agree with that. I, I do think it's kinda sad that I, I've heard that grad students have to do, um, like how to Twitter, uh, seminars to each other [chuckles]

    30. RS

      Yeah.

  10. 35:55 – 37:55

    Open-Endedness and Evolutionary AI

    1. SW

      I was gonna move on next to Tim and open-endedness. He, he was head of open-endedness at Google.

    2. RS

      That's right.

    3. SW

      I don't know what that means.

    4. RS

      [chuckles]

    5. SW

      Uh, but he did a lot, lot of talks.

    6. RS

      Genie-3 is one of the ways that-

    7. SW

      Yeah, right. Yeah, yeah

    8. RS

      ... rainbow teaming, yeah.

    9. SW

      So I, I, I first saw him at, at Speak of ICLR. First time at ICLR when he talked about open-endedness.

    10. RS

      Yeah.

    11. SW

      He's done, he's done a few talks. Can we define what is open-endedness for people who have never been exposed to the problem? They are like, "What do you mean? I thought the only goal of AI is to optimize against a benchmark or scale."

    12. RS

      That's right, yeah. It's a, it's a fuzzy, fuzzy term because there's so many different instantiations of open-ended, uh, thinking. But, uh, one way I often describe it and, and certainly, uh, Tim and Jeff Clune would be even better at describing this, but it's a suite of methods that is more inspired by evolution than, uh, very specific rewards. Uh, so in that sense, it, it thinks more about environments, about co-adaptation. And so an concrete example is in the cybersecurity and LM safety space where you have one LM that tries to attack another LM-

    13. SW

      You have the rainbow-

    14. RS

      ... to say something unsafe.

    15. SW

      Yeah.

    16. RS

      And now the environment is the two having a conversation, and now they co-adapting, right? They're like one makes a better attack, then the first one inoculates itself somehow, like uses that as training data, makes it so it's harder to say something unsafe based on that. And then as the attack stops working, the attacker now tries a different angle, right?

    17. SW

      Yeah.

    18. RS

      And that's why it's not just red teaming, but they're called sort of rainbow teaming.

    19. SW

      They're like, "Don't tell me how to do things. Let me just figure it out myself."

    20. RS

      That's right. Think about the environments that you wanna use. Think about the rewards at a high level that you wanna, uh, inspire towards. Uh, and then let the AI try out many more ideas in this interplay between sometimes humans, but also sometimes other AI agents.

    21. SW

      Yeah. I actually worked lo- um, open-endedness into a sort of model that I have been sort of working on. It, it was the keynote for AIE.

    22. RS

      Mm-hmm.

    23. SW

      Uh, where you start... Y- you know, we have the token loop, we have the agent turns, and then we have goal. And I feel like the way that you're describing Open-endedness is still

  11. 37:55 – 42:33

    What Happens When AI Chooses Its Own Goals?

    1. SW

      somewhat of a goal.

    2. RS

      Mm-hmm.

    3. SW

      Like, like please attack this-

    4. RS

      Right

    5. SW

      ... uh, other agent. But-

    6. RS

      Yeah, you set the rewards

    7. SW

      ... to me-

    8. RS

      You set the environments

    9. SW

      ... the, the loop that makes the other loops is what if the agent can set its own goals?

    10. RS

      Yeah.

    11. SW

      And is it-- is that open-endedness? Like, like you don't give it a goal, just like be a sentient being, and maybe sentient is a very loaded word.

    12. RS

      Right.

    13. SW

      But just set your own directions. What do you think you should do? [chuckles]

    14. RS

      I, I love this direction. I think this is one of the 10 spaces of intelligence, uh, uh, that I lump under metacognition and thinking about thought.

    15. SW

      Okay.

    16. RS

      And it's an interesting one. Whenever people say, "Oh, AI is like, this is, you know, it's gonna stop from here. It's not gonna get that much better," and blah, blah, blah, I'm like, there's so many different spaces of intelligence that we haven't even started exploring yet and hence have made very little progress on. And there, there is kind of an interesting, uh, connection to economics and, uh, capitalism. Like it doesn't make sense for a company to build and spend billions of dollars building a model that instead of following the rewards and objective functions you gave it, may come up with its own subjective functions and its own goals.

    17. SW

      Yeah.

    18. RS

      Right? Uh, and then imagine you're like, "Okay, I spent billions of dollars. Now go develop this new battery, uh, material for me and answer all my emails." And it's like, "Nah, I think it'd be more interesting to evaluate the molecular composition of the atmosphere, uh, on Jupiter."

    19. SW

      Yeah.

    20. RS

      And you're like, "That's not what I paid you billions of dollars for." Like, and so no one's working on that-

    21. SW

      Yeah

    22. RS

      ... for good reasons. And then also understandably, it could-

    23. SW

      It's not useful [chuckles]

    24. RS

      ... it's not, it's not useful, and it could get a little bit weird, right? What if the AI actually does start to really have thoughts on its own? Uh, and what if we don't like those thoughts, right? And so it, it requires a whole different way of thinking about it. I had a great conversation with, uh, a good friend of mine, Sam Gershman, who's a neuroscience professor at Harvard, and like we just jammed on this a little bit on like what are sort of the best meta goals. And you know, I do think like knowledge seeking is, uh, a really good one. I'm currently thinking also about like, uh, the ultimate measure and unit of intelligence broadly construed, and I finally have some... Still too early to share it. Um, it's not-- I haven't fully baked the thoughts yet.

    25. SW

      Like, like, like some replacement for IQ. And-

    26. RS

      IQ is such a terrible definition, right?

    27. SW

      And E-ELO.

    28. RS

      It makes no sense.

    29. SW

      Yeah.

    30. RS

      Yeah, ELOs are terrible too because it's always just like me versus others.

  12. 42:33 – 43:57

    Superintelligence for Science

    1. SW

      Good enough.

    2. RS

      And eventually, so our, our goal, um, really I, I don't talk about it that often because it is a few years out, but our goal is once you have a, a recursive self-improving superintelligence, you then want to apply it to the most important problems, and I think a lot of those are in science and technology and broadly constitutes of inventions and those inventions in, you know, physics to create better, cheaper energy with fission or fusion, in, in chemistry and to create better materials and better batteries and, uh, better solar cells and, and, and so on. In biology, there's so much like I think soon to be low hang, lower and lower hanging fruit because of AI, because of protein and generation, not just folding, but actually generating new proteins like we did in, in ProGen many years ago. Like so much positive impact we had if you take that superintelligence and you apply it to science.

    3. SW

      I do fundamentally believe that. There's a lot of approaches, though. You're not the only team trying and your lab trying.

    4. RS

      Sure.

    5. SW

      Um, you know, there's like a lot of-- especially the physical sciences as well.

    6. RS

      And that's good. Yeah. I do actually think that physic-- like the reason we are only doing it in a few years is that it's a little too early right now. Robotics is not quite there yet. The AI is not quite there yet. But I'm fairly confident in three to five years, all those constraints will be gone, and then applying to real physical robotics experiments and so on, like true robotic process automation-

    7. SW

      Yeah

    8. RS

      ... not in the traditional sort of RPA sense, but like actually having robots run experiments for you will be totally there.

    9. SW

      Yeah.

    10. RS

      It's gonna

  13. 43:57 – 46:24

    GPUs, Compute, and the Limits of AI Takeoff

    1. RS

      be great.

    2. SW

      Just to call back to something that you said early on about slow takeoff. You said that like while really the, the subs- the substrate, that is limiting factor is, let's call it chips, uh, and semiconductors and all these things, and you have raised funding for that and, and, uh, you are, you're investing a lot on that. But have you done the math on like- Is it even achievable? And like, what, what is the, uh, industry concentration needed in order to achieve, like, scale?

    3. RS

      I mean, right now we know that, like, roughly, like, uh, you know, a thousand GPUs cost quite a lot of money.

    4. SW

      Mm-hmm.

    5. RS

      Right? If you wanted, like, tens of thousands of GPUs, you're, you're talking billions and billions of dollars. If you say, like, one GB-- like, 300 is like, you could eventually create models that are, you know, on that substrate, like, are close and similar to human intelligence. Like, and you want, like, you know, thousands and thousands of, uh, AIs to think about really hard problems, uh, in a similar fashion to humanity, like, yeah, that, that's, you know, that's a lot of money. You do the math, it's like, a lot. We don't have that amount of money right now anywhere to, like, build that. Um, now, obviously things can get more efficient. You will have, I think, soon better algorithms that won't be, and better hardware that won't be as energy hungry, um, and so on. Our human brain does quite a lot of flops with much less energy. Um-

    6. SW

      Twenty watts?

    7. RS

      That's exactly right. Yeah. That's the number often is, is quoted. Um, and like, I think more, uh, inventions will happen there, uh, that then will accelerate, uh, the takeoff even further.

    8. SW

      One thing I always try to reconcile when talking, like, with NeoLab founders is, like, you're kind of fighting bitter lesson all the time. You, you have to show initial progress, then you unlock the next tier of funding, then the next tier, then the next tier.

    9. RS

      Which unlocks larger model categories.

    10. SW

      Like, fundamentally, is that true? Like, are you fighting bitter lesson? Are you-- Will the- will you have a s- way in which like, "No, we're changing the slope in some fundamentally different way"?

    11. RS

      I do think we are changing the slopes in, in fundamental ways by making AI much, much more efficient, both in terms of the training as well as the inference.

    12. SW

      Okay.

    13. RS

      Yeah. I think we will-- When you allow AI to do the work that it takes other labs thousands of people and years to do, I think we'll be able to get it down to weeks, and that will be much, much cheaper-

    14. SW

      Yeah

    15. RS

      ... um, and hence, you know, more affordable, accessible to others, and so on.

    16. SW

      Yeah, and you've shared initial results on, on that, uh-

    17. RS

      Yeah

    18. SW

      ... which is, like, conveniently OpenAI has also done to their [chuckles] to their 35.6, so we can talk about it now.

    19. RS

      Yeah. Uh, yeah, so these were-

    20. SW

      Le- let's recap what you've done.

  14. 46:24 – 50:31

    Recursive's Results: AI Beating Humans and Their Agents

    1. RS

      Yeah. So maybe, yeah, just a quick recap here. Uh, we built, uh, this, this, uh, system that isn't the full, even the full RSI system in its glory, but it is a first baby version of this. And then, you know, we don't wanna just have it internally and, and not show anything and, you know, just show some people of what's possible. Um, and so we basically applied this to these three different tasks. One is NanoChat, uh, by my friend Andrej Karpathy, um, uh, just like train a small language model to get, uh, really low bits per byte. And, you know, like hundreds if not thousands of people, uh, used both the agents and themselves to try, uh, to get to that, uh, and then they got to 0.937. We literally took our system and got to a much lower, uh, bits per byte, uh, much, much faster within, like, I think less than two days. Uh, so we took this thing, applied our system to it, and less than two days later, uh, we have-- we outperformed every human and their agents, uh, in, in, uh... have ever worked on this. Same with NanoGPT, and then we're like, well, let's, you know, apply it to something that's even more relevant, uh, to, to real people and to the NVIDIA ecosystem and applied it, uh, to, uh, Sol 1 exec bench, and maybe you can scroll down to some of the, uh, images. They're, they're kind of fun to see.

    2. SW

      Yeah.

    3. RS

      But yeah, you know, like, one you see it's actually made some real inventions that weren't just sort of hyperparameter tuning. Like, actually inventing hash tables and so on is, is quite clever. We have even better results now.

    4. SW

      What do you mean inventing hash t-- You didn't invent hash tables. [laughs]

    5. RS

      Of course, we didn't invent, like, hash tables in the grand scheme of, like, a hash table is like a super basic primitive in, in computer science. But to use it, uh, for language modeling in this scenario inside a transformer and so on, and to actually combine these ideas and put them together, that has then eventually also been invented, but there was a knowledge cutoff, and we did actually check that it didn't have access to that externally. Um, we talk about this a little bit. If you scroll to the next figures, you know, this is also an interesting one in that when you start from a really basic poor, like, tr- vanilla transformer, then we still outperform all of the community together. But if you start from the human seed of an expert like Andrej, then you get even lower. So the human seeds from which you start do, do still matter. Uh, so that was an interesting kind of insight, uh, in my eyes, uh, on this. And then as you go, like, you know, how long does it take to, to actually get to these models to get to similar performance? Uh, it's much faster. And then a similar thing happens with, uh, the speed runs here, where, you know, people have worked on this for, for quite some time, uh, and the model still was able to train a model more quickly. Why do we care about it? Well, speed of training is part of the equation of the cost, uh, and ultimately you wanna have the most intelligence per dollar, right? And so speed and quality are big parts of that. And, you know, the, the, um-

    6. SW

      Yeah, the way I put it is... For, for people who don't understand, they look at the chart, they're like, "Cool, what does it mean?" Uh, you know, if you have, like, a billion-dollar cluster and you can shave off ten percent, that's a hundred million dollars.

    7. RS

      That's exactly right.

    8. SW

      How much is that worth? [laughs]

    9. RS

      Exactly. So when you th- when you look at, like, the kernels, these kernels, yeah, for, for the non-experts, like, uh, these kernels are of- like, used in basically all the models. Every time you use an NVIDIA GPU, you interface with that GPU through these kernels. And so here you see, uh, the leaderboard best, uh, and when it's recursive, um, a- and basically, there are only a handful of kernels, uh, in this whole benchmark where we weren't the best. Um, and so to me, this is, like, really exciting, uh, 'cause it makes-- it, it just showcases what this can do. And again, these weren't, like... We didn't, like, spend months or years, like, developing. In fact, in particular for kernel, uh, uh, CUDA kernels, like, we don't even have really deep CUDA kernel experts in the team. And our system, that, that's sort of the beauty. The system just did all of these things. We didn't invent this. And when we open source and, and release, uh, uh, things in the future and models in the future, like, it won't-- they won't be the best in their, you know, category or class or whatever because we're so smart, but it's because it-- we built a smart AI that does

  15. 50:31 – 54:29

    Reward Engineering and Auto Research

    1. RS

      it for us.

    2. AF

      Do you have anything that you've learned from how to guide good auto-research? A lot of it also builds on human background, right? It's not just as simple as just-

    3. RS

      Mm

    4. AF

      ... "Hey, go optimize this."

    5. RS

      Right.

    6. AF

      Um, but we do see it again and again, right? Like some of the Erdos problems, frontier math is being solved by people, and when they do a writeup, they're like, "Oh, I'm not a mathematician. I have no background in this," you know?

    7. RS

      [laughs]

    8. AF

      "I, I saw some tools and I, I made it work."

    9. RS

      While you're watching the World Cup-

    10. AF

      Yeah

    11. RS

      ... you, like, disprove some conjectures-

    12. AF

      Yep

    13. RS

      ... that's going on, like [laughs]

    14. AF

      Um, any learning that you've had-

    15. RS

      Yeah, the circadian conjecture was like... Yeah, that's pretty cool. To summarize, uh, tips for good auto-research-

    16. AF

      Yeah

    17. RS

      ... versus bad auto-research.

    18. AF

      How did you build the recursive?

    19. RS

      Yeah. So without giving away all the, all the secret sauce, um, maybe some things that are probably obvious to the experts but might still be interesting to some, uh, folks is like reward engineering is one of the most crucial bits, uh, uh, especially, uh, in order to avoid reward hacking. Uh, so you have to be really clever about avoiding-- 'cause as, as you en- uh, AI gets better and better, it will get better and better at, at finding weird ca- like special cases or counter examples and, and things like that. And so I'll give you an example. Like, when you ask to, like, make these hundreds, uh, lines of code faster, and, you know, how do you define fast? Well, you have one line at the beginning that says start-

    20. AF

      [laughs]

    21. RS

      ... your stopwatch and one line at the end, end the stopwatch, and then, you know, tell us how much time, uh, uh, pro-progressed. And so, well, the simplest way is you just put that line that ends the stopwatch, right-

    22. AF

      At the start

    23. RS

      ... you know, at the start, and then boom, it's now faster, right? So this isn't like this, like, super evil AI. It's just, like, a very simple dumb reward hack. Um, and so you have to just very carefully think about all the different angles there. And then I think the m-longer time horizon the tasks are, the harder it gets and the more interesting and clever you have to be to still use these kinds of ideas for it. But yeah, I can't give away too much there.

    24. AF

      Yeah. Seems like rubrics are taking a good spot in that, where for unverifiable domains, you have rubrics. You have a model breakdown, judges' criteria along the way.

    25. RS

      It's a form of verification-

    26. AF

      Yeah

    27. RS

      ... once, once you got enough-

    28. AF

      Yeah

    29. RS

      ... good enough rubrics.

    30. AF

      I said this a long time ago. That's why I've never been that impressed that AI can play games, 'cause I'm like, obviously anything you can simulate and/or verify, you can have un-infinite training data-

  16. 54:29 – 59:24

    The AI Economist and Simulating Entire Economies

    1. RS

      [laughs]

    2. AF

      Okay.

    3. RS

      Um, so the idea is you have all these economic agents. They just wanna optimize their own utility function, which, uh, is, you know, collect resources that make money. Uh, and you can sell resources like wood, uh, and then, uh, over time, as you collect m- enough wood, you can build houses, you can trade with other agents, and you can basically use the houses then also to block off resources-

    4. AF

      Yeah

    5. RS

      ... from other agents.

    6. AF

      Yeah.

    7. RS

      So there's like-

    8. AF

      There's strategy

    9. RS

      ... competitive play and strategy-

    10. AF

      Yeah

    11. RS

      ... and so on. And the point was that we actually wanted to understand what is the best way of taxation and subsid-subsidization-

    12. AF

      [laughs]

    13. RS

      ... to optimize an economy. And this, this kind of research has not yet had its sort of GPT moment. But I believe that countries like Singapore and others should and will eventually use this to instead of doing like, uh, basically partisan politics and like special interest politics of like who donates the most to your campaign and stuff, you say, "Well, here I wanna help the middle class," or whatever you might say is your objective as a politician. And then people say, "Okay, well, how do you want to do that?" And it's like, "Well, here's my fiscal policy. Here's how I will change the taxes and pay these people," and so on. And then you can actually put that into a simulation, and you run that, that a-attempt from the politician against billions and billions of years of other strategies to try to achieve the goal that they set out to do.

    14. AF

      Yeah.

    15. RS

      And then you can say, "Well, if that was your actual goal, then here is, you know, billions of years of a strong simulation that would suggest that you try other ways of doing it, and maybe this t- the taxes and so on, and this, these tax brackets and so on." And this is how you avoid gaming 'cause these agents also try to reward hack to not pay their taxes and-

    16. AF

      Yes

    17. RS

      ... and so on. Uh, I thought this paper was super interesting. Unfortunately, similar to the first paper on, uh, prompt engineering The economists are like, "We don't know any of this math." It is just like-

    18. SW

      It's not even a-

    19. RS

      ... we are just forced to learn-

    20. SW

      It's not even a math. It's just we don't trust your simulation. It's not about math.

    21. RS

      It was-- I, I mean, they just desk rejected the thing. And it's like-

    22. SW

      Yeah, yeah, yeah [laughs]

    23. RS

      ... it's like they, like they didn't even give us like clear re- like clear sort of signals. Um, but like the world of economics unfortunately doesn't have-

    24. SW

      Oh my God [laughs]

    25. RS

      ... proper... Yeah, it doesn't have proper, uh, benchmarks. So you cannot be-- Like eventually, why did neural nets win? Not because people loved it. Like they had all kinds of beautiful integrals and graphical models and so, but it's just worked better.

    26. SW

      Yeah, yeah.

    27. RS

      But in economics it's hard to prove-

    28. SW

      So empiricism versus... Yeah, yeah, yeah. And I, I do have a bit of that econ background where like there's a lot of physics envy where you wanna write the general equation-

    29. RS

      Right

    30. SW

      ... for an economy, uh, versus just simulating it and, uh, using an evolutionary approach.

  17. 59:24 – 1:04:55

    LLM Simulations, Personas, and Mode Collapse

    1. SW

      So like someone else has to, has to take the charge there. I'm just gonna do one question on the simulation thing and then-

    2. RS

      Mm-hmm

    3. SW

      ... I don't know, we can probably move on.

    4. RS

      Mm-hmm.

    5. SW

      Uh, mode collapse, right? Like, you know, LLMs do not model the decision of humans. Uh, spamming it out 8 billion times is not gonna help you [laughs] model humanity. What do we do [laughs] ?

    6. RS

      I do think, uh, you have to be clever about prompting each one individually.

    7. SW

      Yeah.

    8. RS

      And, and I think that will help you kind of get stuck into different, different modes. In a weird way-

    9. SW

      It's like-

    10. RS

      ... people also get stuck in different modes, you know? Like-

    11. SW

      Yeah, yeah

    12. RS

      ... there's a lot of people like don't teach an old dog new tricks kind of thing. Like once people are stuck in their ways, the older they get, the harder it is for them to, to think new ways. And there's this, uh, I think, uh, comment, I forgot who said it, but it's like everything that was invented, uh, before you were born is natural. Everything that is invented when you're 20 is cool and everything that's invented after you're 60 is like unnatural and an abomination and kind of weird.

    13. SW

      Yeah.

    14. RS

      I feel like that's, you know, it's, it's true for a lot of people.

    15. SW

      Yeah.

    16. RS

      Like it is a fashion and, um, I think people will do it. Uh, Tencent had a billion personas paper that gives us good dataset for prompting, uh, simulations.

    17. SW

      Mm-hmm.

    18. RS

      If anyone's looking into this, um, on, on the podcast. They just had like, "You are a 30-year-old grocery store clerk. You are a 50-year-old professor."

    19. SW

      Yeah.

    20. RS

      And then just do a billion of those.

    21. SW

      Checks out.

    22. RS

      So then you just use it.

    23. SW

      [laughs]

    24. RS

      I'm, I'm kind of shocked how, how well a lot of these things actually do map to ul- ultimately similar statistics to real experiments. Yeah.

    25. AF

      Yeah. I think it's also good stuff for people to try that want to get into research, right? Like we've seen train a model only on data before a certain date and see how well it extrapolates out, do the same thing, right? So, um, see do people code more with better coding agents? Can a model that hasn't been trained on this figure that out without web access, right? Extrapolate out. Test these things.

    26. RS

      Yeah.

    27. SW

      Right. You know, just today, I think LM Arena published a interesting result where they basically were able to create a model now to predict your, your ranking.

    28. AF

      Wait, uh, based on what input?

    29. RS

      Your model. I guess you give it your model and it predicts the Elo score.

    30. AF

      I see. Okay. Sure.

  18. 1:04:55 – 1:10:30

    Recursive's Roadmap, Agents, Search, and Finance

    1. RS

      We, like very explicitly, will not start with any of the physical sciences-

    2. AF

      Okay

    3. RS

      ... uh, for now. We will start on AI for AI research. And so the AI for AI research has, I think, still a lot of room to grow. Uh, that's both in terms of making training more efficient and more automated, uh, as well as making inference more, more efficient and potentially local on your laptop. Like, and there are all kinds of interesting angles that have not been explored, uh, that well.

    4. SW

      What's the, um... Go deeper on the local stuff, because I always feel like it's the most inefficient form of AI training.

    5. RS

      Yeah. So just training and inference, I, I can't go into too many details.

    6. SW

      Okay.

    7. RS

      But like, yeah, I think there's just like so many angles, so many different compute substrates that, that have not yet been explored, either for training or for inference.

    8. SW

      Great. Uh, I don't know if you have any other comments on the, on the, the other stuff. I, I would say-

    9. RS

      No

    10. SW

      ... the other thing where, like there's the sort of inference in the s- uh, optimization in the small, but then also there is overall latency, latency end-to-end under conditions of load, uh, which is a, like a very different thing, which is the basically what they actually end up doing. That is a different domain of auto research-

    11. RS

      Mm-hmm

    12. SW

      ... than, uh, than I would say, like improving the kernels.

    13. RS

      Right.

    14. SW

      I think the other thing that I always think about in terms of automating or improving performance end-to-end is how the harness plays into it.

    15. RS

      Mm-hmm.

    16. AF

      Mm-hmm.

    17. SW

      So and p- particularly now, when we say harness, we also mean sandboxes, right?

    18. RS

      Right.

    19. SW

      I'm curious if that is a blocker for you or like, you know, like how, how the agent calls out to tools, basically.

    20. RS

      The number one tool all these agents use is web search, of course, uh, which makes sense. Um, and then I do think the harness is nice to optimize for because it's just so easy, right? It's just language. You look at it, it makes sense, and you can iterate. You don't have to train a massive model for like, you know, a lot of flops, uh, to get to the next state.

    21. SW

      Yeah.

    22. RS

      I'm also a big fan of harness optimization, yeah.

    23. SW

      Yeah, but sandboxing is fine for you.

    24. RS

      Sandboxing is also super important. Um, and then of course, like reward, like hacking and alignment, I, I think are, are super crucial.

    25. SW

      Okay. Um, just want to mention the web search. You happen to also be CEO of a web search company. [laughs]

    26. RS

      [laughs]

    27. SW

      Do you use You.com and-

    28. RS

      Of course

    29. SW

      ... do you use others? Like, uh, should, should the rest of us be using You for web search? I-- when I say You, it's like very funny. It's like you the person-

    30. RS

      Right

  19. 1:10:30 – 1:31:38

    The Upper Bounds and Spaces of Intelligence

    1. RS

      Yeah.

    2. SW

      Yeah. Amazing. You know, as a, as a writer, as a thinker on, on, on these things, I love MECE categorizations. MECE is mutually exclusive, commonly exhaustive, something like that.

    3. RS

      [laughs]

    4. SW

      And so if this is a ME-MECE list of intelligence-

    5. RS

      It is not

    6. SW

      ... it is very-- okay, well-

    7. RS

      [laughs] Sorry.

    8. SW

      Yeah.

    9. RS

      There are all kinds of overlapping.

    10. SW

      Yeah.

    11. RS

      In fact, uh, if you want that kind of list, I think the three principal components of intelligence, uh, are prediction, which is mathematically, uh, quite, uh, similar to compression. Prediction multiplied with actions multiplied with goals. Those are the three principal components. I think all of these ten spaces are combinations of those three-

    12. SW

      Okay

    13. RS

      ... in specific dimensions, if you will. Uh, the reason I call them spaces is that each space has many sub-dimensions, and what I try to do, uh, actually this is just a, a, a side quest almost, uh, to the initial goal, which is to think about the upper bounds of intelligence. And, you know, everyone's like, "Oh, it's exponential." And it's like, well, exponentials at some point have to flatten out, but where do they flatten out when it comes to intelligence? And that led me on this whole... Like it would-- initially it started as a tweet, and then it was like a blog post, and now I'm like at fifty pages, and I'm still not nowhere near-

    14. SW

      It's your second book.

    15. RS

      It, it's a second book, basically. Uh, and so the la-- in my first book, Eureka Machine, I just kind of allude to these ten, uh, at the end, and I'll just to give you a sense. Like visual intelligence is sort of the easiest one to talk about and, uh, fleshed out the most already for me in my head. And so human intelligence has basically binocular vision. I mean, we have two eyes. We have a very narrow band of the electromagnetic frequency spectrum that we can really observe directly ourselves. And so when you think about the upper bounds of a visual intelligence, one you should go into, like you can have like millions and billions of sensors. At some point, you get to problems of how far are these sensors away from each other, such that the speed of light to communicate the content from all of them cannot, like get to a central brain to actually process-

    16. SW

      [laughs]

    17. RS

      ... uh, the visual intelligence, right?

    18. SW

      It's-- yeah.

    19. RS

      And so now you're thinking in along the dimension, in the space of vision intelli-- the, the dimension of numbers of sensors.

    20. SW

      Okay.

    21. RS

      The, the upper bounds are quite literally and figuratively astronomical, and we are super far away from any intelligence that would have this many of number of sensors. But then you go in the next dimension, which is the frequency. You're gonna all the way down to gamma rays, and you can start to try to observe, and you get into basically the upper bounds or I guess in this case, lower bounds, uh, uh, or upper bounds in terms of frequency, is basically quantum uncertainty. Like you just cannot observe certain particles anymore.

    22. SW

      Or you destroy it. Yeah.

    23. RS

      And now imagine you had millions of sensors that can see all the way down to the, like subatomic level as far as physics will allow us to, and then all the way down to seeing like gravitational waves. And now you have millions of those sensors. So that's another dimension is sort of the frequency. And then yet another dimension is like how many categories of things could you memorize and classify differently? We know now for humans, right? There are certain things if you have more terms for it, you'll have a better visual description, uh, for them. And like, you know, animals that don't have, like gorillas maybe have like two hundred words to assign to certain things, mostly visual things. And so human perception is quite special in that sense in terms of classifying all these different physical objects. So these are just like a very simple example. If you go, uh, to knowledge, right? Then it's also like the speed of light cone around all these sensors, and so they're all connected. Like knowledge is connected to visual intelligence if you think also not just visual, but sort of perception intelligence, just like, 'cause it doesn't have to be just what we can see. It can be, again, wider range of electromagnetic frequencies. Then you have language intelligence, which actually recently changed to more communication intelligence 'cause it's more like language has all these different anthropic bounds. Humans can only comprehend and know so many terms in our long-term memory, right? Our vocabularies are somewhat restricted, and the a-active ones are often even smaller than the passive vocabularies of things you can understand. Then, uh, language is ridiculously inefficient when it comes to trans-- basically communicating different types of information and, uh, transporting sort of different bits. Like human language is serial. Obviously- Another bound on, uh, communication intelligence would be to communicate in parallel. But neither will our tongues and mouths work to have multiple, like, streams in parallel, neither can we understand some women slightly better at, like, multitasking than some men.

    24. SW

      [laughs]

    25. RS

      Um, but, like, most people can only listen to one conversation and truly understand it.

    26. SW

      [laughs]

    27. RS

      There's no way that, like, like, in terms of communication intelligence, a, a true upper bound is one in terms of how many, like, knowledge, how many sequences of communication could you-

    28. SW

      Mm-hmm

    29. RS

      ... in parallel sort of, uh, process, right? Then, of course, you have, uh, like, how long are sentences. We only have so much in our working memory, and hence lang- human language has these fairly simple sentences with maybe 40 words or so-

    30. SW

      Mm-hmm

  20. 1:31:38 – 1:33:26

    Goals, High Agency, and Advice for Builders

    1. SW

      and all these things. Uh, I guess maybe one parting thought that you can give to people, uh, you know, like one form of intelligence is goals, as you mentioned. Uh, what do you want people's goals to be? Like how, how do they aspire to better things?

    2. RS

      If you wanna improve your goal intelligence, uh, in the current definition that I'm thinking about it, uh, it is often about how much can you... Oh, how far do I go? Uh, this is like a lot of entropy and free energy and stuff I'm currently thinking about.

    3. SW

      Oh, really? Okay.

    4. RS

      But it might be too, it might be too far, uh, out there for, for people to be like immediately actionable. [laughs]

    5. SW

      [laughs]

    6. RS

      So I think like, you know, if I actually gave real advice to real people, I'd be like, "Get a good education, think about AI, think about how you get high agency," and so on. But it's different to like in the grand scheme of things, how can you harness a lot of energy and transform, uh, you know, entropy into interesting states and so on.

    7. SW

      Yeah.

    8. RS

      So there's, uh, there, there are different levels o- of abstractions, uh, that we can, uh, think about here. But my, my advice for people, uh, like just sort of more down to earth is think about something you're passionate about, if you're studying, for instance, and then see how you combine that with AI. I think the more and more you have a true passion about a change you wanna see in the world, uh, the more you wanna connect that to AI in order to amplify your ability, uh, to get there.

    9. SW

      Yeah. I think that's a reasonable, uh, first step. I, I do think, I do think our listeners operate on multiple abstractions as well. One thing I did get from Ajni, uh, Mita was also like, yeah, just use anything that is very GPU heavy, and like that will guide you towards the right thing.

    10. RS

      [laughs]

    11. SW

      Which is like, yes, it, it is more compute heavy and, and therefore it will be probably more worth it. [laughs] Um, so well, thank you so much. Yeah. I think that was a really-

    12. RS

      Thank you

    13. SW

      ... uh, great discussion.

    14. RS

      Yeah, super fun. Appreciate it. Thanks for listening. [upbeat music]

Episode duration: 1:33:26

In the uListen app, the transcript lights up word by word as you listen. Get uListen

Transcript of episode eDFXtSg3zB8

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.

iOSAndroidClaudeChrome