Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment

Carl Shulman 0:00 Human-level AI is deep, deep into an intelligence explosion. Things like inventing the transformer, or discovering chinchilla scaling and doing your training runs more optimally, or creating flash attention. That set of inputs probably would yield the kind of AI capabilities needed for intelligence explosion. (air whooshing) We have a race between, on the one hand, the project of getting strong interpretability and shaping motivations, and on the other hand, these AIs, in ways that you don't perceive, make the AI takeover happen. (air whooshing) We spend more compute by having a larger brain than other animals, and then we have a longer childhood. It's analogous to, like, having a bigger model and having more training time with it. (air whooshing) It seemed very implausible that we couldn't do better than completely brute force evolution.

Carl Shulman 0:47 How quickly are we running through those orders of magnitude?

Dwarkesh Patel 0:50 Okay, today I have the pleasure of speaking with Carl Shulman. Many of my former guests, and this is not an exaggeration, many of my former guests have told me that a lot of their biggest ideas, perhaps most of their biggest ideas, have come directly from Carl. Especially when it has to do with the intelligence explosion and its impacts. And so I decided to go directly to the source, and we have Carl today on the podcast. Carl keeps a super low profile, but he is one of the most interesting intellectuals I've ever encountered, and this is actually his second podcast ever. So we're gonna get to get deep into the heart of many of the most important ideas that are circulating right now, uh, directly from the source. So, and b- by the way, so Carl is also an advisor to The Open Philanthropy Project, which is one of the biggest funders on causes having to do with AI and its risks, not to mention global health and well-being, and he is a research associate at the Future of Humanity Institute at Oxford. So Carl, it's a huge pleasure to have you on the podcast. Thanks for coming.

Carl Shulman 1:47 Thank you, Dhurakhar. Uh, I've enjoyed seeing, uh, some of your episodes recently, and I'm, uh, glad to be on the show.

Dwarkesh Patel 1:54 Excellent. Let's talk about AI. Before we get into the details, give me the sort of big picture explanation of the, uh, feedback loops and just the general, uh, dynamics that would start when you have something that is approaching human-level intelligence.

Carl Shulman 2:10 Yeah, so I think the- the way to think about it is we have a process now where humans are developing new computer chips, new software, um, running larger training runs, and takes a lot of work, uh, to keep Moore's Law, uh, chugging. Well, it was. It's slowing down now. Um, and it takes a lot of work to develop things like transformers, um, to develop, uh, a lot of the improvements to AI and neural networks, uh, that are advancing things. And the core method that I think I want to highlight, um, on this podcast, uh, and I think is underappreciated, uh, is the idea of input-output curves. So we can- we can look at the increasing difficulty of improving chips, uh, and s- so sure, each time you double the performance of computers it's harder. And as we approach physical limits, eventually it becomes impossible. But how much harder? Uh, so there's a- a paper, uh, called, uh, Ideas Getting Harder to Find, uh, that was published a few years ago, um, from the, like, 10 years ago, uh, at MIRI, we did, uh... I mean, I- I did, uh, an early version of this, uh, of this analysis, uh, using mainly, um, data from Intel and, like, the large semiconductor fabricators. Uh, anyway, and so in- in- in this paper, uh, they cover a period where the productivity of computing went up a million-fold. So you could get a million times the computing operations per second per dollar. Big change. But it got harder. So the- the amount of investment, the labor force, uh, required to make those continuing advancements went up and up and up. Uh, indeed it went up 18-fold over that period. And now- so some take this to say, "Oh, diminishing returns. Things are just getting harder and harder, and so it'll be the end of progress eventually." However, in a world where AI is doing the work, that doubling of computing performance translates pretty directly to a doubling or better of the effective labor supply.

Dwarkesh Patel 4:31 Mm.

Carl Shulman 4:32 That is, if when we had that million-fold compute increase, we used it to run artificial intelligences, uh, who would replace human scientists and engineers, then A ) the 18X increase in the labor demands of the industry would be trivial. We- we're getting more than one doubling of the effective labor supply that we need for each doubling of the labor requirement. A- and in that data set, it's, like, over four. So we double- we double compute. Okay, now we need somewhat more researchers. But a lot less than twice as many. And so okay, we- we use up some of those doublings of compute on the increasing difficulty-

Dwarkesh Patel 5:22 Mm.

Carl Shulman 5:22 ... of further research. But most of them are left to expedite the process. So if you- you double your labor force-

Dwarkesh Patel 5:31 Yeah.

Carl Shulman 5:31 ... that's enough to get several doublings of compute. You u- you use up one of them on meeting the increased demands from diminishing returns.

Dwarkesh Patel 5:40 Yep.

Carl Shulman 5:41 The others-

Dwarkesh Patel 5:42 Yep.

Carl Shulman 5:43 ... can be used to accelerate the process. So you have your- your first doubling takes however many months. Your next doubling can take-

Dwarkesh Patel 5:51 Mm.

Carl Shulman 5:52 ... a smaller fraction of that. The next doubling, less, and so on. At least in so far as this, the outputs you're generating compute for AI in this story-... are able to serve the function of the necessary inputs. If there are other inputs that you need, eventually those become a bottleneck, uh, and you wind up more restricted on those.

Dwarkesh Patel 6:15 Got it, okay. So yeah, I think, I think the Bloom paper had that 30, there was 35% increase in, was it transfer density or cost per flop? And there was a 7% increase per year in the number of researchers required to sustain that pace.

Carl Shulman 6:29 Something in that vicinity, yeah. It's like-

Dwarkesh Patel 6:31 Yeah.

Carl Shulman 6:31 ... four, four to five, uh, doublings of compute per doubling of labor inputs.

Dwarkesh Patel 6:38 I guess there's a lot of questions we can delve into in terms of whether you would expect a similar scale with AI, and whether it makes sense to think of AI as a population of researchers that keeps growing with compute itself. Actually, let's go there. So can you explain the intuition that compute is a good proxy for the number of AI researchers, so to speak?

Carl Shulman 6:59 Uh, so far talked about hardware-

Dwarkesh Patel 7:01 Yeah.

Carl Shulman 7:01 ... as an initial example because we had good data-

Dwarkesh Patel 7:03 Yep.

Carl Shulman 7:04 ... um, about a past period. Uh, you can also make im-, uh, improvements on the software side. Uh, when we think about an intelligence explosion, that can include AIs doing work on making hardware better, making better software, making more hardware. Um, but the basic, uh, idea for the hardware is, is especially simple, uh, in that if you have a worker, an AI worker that can substitute for a human, if you have twice as many computers, you can run two separate instances of them.

Dwarkesh Patel 7:36 Mm-hmm.

Carl Shulman 7:36 Uh, and then they can do two different jobs, manage, uh, two different, uh, machines, work on, uh, two different design problems. Uh, now you can get more gains than just what you would get by having two instances. We get improvements from using some of our compute, not just to run more instances of the existing AI, but to train larger AIs. So there's hardware technology, how much you can get per dollar you spend on hardware.

Dwarkesh Patel 8:05 Yeah.

Carl Shulman 8:05 And there's software technology and the software can be copied freely. Uh, so if you've, if you've got the software, it doesn't necessarily make that much sense to say that, oh, we've got, you know, 100 Microsoft Windows. Uh, you can make as many copies as you need of them for whatever (laughs) Microsoft will charge you. Uh, but for hardware it's different. Um, it matters how much we actually spend on the hardware at a given price. Uh, and if we look at the changes that have been driving AI recently, that is the thing that is really off-trend. We are spending tremendously more money on computer hardware for training big AI models.

Dwarkesh Patel 8:48 Yep, okay. So there's, there's the investment in hardware, there's the hardware technology itself, and then there's the software progress itself. The AI is getting better because we're spending more money on it, because our hardware itself is getting better over time, and because we're, like, developing better models or better adjustments to those models. Where is the loop here?

Carl Shulman 9:08 The work involved in designing new hardware and software is being done by people now.

Dwarkesh Patel 9:14 Mm-hmm.

Carl Shulman 9:15 Uh, they use computer tools to assist them, but, like, computer time is not, like, the primary cost, um, for NVIDIA designing chips, um, for TSMC producing them, for ASML, uh, making lithography equipment, uh, to serve the TSMC fabs. Uh, and even in AI software research, that has become quite compute-intensive. Uh, but I think we're still, uh, in the range where, you know, at, at a place like DeepMind, salaries were still, uh, larger than compute, uh, for the experiments. Although tremendously, tremendously more of the expenditures were on compute, uh, relative to salaries than in the past. If you take all of the work that's being done, uh, by those humans, uh, there's like low tens of thousands of people working at NVIDIA designing GPUs specialized for AI. Um, I think there's more like 70,000 people, uh, at TSMC, uh, which is the leading, uh, producer of cutting edge chips. There's a lot of additional people, uh, at companies like ASML that supply them, uh, with the tools they need. And then a company like DeepMind, I think, uh, from their public, uh, filings, uh, they recently had 1,000 people. OpenAI, I think, is a few hundred people. Um, Anthropic is less. If you add up things like Facebook AI Research, Google Brain, other R&D, you get thousands or tens of thousands of people, uh, who are working on AI research. Uh, we'd want to zoom in on those who are developing new methods-

Dwarkesh Patel 10:56 Yeah.

Carl Shulman 10:56 ... rather than narrow applications. So inventing the transformer-

Dwarkesh Patel 10:59 Yeah.

Carl Shulman 11:00 ... definitely counts. Optimizing for some particular business's dataset cleaning, probably not.

Dwarkesh Patel 11:06 Yeah.

Carl Shulman 11:07 Um, but so those people are doing this work, they're driving quite a lot of progress. What we observe in the, the growth of people relative to the growth of those capabilities, uh, is that pretty consistently, the capabilities are doubling on a shorter timescale than the people-

Dwarkesh Patel 11:27 Mm-hmm.

Carl Shulman 11:27 ... required to do them are doubling. And so there's work... So I, we, we talked about hardware and how historically it was pretty dramatic, like, uh, four or five doublings of compute efficiency per doubling of, of human inputs. Uh, I think that's a bit lower now as we get towards the end of Moore's Law. Although interestingly, not as much lower as you might think, because the growth of inputs has also slowed recently. On the software side, there's some work, uh, by, um, Tamam Besiroglu, um, and I think collaborators, um, they, uh, it may have been a the-thesis, uh-... it's called, uh, Are Models Getting Harder to Find? and so it's applying the same sort of analysis as the "Are Ideas-

Dwarkesh Patel 12:11 Yeah.

Carl Shulman 12:11 ... Getting Harder to Find?" Uh, and you can look at growth rates of, um, papers from, uh, citations, employment at these companies, and it seems like the doubling time of these, like, workers driving the software advances, uh, is, like, several years. Um, or at least a couple years. Whereas the doubling of effective compute from algorithmic progress is faster. So there's a group called Epoch, they rec- they received grants from Open Philanthropy, uh, and they do work collecting datasets, uh, that are relevant to forecasting AI progress. And so their headline results, uh, for what's the rate of progress in hardware, in software, um, and just, like, growth in budgets are as follows. So for hardware, they're looking at, like, a doubling of hardware efficiency that's, like, two years. It's possible it's a bit better than that when you take into account certain specializations for AI workloads. For the growth of budgets, uh, they find a doubling time that's, like, something like six months in recent years, which is pretty tremendous relative to, uh, the historical rates. We should maybe get into that later. Uh, and then on the algorithmic progress side, mainly using ImageNet type datasets right now, they find a doubling time that's less than one year. And so (laughs) you, you combine all of these things, uh, and the, like, the growth of effective compute for training big, big AIs, uh, it's, it's pretty, pretty drastic.

Dwarkesh Patel 13:47 I s- think I saw an estimate that GPT-4 costs, like, $50 million, around that range, to train. Now suppose that, like, AGI takes 1,000x that, um, if you were to just scale up GPT-4, um... It might not be that, I'm ju- just for the sake of example. Some o- so part of that will come from companies just spending a lot more to train the models, and that just greater investment. Part of that will come from them having better models, so that what would have taken a 10X increase in the model to get, naively, you can do with having a better model, that you only need to do scale up. You, you get the same effect of gr- increasing it by 10X just from having a better model. And so yeah, you're, you can spend more money on it to turn in a bigger model, you can just have a better model, or you can y- have chips that are cheaper to train, so you get more compute for the same dollars. And okay, so th- those are the three you're describing, th- the ways in which the, quote-unquote, "effect of a compute" would increase.

Carl Shulman 14:40 Yeah. From the, looking at it right now, it looks like, yeah, you might get two or three doublings of effective compute for this thing that we're calling software progress, um, which is, which people get by asking, "Well, how much less compute can you use now to achieve the same benchmark as you achieved before?" There are reasons to not fully identify this with, like, software progress, as you might naively, naively think of it, uh, because some of it can be enabled by the other. So like, when you have a lot of compute, you can do more experiments-

Dwarkesh Patel 15:11 Mm.

Carl Shulman 15:11 ... uh, and find algorithms that work better. Um, sometimes the, the additional compute, you can get higher efficiency by running a bigger model, we were talking about earlier.

Dwarkesh Patel 15:21 Yeah.

Carl Shulman 15:22 And so that means you're, you're getting more for each GPU that you have because you made this, like, larger expenditure, and you could... That m- can look like a, a software improvement because this model, it's, it's not a ha- it's not a hardware improvement directly because it's doing more with the same hardware. But you wouldn't have been able to achieve it without having a ton of GPUs to do the big training run.

Dwarkesh Patel 15:44 The, the feedback loop itself, i- involves the AI that is the result of this greater effect of compute helping you train better AI, right? Or use, uh, less effective compute in the future to train better AI.

Carl Shulman 15:58 It, it can help on the hardware design.

Dwarkesh Patel 15:59 Yeah.

Carl Shulman 16:00 So like, NVIDIA is a fabless chip design company. They don't make their own chips.

Dwarkesh Patel 16:04 Mm.

Carl Shulman 16:05 They send files of instructions to TSMC, uh, which then fabricates-

Dwarkesh Patel 16:10 Yeah.

Carl Shulman 16:10 ... the, the chips, uh, in their own facilities. Uh, and so the work of those, uh, you know, 10,000 plus people, if you could automate that, uh, and have the equivalent of a million people doing that work, then I think you would pretty quickly, um, get the kind of improvements that can be achieved with the existing, uh, nodes, uh, that TSMC is operating on. You could get a lot of those chip design gains basically, like, doing the job of improving chip design that those people are working on now, but get it done faster.

Dwarkesh Patel 16:45 Mm.

Carl Shulman 16:46 So that's one thing. I think that's less important for-

Dwarkesh Patel 16:48 Right.

Carl Shulman 16:48 ... the intelligence explosion. Um, the reason being that when you make an improvement to chip design, it only applies to the chips you make after that. If you make an improvement in AI software, it has the potential to be immediately applied to, to all of the GPUs that you already have.

Dwarkesh Patel 17:08 Mm-hmm.

Carl Shulman 17:08 Yeah, and so the thing that I think is most disruptive and most important, has the leading edge of the change from AI automation

Carl Shulman 17:18 of the inputs to AI is on the software side.

Dwarkesh Patel 17:21 Where, uh, wha- at what point would it get to the point where the AIs are helping develop better software or better models for future AIs? Some people claim today, for example, that, you know, programmers at OpenAI are using Copilot to write programs now. So in some sense, you're already having that sort of feedback loop, that I'm a little skeptical of that, um, as a mechanism. At what point would it be the case that the AI is contributing significantly in the sense that would almost be the equivalent of having additional researchers to AI progress in software?

Carl Shulman 17:53 The, the quantitative magnitude of the help is absolutely central.

Dwarkesh Patel 17:57 Yeah.

Carl Shulman 17:57 So there, there are plenty of companies that, like, make some product that, like, very slightly boosts productivity.

Dwarkesh Patel 18:03 Yeah.

Carl Shulman 18:03 So when Xerox makes fax machines, it maybe increases people's productivity, uh, in office work by 0.1% or something.

Dwarkesh Patel 18:12 Yeah.

Carl Shulman 18:12 You're not gonna have explosive growth-

Dwarkesh Patel 18:15 Yeah.

Carl Shulman 18:15 ... out of that because-Okay. Now, "0.1% more, uh, you know, effective R&D, uh, at Xerox than any customers buying-"

Dwarkesh Patel 18:24 Yeah.

Carl Shulman 18:24 ... "the machines." Eh, not, not that important. So I think the, the thing, the thing to look for, uh, is when is it the case that the, the contributions from AI are starting to, uh, become as large or larger as the contributions, uh, from humans? So like, uh, when this is boosting their effective productivity by 50 or 100%-

Dwarkesh Patel 18:49 Yeah.

Carl Shulman 18:50 ... and you, like, if you then go from, you know, eight months doubling time, say, for effective compute from software innovations, things like inventing the transformer or discovering chinchilla scaling-

Dwarkesh Patel 19:02 Yeah.

Carl Shulman 19:02 ... and doing your training runs more optimally or creating flash attention. Yeah, if you move that from, say, eight months to four months-

Dwarkesh Patel 19:09 Yeah.

Carl Shulman 19:10 ... and then the next time you apply that, it significantly increases the boost you're getting-

Dwarkesh Patel 19:16 Mm-hmm.

Carl Shulman 19:16 ... uh, from the AI. So now, maybe instead of giving a 50% or 100% productivity boost, now it's more like a 200%.

Dwarkesh Patel 19:21 Yeah.

Carl Shulman 19:22 Um, and so it doesn't have to have been able to automate everything involved in the process of AI research, it can be it's automated a bunch of things and then those are being done in extreme profusion. Because any, uh, a thing, a thing that A- AI can do, you have it done much more often because it's so cheap. Uh, and so it's not a threshold of, "This is human level AI, it can do everything a human can do with no weaknesses in any area." It's that even with its weaknesses, it's able to bump up the performance so that instead, instead of getting, like, the results we would have with, say, the 10,000 people working on finding these innovations, we get the results that we would have if we had twice as many of those people with the same kind of skill distribution. And so that's a, it's like a demanding challenge.

Dwarkesh Patel 20:15 Yeah.

Carl Shulman 20:15 It's like, uh, you need quite a, quite a lot of capability for that. But it's also important that it's significantly less than this is a system where there's no way you can point at it and say in any respect it is weaker than a human. A system that was just as good as a human in every respect but also had all of the advantages of an AI, that is just way beyond this point. Like, if you consider that there's, like, the, the output of our existing fabs make tens of millions of advanced GPUs per year. Those GPUs, if they were running sort of AI software that was as efficient as humans, is sample efficient, it doesn't have any major weaknesses. So they can work four times as long, uh, you know, the 168-hour work week, they can have much more education than any human. So it's, you know, yeah, a human, you know, they got a PhD. You know, it's like, uh, wow, it's like 20 years of education, uh, maybe longer if they, they take a, if they take a slow, uh, slow route on the PhD. Um, it's just normal for us to train large models by eat the internet, eat all the published books ever, um, read everything on GitHub, uh, and get good, uh, at predicting it. Um, so, like, the level of education vastly beyond any human. The degree to which the models are focused on task, uh, is higher than all but, like, the most motivated humans when they're really, really gunning for it. Uh, so you combine the things, tens of millions of GPUs, each GPU, uh, is doing the work of the very best humans in the world. And, like, the most capable humans in the world, uh, can command salaries that are a lot higher than the average. And particular in a field like STEM or narrowly AI. Like, there's no human in the world who has a thousand years of experience with TensorFlow or, or let alone the new AI technologies that were invented the year-

Dwarkesh Patel 22:20 Yeah.

Carl Shulman 22:20 ... the year before. But if, if they were around, uh, yeah, they'd, they'd be paid millions of dollars a year. Uh, and so when you consider this, okay, tens of millions of GPUs, each is doing the work of maybe 40, maybe more, uh, of these kind of existing workers. This is like going from a workforce of tens of thousands to hundreds of millions. You immediately make all kinds of discoveries then. You immediately develop all sorts of tremendous technologies. So human level AI is deep, deep into an intelligence explosion.

Dwarkesh Patel 22:59 Mm. Yeah.

Carl Shulman 22:59 Intelligence explosion has to start-

Dwarkesh Patel 23:01 Yeah.

Carl Shulman 23:01 ... with something weaker than that.

Dwarkesh Patel 23:02 Yup, yup, yup. Yeah. What, what is the thing it starts with and how close are we to that? Because if you think of a researcher at OpenAI or something, you know, these are... To be a researcher is not just completing the Hello World, uh, prompt that Copilot does, right? It's like, you had to choose a new idea, you had to figure out the right way to approach it, you perhaps have to manage the people who are also working with you on that problem. It, you know, it's like, it's, uh, it's incredibly complicated skill, portfolio of skills rather than just a single skill. So yeah. What, what, what, what is the point at which that feedback loop starts where you can even... You're not just doing the 0.5% increase in productivity that a sort of AI tool might do, but is actually the equivalent of a researcher or close to it? Like, what, what is that point?

Carl Shulman 23:47 So I think may- maybe a way, uh, to look at it is to give some illustrative examples-

Dwarkesh Patel 23:51 Yeah.

Carl Shulman 23:51 ... of, like, the kinds of capabilities that you might see. And so because these systems have to be a lot weaker-

Dwarkesh Patel 23:59 Yeah.

Carl Shulman 23:59 ... than the sort of human level things, what we'll, we'll have is intense application of the ways in which AIs have advantages partly offsetting their weaknesses. And so AIs are cheap, we can call a lot of them, uh, to do many small problems. Uh, and so you'll have situations where you have dumber AIs that are deployed thousands of times-... uh, to equal, say, one human worker. Uh, and they'll be doing things like, um, these voting algorithms where you have an LLM, you generate a bunch of different responses, uh, and take a majority vote among them that improves performance sum. Uh, you'll have things like the AlphaGo kind of approach, um, where you use the neural net to do search, uh, and you go deeper with the search by plowing in more compute, which helps to offset the inefficiency and weaknesses of the model on its own. Uh, you'll do a thing that would just be totally impractical, um, for humans because of the sheer number of steps. And so an example of that would be designing synthetic training data. Uh, so humans do not learn by just going into the library and opening books at random pages. Um, it's actually much, much more efficient, uh, to have things like schools and classes, uh, where they teach you things in a, an order that makes sense, that's focusing on the skills that are more-

Dwarkesh Patel 25:31 Yeah.

Carl Shulman 25:31 ... valuable to learn. Uh, they give you tests and exams that are designed to try and elicit the skill they're actually trying to teach. Um, and right now, we don't bother with that because we can hoover up more data from the internet. We're getting towards the end of that. But yeah, as the AIs get more sophisticated, they'll be better able to tell, uh, what is a useful kind of skill to practice and to generate that. And we've done that in other areas. So AlphaGo, uh, the original Al- version of AlphaGo was booted up with data from human Go play, uh, and then improved, uh, with reinforcement learning and Monte Carlo tree search. Uh, but then AlphaZero, with a somewhat more sophisticated model, uh, benefited from some, some other improvements, um, but was able to go from scratch. Uh, and it generated its own data through self-play, uh, so which, getting data of a higher quality than the human data, there are no human players that good, uh, available in the dataset, and also, a curriculum, so that at any given point, it was playing games against an opponent of equal skill itself. Uh, and so it was always in a area when it was easy to learn. If you're, if you're just always losing no matter what you do, or always winning no matter what you do, it's hard to distinguish, uh, which things are better and which are worse. And when we have somewhat more sophisticated AIs that can generate training data and tasks for themselves, for example, if the AI can generate a lot of unit tests and then can try and produce programs that pass those unit tests, uh, then the interpreter is providing a training signal. And the, the AI can get good at figuring out what's the kind of programming problem that is hard for AIs right now, that will develop more of the skills that I need, uh, and then do them. And no, you're not gonna have, you know, employees at OpenAI write, like, a billion programming problems. That's just not gonna happen. Uh, but you are gonna have AIs given the task of producing those enormous number of programming challenges.

Dwarkesh Patel 27:44 In LLMs themselves is, you know, there's a paper out of Anthropic called Constitution AI or Constitution RL, where they basically had the program just, like, talk to itself and say, like, "Is this response helpful? If not, how can I make this more helpful?" And the, the response is improved, and then you train the model on the more helpful responses that it generates by talking to itself so that it generates them, uh, natively. And you could imagine, you know, more sophisticated ways to, to do that or better ways to do that. Okay, so but then the question is, listen, you know, GPT-4 already costs like 50 million or 100 million or whatever it was. Even if we have greater effective compute from hardware increases and better models, it's hard to imagine how we could sustain, like, four or five more orders of magnitude greater size, effective size, than GPT-4 unless we're jump- dumping in, like, trillions of dollars, like, the entire, the entire economies of big countries into training the next version. So, the question is do we get something that can significantly help with AI progress before we run out of the, the sheer, uh, money and scale and compute that would be required to train it? Do you have a take on that?

Carl Shulman 28:51 Well, first I'd say remember that there are these three contributing trends.

Dwarkesh Patel 28:55 Yeah.

Carl Shulman 28:56 So the new H100s are significantly better than the A100s and a lot of companies are actually waiting for their deliveries-

Dwarkesh Patel 29:04 (laughs)

Carl Shulman 29:04 ... uh, of H100s-

Dwarkesh Patel 29:06 Yeah.

Carl Shulman 29:06 ... uh, to do even bigger, uh, training runs. That's all along with the, the work of, uh, hooking them up into clusters and engineering the thing. Yeah, so all of those factors are contributing. Uh, and of course, mathematically, (laughs) uh, yeah, if you do four orders of magnitude more than 50 or 100 million, then you know, you're getting to a trillion dollar territory. And yeah, I think the, the way to look at it is at each step along the way, does it look like it makes sense to do the next step? Uh, and so from where we are right now, seeing the results with GPT-4 and ChatGPT, uh, companies like Google and Microsoft, uh, and whatnot, are pretty convinced that this is very valuable.

Dwarkesh Patel 29:53 Mm-hmm.

Carl Shulman 29:53 Um, you have, like, talk at Google, um, and Microsoft with Bing that, well, it's like billion dollar, billion dollar matter to change market share and search, uh, by a percentage point.

Dwarkesh Patel 30:06 Mm-hmm.

Carl Shulman 30:06 Um, and so that can fund a lot. And on the far end, on the extreme, if you automate human labor, we have a $100 trillion economy, most of that economy is paid out in wages.

Dwarkesh Patel 30:21 Mm-hmm.

Carl Shulman 30:21 So, like, between 50 and $70 trillion per year-... if you create AGI, it's going to automate all of that-

Dwarkesh Patel 30:30 Mm-hmm.

Carl Shulman 30:31 ... uh, and inc- keep increasing beyond that. Uh, so the, the value of the, of the completed project is very much worth, uh, throwing our whole economy into it, if you were gonna get the good version, not the catastrophic destruction of the human race, uh, or, you know, some, uh, some other disastrous outcome.

Dwarkesh Patel 30:53 Mm-hmm.

Carl Shulman 30:53 And in between, it's a question of, well, does the next step, how risky and uncertain is it and how much growth in the revenue you can generate with it, uh, do you get? And so if we're moving up to a billion dollars, I think that's absolutely gonna happen, these large tech companies have R&D budgets, tens of billions of dollars. And when you think about it, like, in the, in the relevant sense, like, all the, all the employees at Microsoft who are doing software engineering, that's like contributing to creating software objects.

Dwarkesh Patel 31:25 Yeah.

Carl Shulman 31:25 It's not, it's not weird to spend tens of billions of dollars on a product that would do so much.

Dwarkesh Patel 31:34 Yeah, yeah.

Carl Shulman 31:34 Uh, and I think it's, it's becoming more clear that there is sort of market opportunity to fund the thing. Uh, going up to a hundred billion dollars, that's like, okay, the existing R&D budgets spread over multiple years.

Dwarkesh Patel 31:48 Yeah.

Carl Shulman 31:48 Um, but if you, if you keep seeing that when you scale up the model, uh, it substantially improves the performance, it opens up new applications, you're not just improving your search, uh, but maybe, you know, it makes self-driving cars work, uh, you replace bulk software engineering jobs, or if not replace them, amplify productivity. Uh, in this kind of dynamic, you actually probably want to employ all the software engineers you can get, as long as they're able to make any contribution because the returns-

Dwarkesh Patel 32:16 Yeah.

Carl Shulman 32:16 ... of improving stuff in AI itself gets so high. But yeah, so I think that, that can go up to 100 billion and at 100 billion, you're using like a significant fraction of our existing fab capacity. Like right now, the revenue of NVIDIA is like 25 billion. Uh, the revenue of TSMC I believe is like, uh, it's over 50 billion. Last I, I checked in 2021, uh, NVIDIA was maybe, uh, seven and a half percent, less than 10% of TSMC revenue. Uh, so there's a lot of room, and most of that was not AI chips.

Dwarkesh Patel 32:59 Mm-hmm.

Carl Shulman 32:59 Um, you know, they have a large gaming segment.

Dwarkesh Patel 33:01 Yeah.

Carl Shulman 33:01 There are data center GPUs that are used for video and the like. Um, so there's room for, um, more than an order of magnitude, uh, increase by redirecting, uh, existing fabs to produce more AI chips and then just actually using the AI chips that these companies have in their cloud for the big training runs.

Dwarkesh Patel 33:23 Mm-hmm.

Carl Shulman 33:24 Uh, and so I think that, that's enough to go to the 10 billion and then combine with stuff like the H100s to go up to 100 billion.

Dwarkesh Patel 33:30 Just to emphasize for the audience the initial point about revenue you made, if, if it cost OpenAI a $100 million to train GPT-4, and it generates $500 million in revenue, you, you know, you pay back your expenses with the 100 million, you have 400 million for your next training run, then you train your GPT-4.5, you know, you get, let's say $4 billion of revenue out of that, that's where the for- feedback loop of sort of revenue comes from, where you're automating tasks and therefore you're making money, you can use that money to automate more tasks. On the ability to, uh, redirect the fab production towards AI chips, so, uh, then the TLDR on you want $100 billion worth of compute, I mean, fabs take what, like a decade or so to build? Um, so given the ones we have now and the ones that are gonna come online in the next decade, is there enough to sustain a $100 billion of GPU compute if you wanted to spend that on a training run?

Carl Shulman 34:26 Yes, you, you could definitely make the 100 billion one. As you go up to a trillion dollar run and larger-

Dwarkesh Patel 34:32 Mm-hmm.

Carl Shulman 34:33 ... um, it's going to involve more fab construction. And yeah, fabs can take a long, a long time to build. On the other hand, uh, if in fact you're getting very high revenue, uh, from the AI systems and you're actually bottlenecked on the construction of these fabs, uh, then their price could skyrocket.

Dwarkesh Patel 34:55 Mm-hmm.

Carl Shulman 34:55 And that, uh, lead to measures we've never seen before, uh, to expand and accelerate fab production. Like if you, if you consider, so at the limit as you're getting models that approach, uh, human-like capability, if you imagine things that are getting close to like brain-like efficiencies plus AI advantages, uh, we were talking before about, well, a GPU, um, that is supporting an AI, uh, really it's a cluster of GPUs supporting AIs that do things, uh, in parallel with data parallelism.

Dwarkesh Patel 35:28 Yeah.

Carl Shulman 35:28 But if that can work four times as much as a human, uh, a highly skilled, motivated, focused human with levels of education that have never been seen in the human population, uh, and so if like a typical software engineer can earn hundreds of thousands of dollars, the world's best software engineers, uh, can earn millions of dollars today and maybe more in a world where there's so much demand for AI. Um, and then times four, uh, for working all the time, well, I mean, if you, if you have, if you can generate like close to $10 million a year, um, out of the future version of H100-

Dwarkesh Patel 36:06 Mm-hmm.

Carl Shulman 36:07 ... uh, they cost tens of thousands of dollars with a huge profit margin now, uh, and profit margin could be, um, could be reduced with like large production, uh, that is a big difference. That, that chip, uh, pays for itself almost instantly. Um, and so you could, you could support-... paying 10 times as much to have these fabs constructed more rapidly. You could have, if AI is starting to be able to contribute, uh, you can have, could have AI contributing more of the skill technical work that makes it hard for, say, NVIDIA to suddenly find, uh, thousands upon thousands of top-quality engineering hires, uh, if AI can provide that. Now, if AI hasn't reached that level of performance, then this is how you can have things stall out. And like a world where AI progress stalls out is one where you go to the $100 billion and then over succeeding years, trillion dollar, uh, things, software progress, um, turn- turns to stall, turns out to- to stall, you- you lose the gains that you are getting from moving researchers from other fields, lots of physicists and people from other areas of computer science have been going into AI, but you sort of tap out those resources, uh, as it, AI becomes a larger proportion of the research field. Uh, and like, okay, you, you've put in all of these inputs, but they just haven't yielded AGI yet. I think that set of inputs probably would yield the kind of AI capabilities needed for intelligence explosion, but if it doesn't, after we've exhausted this current scale up of like increasing-

Dwarkesh Patel 37:46 Yeah.

Carl Shulman 37:46 ... the share of our economy that is trying to make AI, if that's not enough, then after that you have to wait for the slow grind of things like general economic growth, population growth and such, and so things slow. And that- that results in my credences and this kind of advanced AI happening to be relatively concentrated like over the next 10 years compared to the rest of the century, because we just can't, we can't keep going with this rapid redirection of

Carl Shulman 38:15 resources into AI. That's- that's a one-time thing.

Dwarkesh Patel 38:18 If the current scale-up works, it's going to happen, we're gonna get to AGI really fast, like within the next 10 years or something. If the current scale-up doesn't work, all we're left with is just like our economy growing like 2% a year, so we have like 2% a year more resources to spend on AI. And at that scale you're talking about decades before you can, um, just through sheer brute force you can, you know, train the $10 trillion model or something. Let's talk about why you have your- your thesis that the current scale-up would work. What- what is the evidence from AI itself or maybe from primate evolution and the evolution of other animals? Just give me the whole, (laughs) the whole confluence of reasons that make you think-

Carl Shulman 38:56 I- I think maybe the best way to look- look at that might be to consider when I first became interested in this area, so in the 2000s, which was before the deep learning revolution, how would I think about timelines? How did I think about timelines? And then how have I updated based on what has been happening with deep learning? Uh, and so back then, I would have said we know the brain is a physical object, an information processing device. It works, um, it's possible, and not only is it possible, it was created by evolution on Earth. And so that gives us something of an upper bound in that this kind of brute force was sufficient. Uh, there are some complexities, uh, with like, well, what if it was a freak accident-

Dwarkesh Patel 39:46 Mm-hmm.

Carl Shulman 39:47 ... and it, you know, that didn't happen on all of the other planets and that added some value? Um, I have a paper with Nick Bostrom on this. Um, I think basically that's not that important an issue. Um, there's convergent evolution. Like octo- octopi are also, uh, quite sophisticated. If the, if a special event was at the level of forming cells at all, um, or forming brains at all, we get to skip that because we're choosing to build computers and we already exist. We have that- that advantage. So say evolution gives something of an upper bound, really intensive massive brute force search, um, and things like evolutionary algorithms can produce intelligence.

Dwarkesh Patel 40:28 Doesn't the fact that, um, octopi and I guess other mammals, they got to the point of being like pretty intelligent but not human level intelligent, is that some evidence that there's a hard step between a cephalopod and a human?

Carl Shulman 40:41 Yeah. So that- that would be a place to look.

Dwarkesh Patel 40:44 Yeah.

Carl Shulman 40:44 Um, it doesn't seem particularly compelling. One source of evidence on that, uh, is work by, um, Herculano-Houzel. Uh, I hope I haven't mispronounced her name, but she's a neuroscientist who has dissolved the brains of many creatures, uh, and by counting, uh, the, uh, the nuclei she's able to, uh, determine how many neurons are- are present, uh, in different species and find a lot of interesting trends, uh, in scaling laws. And she has a paper, um, discussing the human brain as a scaled-up primate brain.

Dwarkesh Patel 41:21 Mm-hmm.

Carl Shulman 41:22 Uh, and across like a wide variety of animals and in mammals in particular, um, there are certain characteristic, uh, changes in the relative number of neuron size of different brain regions, uh, as things scale up. Um, there's a lot of, yeah, there's a lot of structural, um, structural simi- similarity there, and you can explain a lot of what is different about us with a pretty brute force story-

Dwarkesh Patel 41:50 Mm-hmm.

Carl Shulman 41:50 ... uh, which is that you expend resources on having a bigger brain, keeping it in good order, giving it time to learn, so we have an unusually long childhood, unusually long neonate period. We spend more compute by having a larger brain than other animals, uh, three, more than three times as large as chimpanzees, and then we have a longer childhood, um, than- than chimpanzees and much more than many, many other creatures. So we're spending more compute in a way that's analogous to like having a bigger model and having more training time with it.

Dwarkesh Patel 42:23 Yeah.

Carl Shulman 42:23 Um, and given that we see, um, with our AI models-... um, this sort of, like, large consistent benefits from increasing compute spent in those ways, and with qualitatively new capabilities, uh, showing up over and over again, uh, particularly in areas that sort of AI skeptics, uh, call out. In my experience, like, over the last 15 years, um, the things that people call out as like, "Ah, but the AI can't do that and it's because of a fundamental limitation."

Dwarkesh Patel 42:53 For sure.

Carl Shulman 42:53 Uh, I've gone through a lot of them. You know, there were Winograd schemas, uh, catastrophic forgetting, quite a number. Um, and yeah, they have repeatedly gone away, uh, through scaling. And so there's a, th- there's a picture that we're, we're seeing supported from biology and from our experience with AI, where you can explain, like, yeah, in general, there are trade-offs where the extra fitness you get from a brain is not worth it, and so cre- creatures wind up mostly with small brains because they can save that biological energy and that time to reproduce, for digestion, and so on. And humans, we actually seem to have wound up in a, a niche, uh, that was then self-reinforcing, where we greatly increased the returns to having large brains. And language and technology are the sort of obvious candidates. When you have humans around you who know a lot of things and they can teach you, and compared to almost any other species, we have vastly more instruction from parents and the society of the young'un, then you're getting way more from your brain. Because you can get, per minute, you can learn a lot more useful skills, and then you can provide the energy you need to feed that brain, um, by hunting and gathering, by having fire that makes digestion easier. And basically how this process goes on, it's increasing the marginal increase in reproductive fitness you get from allocating more resources along a bunch of dimensions-

Dwarkesh Patel 44:33 Mm-hmm.

Carl Shulman 44:33 ... towards cognitive ability. And so, um, that's bigger brains, longer childhood, having our attention be more on learning. So humans play a lot, and we, we keep playing as adults, which is a very weird thing compared to other animals. Um, we're more motivated to copy other humans around us, uh, than, like, even than the other primates. And so these are sort of motivational changes that keep us using more of our attention and effort on learning, which pays off more when you have a bigger brain and a longer lifespan in which to learn it. Many, many creatures are subject to lots of predation or disease. And so if you try, you know, you're, you're a mayfly or a mouse, if you try and invest in, like, a giant brain and a very long childhood, you're quite likely to be killed by some predator or some disease-

Dwarkesh Patel 45:24 Mm-hmm.

Carl Shulman 45:24 ... before you're able to actually use it. And so that means you actually have exponentially increasing costs in a given niche. So if, if, if I have a 50% chance of dying every few months, as a, you know, a little mammal or a little lizard or something, that means the cost of going from three months to 30 months of learning and childhood development-

Dwarkesh Patel 45:44 Yeah.

Carl Shulman 45:44 ... is now 10 times, uh, the loss. It's now, it's two to the negative 10. So one th- a factor of 1,024 reduction in the benefit I get from what I ultimately learn, because 99.9% uh, of the animals will have been killed before that point. We're in a niche where we're like a large, long-lived animal with language and technology, so wh- where we can learn a lot from our groups. And that means it pays off to really, um, just expand our investment on these multiple fronts in intelligence.

Dwarkesh Patel 46:19 That, that's so interesting. Um, I, uh, just for the audience, the, the calculation about, like, two to the whatever months is just like, you have a half chance of dying this month, a half chance of dying next month, um, you multiply those together. Okay. There's other species, though, that do live in flocks or as, um, packs where you could imagine... I mean, uh, they do have, like, a, a smaller version of the development of cubs into, that are, like, play with each other. Why isn't this a hill on which they could have climbed to human level intelligence themselves? If it's something like language or technology, um, humans were getting smarter before we got language. I mean, uh, obviously we had to get smarter to go get language, right? Uh, we couldn't just get language without becoming smarter. So yeah, where did... Wh- wh- what... It seems like, uh, there should be other species that should have beginnings of this sort of cognitive revolution, especially given how valuable it is, given, listen, we've dominated the world. You would think there'd be selective pressure for it.

Carl Shulman 47:16 Evolution doesn't have foresight.

Dwarkesh Patel 47:17 Yeah.

Carl Shulman 47:18 The thing in this generation that gets more surviving offspring and grandchildren, that's the thing that becomes more common. Uh, evolution doesn't look ahead and say, "Oh, in, in a million years, you'll have a lot of descendants." Uh, it's-

Dwarkesh Patel 47:33 Yeah.

Carl Shulman 47:34 ... what, what survives and reproduces now. Uh, and so in fact there, there are correlations where, uh, social animals, uh, do on average, uh, have larger brains. Uh, and part of that is probably, uh, that the additional social applications of brains, like keeping track of which of your group members have helped you before so that you can reciprocate. You scratch my back, I'll scratch yours. Uh, remembering who's dangerous within the group. That sort of thing, um, is an, is an additional application of intelligence. Um, and so there's some correlation there. Um, but what it, what it seems like is that, yeah, in most of these cases, um, it's enough to invest more, but not invest to the point where a mind can easily develop language and technology-

Dwarkesh Patel 48:24 Ah.

Carl Shulman 48:24 ... and pass it on. And so there are... You see bits of tool use, uh, in some other primates who have an advantage that... So compared to, say, the whales who have... They have quite large brains, partly because they are so large themselves and they-... have some other, other things. But they don't have hands, which means that reduces a bunch of ways in which brains can pay off, uh, and investments in the functioning of that brain. But yeah, so primates will use sticks, um, to extract termites. Capuchin monkeys will open clams by smashing them with a rock. Uh, so there's bits o- bits of tool use. But what they don't have is the ability to sustain culture. A particular primate will maybe discover one of these tactics, and maybe it'll be copied by their immediate group. Uh, but they, they're not holding onto it that well. They're like, well, when they see the other animal do it, they can copy it in that situation. They don't actively teach each other. Their population locally is quite small, so it's easy to forget things, easy to lose information. And in fact, they, they remain technologically stagnant for hundreds of thousands of years. Uh, and we can actually look at some human situations. So there's a, an old paper, uh, I believe by, uh, The Economist, uh, Michael Kramer, talks about technological growth in the different continents, uh, for human societies. And so you have Eurasia is the largest integrated, uh, connected area. Africa is partly connected to it, but the Sahara Desert, uh, restricts the flow of information and technology and such. Uh, and then you had the Americas, which were after the colonization from the land bridge were largely separated and, uh, are smaller than Eurasia. Then Australia, and then you had, like, smaller island situations like Tasmania. Uh, and so technological progress seemed to have been faster the larger-

Dwarkesh Patel 50:19 Mm-hmm.

Carl Shulman 50:19 ... the connected group of people. Uh, and in the smallest groups, so like in Tasmania, you had a relatively small population and they actually lost technology. Um, so things like they lost some, like, fishing techniques. Uh, and if you have a small population and you have some limited number of people who know a skill and they happen to die or it happened, there's like, you know, some, some change in circumstances that causes people not to practice or pass on that thing, uh, and then you lose it. And if you have few people, you're doing less innovation. The rate at which you lose technologies to some kind of local disturbance and the rate at which you create new technologies can wind up in balance. And the great change of hominids and humanity, uh, is that we wound up in this situation where we were accumulating faster than we were losing. And as we accumulated, those technologies allowed us to expand our population, they created additional demand for intelligence, uh, so that our brains became three times as large.

Dwarkesh Patel 51:24 As, as chimpanzees, right?

Carl Shulman 51:25 As chimpanzees, yeah.

Dwarkesh Patel 51:26 Yeah.

Carl Shulman 51:26 And our ancestors who had a similar-

Dwarkesh Patel 51:28 Mm-hmm.

Carl Shulman 51:29 ... brain size.

Dwarkesh Patel 51:30 Okay. And then the crucial point, uh, I guess in relevance to AI is that the selective pressures against intelligence in other animals are not, uh, acting against these neural networks because we are, you know, we're, they're not gonna get, like, eaten by a predator if they spend too much time becoming more intelligent. Um, we're, we're, like, explicitly training them to become more intelligent. So we have, like, good first principles reason to think that if it was scaling that made our minds this powerful and if the things that prevented other animals from, uh, scaling are not impinging on these neural networks, that these things should just continue to become very smart.

Carl Shulman 52:06 Yeah. We are, we're growing them in a technological culture where there are jobs like software engineer-

Dwarkesh Patel 52:11 Yeah.

Carl Shulman 52:11 ... that depend much more on sort of cognitive output and less on things like metabolic resources devoted to the immune system-

Dwarkesh Patel 52:20 Mm-hmm.

Carl Shulman 52:21 ... uh, or to, like, building big muscles to throw spears.

Dwarkesh Patel 52:24 Yeah. Th- this is kind of a side note, but I'm just kind of interested. I think you referenced at some point chinchilla scaling. For the audience, this is a paper from DeepMind which describes if you have modeled a certain size, what is the optimum amount of data that ch- it should be trained on? So you can imagine bigger models, you can, you can use more data to train them. And then this way, you can figure out, where should you spend your compute? Should you spend it on making the model bigger or should you spend it on training it for longer? I'm curious if, uh, in the case of different animals, in some sense they're like model sizes or how big their brain is and their training data sizes, like how long they're cubs or how long they're infants or toddlers or, um, before they're full adults. Is there- is there some sort of like scaling law of...

Carl Shulman 53:04 Yeah, I mean, so the- the- the chinchilla scaling is in- interesting because they, we were talking earlier about the cost function for having a longer childhood.

Dwarkesh Patel 53:12 Yeah.

Carl Shulman 53:13 And so where it's like exponentially increasing in the amount of-

Dwarkesh Patel 53:17 Yeah.

Carl Shulman 53:17 ... training compute you have when you have exogenous forces that can kill you. Um, whereas when we do big training runs, the cost of throwing in more GPUs is almost linear. Uh, and it's much better (laughs) to be l- you know, linear than exponentially decayed.

Dwarkesh Patel 53:30 Mm-hmm. Oh, that- that's a really good point, yeah.

Carl Shulman 53:31 As we expand resources.

Dwarkesh Patel 53:32 Yeah.

Carl Shulman 53:32 And so chinchilla scaling would suggest that, like, yeah, the opt for a- a brain of sort of human size, it would be optimal to have many millions of years of education. But obviously, that's-

Dwarkesh Patel 53:45 Mm-hmm.

Carl Shulman 53:45 ... impractical because of exogenous mortality for humans. Uh, and so there's a, a fairly compelling argument that relative to the situation where we would train AI that animals are systematically way under-trained.

Dwarkesh Patel 54:02 That's so interesting.

Carl Shulman 54:02 And now they're, they're more efficient than our models. We still have room to improve our algorithms to catch up with the efficiency of brains. But they are laboring under that disadvantage, yeah.

Dwarkesh Patel 54:16 That is so interesting. Okay, so I guess another question you could have is humans got started on this evolutionary hill climbing route where we're getting more intelligent and has more benefits for us. Why didn't we go all the way on that route? If intelligence is so powerful, why aren't all humans as smart as we know humans can be?... right? A- at least that smart. Um, i- if intelligence is so powerful, like, why hasn't there been stronger selective pressure? I understand, like, oh, listen, hip size, you can't, like, give birth to a really big-headed baby or whatever, but you would think, like, evolution would figure out some way to, uh, offset that, such be- if intelligence has such big power and such- it's so useful.

Carl Shulman 54:56 Yeah, I- I think if you actually look at it quantitatively that's- that's not true, and even in- in sort of recent history there has been a- it looks like a pretty close balance between the costs and the benefits-

Dwarkesh Patel 55:09 Mm.

Carl Shulman 55:09 ... of having, uh, more cognitive abilities. And so you say, like, ah, you know, who needs to worry about, like, you know, the metabolic costs? Like, you need m- humans put, like, or order 20%-

Dwarkesh Patel 55:23 Mm-hmm.

Carl Shulman 55:24 ... of our metabolic energy into the brain, and it's higher for, like, young children.

Dwarkesh Patel 55:28 Yeah.

Carl Shulman 55:29 So 20% of the e- along w- and then, you know, there's, like, breathing, and digestion, and the immune system, and so r- for most of history people have been dying left and right.

Dwarkesh Patel 55:41 Mm.

Carl Shulman 55:41 Like, a very large proportion of people will die of infectious disease.

Dwarkesh Patel 55:46 Yeah, yeah.

Carl Shulman 55:46 And if you put more resources into your immune system you survive-

Dwarkesh Patel 55:51 Mm.

Carl Shulman 55:52 ... so it's like life or death pretty directly via that mechanism. Uh, and then this is- this related also in people die more of disease during famine-

Dwarkesh Patel 56:03 Mm.

Carl Shulman 56:03 ... and so there's boom or bust, and so if you have 20% less metabolic requirements or has- and you're a child-

Dwarkesh Patel 56:09 Mm.

Carl Shulman 56:10 ... and if you have a lot more, I mean, it's like 40 or 50% less metabolic requirements, you're much more likely to survive that famine. So these- these are- these are pretty big. Uh, and then there's a trade-off about just cleaning mutational load. So every generation new mutations and errors happen in the process of reproduction and so, like, we know there are many, um, genetic abnormalities, uh, that occur through new mutations each generation. And in fact, uh, we ha- Down syndrome is the chromosomal abnormality that you can survive. All the others just-

Dwarkesh Patel 56:45 Kill you.

Carl Shulman 56:45 ... kill the embryo, um, and so we never see them. Um, but, like, yeah, D- Down syndrome o- occurs a lot, and there are many other lethal mutations and there's, as you go to the less damaging ones there are enormous numbers of less damaging mutations that are degrading every system in the body. And so evolution each generation has to pull away at some of this mutational load, and the priority with which that mutational load is pulled out scales in proportion to how much the traits it's affecting-

Dwarkesh Patel 57:18 Ah. Yeah, yeah.

Carl Shulman 57:18 ... impact fitness. So now you got new mutations that impact your resistance to malaria-

Dwarkesh Patel 57:24 Yep.

Carl Shulman 57:25 ... you got new mutations that damage brain function-

Dwarkesh Patel 57:27 Mm-hmm.

Carl Shulman 57:28 ... um, and then of those mutations are purged each generation. Uh, if malaria is a bigger difference in mortality than, like, the incremental effectiveness as a hunter-gatherer you get from, you know, being slightly more intelligent, then you'll purge that mutational load first.

Dwarkesh Patel 57:46 Sure.

Carl Shulman 57:47 Uh, and similarly if there's, like, you know, humans have been vigorously adapting to new circumstances, uh, so since agriculture people have been developing things like the ability to, um, you know, have, uh, amylase to digest, uh, breads, the ability to, like, digest milk, um, and if you're evolving for all of these things and if some of the thing that give an advantage for that incidentally carry along nearby them some negative effect on another trait, then that other trait can be damaged. So it really matters how important to survival and reproduction cognitive abilities were compared to everything else that organism has to do, and that in particular, like, surviving feast and famine, uh, having, like, the physical abilities to do-

Dwarkesh Patel 58:34 Yeah.

Carl Shulman 58:34 ... hunting and- hunting and gathering and, like, even if you're, like, very good at planning your hunting, um, being able to throw a spear harder (laughs) -

Dwarkesh Patel 58:41 (laughs)

Carl Shulman 58:42 ... can, you know, be a big difference, and that needs energy to build those muscles and then to sustain them.

Dwarkesh Patel 58:47 Mm-hmm.

Carl Shulman 58:47 Uh, and so given all of these- all of these factors, um, it's like, yeah, it's n- it's not a- a slam dunk-

Dwarkesh Patel 58:55 Mm.

Carl Shulman 58:56 ... to invest at the margin, and like today, like, having bigger brains, for example, is- it's associated with, like, greater cognitive ability-

Dwarkesh Patel 59:04 Mm.

Carl Shulman 59:04 ... but it's, like, it's modest. Um, large-scale pre-register studies- pre-registered studies with MRI data, you know, is- it's, like, you know, in a range maybe like a correlation of .25, .3 and the standard deviation of brain size is, like, 10%.

Dwarkesh Patel 59:22 Mm-hmm.

Carl Shulman 59:22 So if you double the size of the brain, so go, and the existing brain costs like 20% of metabolic energy, go up- go up to 40%, okay, that's like eight- eight standard deviations of brain size. If the correlation is, like, say it's .25, um, then yeah, like, you can- you- you get gain from that, eight standard deviations of brain size, two standard deviations of cognitive ability, and like in our modern society where cognitive ability is very rewarded and, like, uh, you know, finishing- finishing school, becoming an engineer, or a doctor, or whatever can- can pay off a lot financially, still the, like, the average observed return, um, in, like, income is, like, a 1 or 2% proportional increase. There's- there's more effects at the tail, there's more effect in professions like STEM, uh, but on the whole it's not like, you know, if it was like a 5% increase or a 10% increase then you could- you could- you could tell a story where, yeah, this is hugely increasing-

Dwarkesh Patel 1:00:29 Yeah.

Carl Shulman 1:00:29 ... the amount of food you could have, you could support more children, but it's like, it's a modest effect and the metabolic cost would be large, and then throw in these other- these other aspects and I think it's... You can tell the story and also we can just- we can see there was not-... very strong, rapid directional-

Dwarkesh Patel 1:00:46 Yeah.

Carl Shulman 1:00:47 ... selection on the thing. Which there would be if, like, you know, you could (laughs) by solving a, like a, by solving, like, a math puzzle you could defeat malaria. (laughs) Um, like, then, then there would be more evolutionary pressure.

Dwarkesh Patel 1:01:00 That is so interesting. And, uh, not to mention of course that, um, yeah, if you had, like, two X the brain size you're, you would li- without C-section you would, like, your mother would, or both would die. That I, this is a question that I've actually been curious about for, like, over a year. And I, like, l- briefly tried to look up an answer. This is, uh, I know this is off-topic, but I've... So h- I apologize to the audience, but I was super interested, and that was, like, a, that was, like, the most comprehensive and interesting answer I could have hoped for. Okay. So yeah. We have a good explanation of good first principles, evolutionary reason for thinking that intelligence scaling up to humans is not, um, implausible just by throwing more scale at it.

Carl Shulman 1:01:36 I, I would also add, uh, this was something that would have mattered to me more in the 2000s, uh, we also have the brain right here with us, uh, for, available for neuroscience to reverse engineer-

Dwarkesh Patel 1:01:47 Mm.

Carl Shulman 1:01:47 ... its properties. And so uh, in the 2000s when I, when I said, "Yeah, I expect this by, you know, middle of this century-ish," that was a backstop if we found it absurdly difficult to get to the algorithms and then we would learn from neuroscience. But in the actual, the actual history, it's, it's really not like that. We develop things in AI and then also we can say, "Oh, yeah. This is sort of like this thing, uh, in neuroscience." Or maybe there's just a good explanation. But it's not n- as though neuroscience is driving AI progress. It's, it's turns out not to be that necessary.

Dwarkesh Patel 1:02:18 Yeah, similar to, I guess, you know, how, uh, planes were inspired by the existence proof of, of birds but we, it, it would, jet, jet engines don't flap. All right. So yeah. Uh, scaling, uh, good reason to think scaling might work. So we, we spend $100 billion and we have something that is like human level or can do, help significantly with AI research.

Carl Shulman 1:02:39 I mean that, that might be on the earlier end-

Dwarkesh Patel 1:02:41 Mm-hmm.

Carl Shulman 1:02:41 ... um, but I mean, I would, I definitely would not rule that out

Carl Shulman 1:02:45 given the rates of change we've seen-

Dwarkesh Patel 1:02:47 Okay.

Carl Shulman 1:02:47 ... with the last few scale ups.

Dwarkesh Patel 1:02:49 All right. So at this point, somebody might be skeptical, okay, like, listen, we already have a bunch of human researchers, right? Like, the incremental researcher, how, how powerful is that? And then you might say, well, no, this is like thousands of researchers. I don't know how to express this, uh, skepticism exactly, but sceptics is skeptical of just generally the effect of scaling up the number of people working on the problem to rapid, rapid progress on that problem. W- what, somebody might think, okay listen, with humans the reason population working on a problem is such a good proxy for progress on that problem is that there's already so much variation that is accounted for when you say there's like a million people working on a problem. You know there's like a, a co- hundreds of super geniuses working on it, thousands of people who are like very smart working on it. Whereas with an AI, all the copies are like the same level of intelligence. Um, and if it's not super genius intelligence, the, uh, the, the, the, the total quantity might not matter as much.

Carl Shulman 1:03:43 Yeah, I'm not sure what your, um, model is here. So is this the model that the diminishing returns kick-off suddenly has a cliff right where we are? And so like, there was, there were results in the past from throwing more people at problems. Um, and I mean, this has been useful in historical prediction. Um, one of the, there's this idea of experience curves and Wright's Law, um, basically measuring cumulative production in a field or, which is a, also gonna be a mea- a measure of like the scale of effort and, and investment. And people have used this correctly, uh, to argue that renewable energy technology like solar, uh, would be falling rapidly in price because it was going from a, a low base of very small production runs, not much investment in doing it efficiently, um, and yeah. Uh, climate advocates correctly called out, um, people and people like David Roberts, um, and the futurist Ramez Naam, uh, actually has some, some interesting, uh, writing on this that, yeah, correctly called out that there would be a really drastic, uh, fall in prices of solar and batteries because of the increasing investment going into that. Um, the Human Genome Project would be another. So I'd say that there's like, yeah, real, real evidence. These observed correlations from, like, ideas getting harder to find have, have held over a fair, uh, a far range of data and over quite a lot of time. Uh, so I'm wondering-

Dwarkesh Patel 1:05:20 But, but what you're specifically-

Carl Shulman 1:05:21 ... what, what, what's the, yeah, the, the nature of the deviation you're thinking of.

Dwarkesh Patel 1:05:25 That, um, we're talking about, uh, maybe th- this is like a good way to describe what happens when more humans enter a field. But does it even make sense to say like a greater population of AIs is doing AI research if there's like more GPUs running a copy of GPT-6 doing AI research? M- it just like, how, how applicable are these, uh, economic models of human, uh, the quantity of humans working on a problem to the, to the magnitude of AIs working on a problem?

Carl Shulman 1:05:53 Yeah. So if you have AIs that are directly automating, uh, you know, particular jobs that humans were doing before, then we say, "Well, with additional compute, we can run more copies of them, uh, to do more of those tasks simultaneously. Uh, we can also run them at greater speed." And so some people have an intuition that like, "Well, you know what matters is, like, time. It's not how many people are working on a problem at a g- at a given point." Um, I think that that doesn't bear out super well. Um, but AI can also be run faster than humans. And so if you have a, a set of AIs, um, that can do the work of the individual human researchers and run at 10 times or 100 times, uh, the speed and we ask, "Well, could the human research community have solved these algorithmic problems, do things like invent transformers, uh, over 100 years?"... uh, if we have this, um, we have AIs with a population, effective population similar to that of the humans but running 100 times as fast. Uh, and so you ha- you have to tell a story where, no, the AI, they can't really do the same things as the humans. Uh, and we're talking about what happens when, uh, the AIs are more capable of, in fact, doing that.

Dwarkesh Patel 1:07:12 Although they become more capable as l- lesser capable versions of themselves help us ma- make themselves more capable, right? So w- you have to, like, kickstart that at some point. Is there an example in analogous situations, is intelligence unique in the sense that you have a feedback loop of w- with a learning curve or something else, a system's outputs are feeding into its own inputs in a way that... Because if we're talking about something like Moore's law or the cost of solar, you do have this way where, where, you know, more people are, we're throwing more people at the problem and it's, um, we're, you know, we're making a lot of progress. But we don't have the sort of additional part of the model where Moore's law leads to more humans somehow, uh, and the more humans are becoming researchers.

Carl Shulman 1:07:58 So you do actually have a version of that in the case of solar. So you have a small infant industry that's doing things like providing solar panels for space satellites and then getting increasing amounts of subsidized government demand because of, uh, you know, worries about fossil fuel depletion and then climate change. You, you can have the dynamic where visible successes, uh, with solar or, like, lowering prices then open up new markets. So there's a particularly huge transition where renewables become cheap enough to replace large chunks of the electric grid. Uh, earlier, you're li- you're dealing with very niche situations like, yeah, so the satellites where you have (laughs) , uh, it's very difficult to refuel a satellite, uh, in place and then remote areas and then moving to, like, you know, the super sunny, the sunniest areas in the world with the biggest solar subsidies. Um, and so there was an element of that where more and more investment has been thrown into the field and, like, the market has rapidly expanded as the technology improved. But I think the, the closest analogy, um, is actually the long run growth of human civilization itself. And I know you had Holden Karnofsky from the Open Philanthropy Project on earlier and discussed some of this research about the long run acceleration of human population and economic growth. And so developing new technologies allowed human population to expand, uh, humans to o- occupy new habitats and new areas and then to invent agriculture which supported larger populations and then even more advanced agriculture in the modern industrial society. And so their total technology and output allowed you to support more humans who then would discover more technology, uh, and continue the process. Now, that was boosted because on, on top of expanding the population, the share of human activity that was going into invention and innovation went up. And that was a key part of the Industrial Revolution. There was no such thing as a corporate research lab or, like, an engineering university, um, prior to that. And so you, you were both increasing the total human population and the share of it going in. But this population dynamic is pretty, is pretty analogous. Humans invent farming, they can have more humans, then they can invent industry and so on.

Dwarkesh Patel 1:10:21 Mm. So maybe somebody would be skeptical that with AI progress specifically, it's not just a matter of, you know, some, like, um, some farmer fig- figuring out crop rotation or some blacksmith figuring out how to do metallurgy better. You in fact, even to make the, uh, for the 50% improvement in productivity, you basically need something on the IQ that's close to Ilya Sutskever. Eh, so there's, like, a discontinuous, um... You're, like, contributing very little to productivity and then you're like Ilya and then you contribute a lot. But the, the becoming Ilya is... You, you see what I'm saying? There's not, like, a gradual increase in capabilities at least with the feedback loop.

Carl Shulman 1:10:57 You're imagining a case where, uh, the distribution, uh, of tasks is such that there's nothing that you can, where individually automating it particularly helps. Uh, and so the ability to contribute to AI research is really end loaded. Is that what you're saying?

Dwarkesh Patel 1:11:14 Yeah. Uh, I mean, we already see this in, in these sorts of, like, really high IQ, uh, companies or projects where theoretically, I guess Jane Street or OpenAI could hire, like, a bunch of, uh, you know, mediocre people to do... There's a comparative advantage. They could do some, uh, menial tasks and that could free up the time of the really smart people. But they don't do that, right? Uh, transaction costs, whatever else.

Carl Shulman 1:11:37 Self-driving cars would be another example where you have a very high quality threshold. And so when you, your performance as a driver is worse than a human, like, you have 10 times the accident rate or 100 times the accident rate, then the cost of insurance for that, which is a, a proxy for people's willingness to ride the car and stuff too-

Dwarkesh Patel 1:11:56 Yeah.

Carl Shulman 1:11:56 ... um, would be such that the insurance costs would absolutely dominate. So even if you have zero labor cost, it's offset by the increased insurance costs.

Dwarkesh Patel 1:12:03 Right.

Carl Shulman 1:12:03 Uh, and so there are lots of cases like that where, like, partial automation is not in practice, uh, very usable because complementing other resources, you're gonna use those other resources le- less efficiently. Um, you know, in, uh, in a post-AGI future, um, and the same thing can apply to humans. So people can say, "Well, comparative advantage, you know, even if AIs can do everything better than a human, uh, well, it's still worth something." A human can do, do something, make an em- you know, they can lift a box. That's something. Um, now there's a question of property rights if, well, if they could just-

Dwarkesh Patel 1:12:43 Slipper rocks. (laughs)

Carl Shulman 1:12:43 ... replace a human and use them to make more robots.

Dwarkesh Patel 1:12:45 Yeah, yeah.

Carl Shulman 1:12:46 Um, but even, even absent that, uh, in such an economy, you wouldn't want to let a human worker-... into any industrial environment because, you know, in a clean room, they'll be emitting all kinds of skin cells and messing things, things up. You need to have an atmosphere there, you need a bunch of supporting tools and resources and materials. And those supporting resources and materials will do a lot more productively working with AI and robots, rather than a human. So you don't wanna let, let a human anywhere near the thing, just like, you know, in a, uh, you don't, you don't wanna have a gorilla wandering around in a China shop even if you've trained it to most of the time pick up a box for you if you give it a banana. It's just not worth it to have it wandering around your China shop.

Dwarkesh Patel 1:13:27 Yeah, yeah, yeah. But, like, why, why is that not a good objection to-

Carl Shulman 1:13:30 I mean, I think that, that is one of the, the ways, uh, in which partial automation, uh, can fail to really translate into a lot of economic value.

Dwarkesh Patel 1:13:40 Mm-hmm.

Carl Shulman 1:13:40 Um, that's something that will attenuate as we go on and as the AI is more able to work independently and more able to handle its own, uh, its own screw-ups, get more, more reliable.

Dwarkesh Patel 1:13:53 But the way in which it becomes more reliable is by AI progress speeding up, which happens if AI can contribute to it.

Carl Shulman 1:14:01 Yeah.

Dwarkesh Patel 1:14:01 But if there is the, if there is some sort of reliability bottleneck that prevents it from contributing to that progress, then you don't have the loop, right? So...

Carl Shulman 1:14:08 Yeah, I mean, this is, this is why we're not there yet.

Dwarkesh Patel 1:14:11 Right. But then what is the reason to think we'll be there at-

Carl Shulman 1:14:13 The broad reason is we have these inputs are scaling up. There's a, so Epoch, which I mentioned earlier, uh, they have a paper, I think it's called Compute Trends in Three Eras of Machine Learning, uh, or something like that. Uh, and so they look at the compute expended on machine learning systems since the found- founding of the field of AI at the beginning of the 1950s. Um, and so it mostly, it grows with Moore's Law. Um, and so people are spending a similar amount on their experiments, um, but they can just buy more with that because the compute is coming. And so that data, I mean, it covers ov- over 20 orders of magnitude, maybe like 24, uh, and of all of those increases since 1952, a little more than half of them happened between 1952 and 2010 and all the rest is since 2010. So we're, we've been scaling that up, like, four times as fast as was the case for most of the history of AI. We're running through the orders of magnitude of possible resource inputs you could need for AI much, much more quickly than we were for most of the history of AI. That's why this is a period of, like, with a very elevated chance of AI per year because we're moving through-

Dwarkesh Patel 1:15:38 Mm-hmm.

Carl Shulman 1:15:38 ... so much of the space of inputs per year. And indeed, it looks like this scale up taken, taken to its conclusion, uh, will cover another bunch of orders of magnitude. Uh, and that's actually a large fraction of those that are left before you start running into saying, well, this is gonna have to be like evolution with the sort of simple hacks we get to apply. Like, we're selecting for intelligence the whole time, we're not going to make, do the same mutation that causes fatal, fatal childhood cancer a billion times even though, I mean, we, we keep getting the same fatal mutations even though they've been done many times. We use gradient descent, which takes into account the derivative, uh, of improvement on the loss all throughout the network and we, we don't throw away all of the contents of the network with each generation where you compress down to a little DNA. So there's that, that bar of like, well, if you're gonna do brute force like evolution combined with these sort of very simple ways, we can save orders of magnitude on that. We're gonna cover, I think, a fra- a fraction that's like half of that distance, uh, in this scale up over the next 10 years or so. And so if you started off with a kind of vague uniform prior, you're like, "Well, it's probably, you probably can't make AGI with, like, the amount of compute that would be involved in a fruit fly existing for a minute."

Dwarkesh Patel 1:17:01 Mm-hmm.

Carl Shulman 1:17:02 Which would be the early days of AI.

Dwarkesh Patel 1:17:03 Yeah.

Carl Shulman 1:17:04 Um, you know, maybe you, maybe you would get lucky we were able to make calculators because calculators benefited from, like, very reliable serially fast computers and where we could take a tiny, tiny, tiny, tiny fraction of a human brain's compute and use it for a calculator. We couldn't take an ant's brain and rewire it to calculate, it's hard to (laughs) hard to, uh, manage ant farms let alone get them, uh, to do arithmetic for you. Uh, and so there were some things where we could exploit the differences between biological brains and computers, uh, to do stuff super efficiently on computers. We would doubt that we would be able to do so much better than biology that we, with that tiny fraction of an insect's brain, uh, we'd be able to get AI early on. On the far end, it seemed very implausible that we couldn't do better than completely brute force evolution. And so in between you have some number of orders of magnitude of inputs-

Dwarkesh Patel 1:18:00 Mm-hmm.

Carl Shulman 1:18:00 ... where it might be. And like in the 2000s I would say, "Well, you know, I'm gonna have a pretty uniform-ish prior. I'm gonna put weight on it happening at, like, the sort of equivalent of, like, 10 to the 25 ops, 10 to the 30, 10 to the 35, um, and sort of spreading out over that and then I can update on other information." And in the short term I would say, like in 2005, I would say, "Well, I don't see anything that looks like the cusp of AGI so I'm also gonna lower my credence for like the next five years or the next 10 years." And so that, that, that would be kind of like a vague prior. And then when we take into account like, well, how quickly are we running through those orders of magnitude? If I have a uniform prior-

Dwarkesh Patel 1:18:42 Mm-hmm.

Carl Shulman 1:18:43 ... I assign half of my weight to the first half (book thuds) of remaining orders of magnitude and if we're gonna run through those over the next 10 years and some-... uh, then that calls on me to put half of my credence conditional on where we're ever gonna make it, AI, which seems likely, w- with a material object easier than evolution. I've gotta put similarly, a lot of my credence on AI happening in the scale up and then that's supported by what we're seeing in terms of the, uh, the rapid advances and capabilities, uh, with AI and LLMs in particular.

Dwarkesh Patel 1:19:16 Okay. The, that, that's actually a really interesting point. So now, but somebody might say, "Listen, there's not some sense in which AIs could universally speed up the progress of open AI by 50% or 100% or 200%. If they're not able to do everything better than Ilya Sutskever can, there's going to be something in which we're bottlenecked by the human researchers." And bottleneck effects dictate that, you know, the slowest moving part of the organization will be the one that kind of determines the speed of the progress of the whole organization or the whole project, which means that unless you get to the point where you're like, doing everything and, everybody in the organization can do, you're not going to significantly speed up the progress of the whole project as a whole.

Carl Shulman 1:19:57 Yeah. So that, that is a hypothesis and I think there's a lot of truth to it. So when we think about like the ways in which AI can contribute, so there are things we talked about before, like the AIs setting up their own curriculum, uh, and that's something that Ilya can't do directly and doesn't do directly. Uh, and there's a question, uh, how much does that improve performance? Um, there are these things where the AI helps, uh, to just like produce some code for, for some task and- and it's beyond hello world at this point, but I mean, the- the sort of thing that I hear from, uh, AI researchers at leading labs is that, you know, on their, on their core job where they're like most expert, it's not helping them that much. Uh, but then, you know their job often does involve, oh, I've got to, I've got to code something that's out of my usual area of expertise, um, or I want to research this question and it helps them there. Um, and so that saves some of their time and frees them to do more of the bottlenecked work. Uh, and then I think the, the idea of, uh, well, um, is, you know, is e- is everything, uh, dependent on Ilya and is Ilya so much better than the, uh, hundreds of other, uh, employees? I think there are a lot of people who are contributing, they're doing a lot of tasks. Um, and so you can have quite a- a lot of- of gain from automating some areas where you then do just an absolutely enormous amount of it relative to what you would have done before because things like designing the custom curriculum, you like, maybe had some humans put some work into that. Um, but you're not gonna employ billions of humans to produce it at scale. And so it winds up being a larger share of the progress, uh, than it was before. You get some benefit from these sorts of things where, ah, yeah, there's like pieces, pieces of my job, um, that now I can hand off to the AI and lets me focus more on the things that the AI still can't do. Uh, and then at the, later on, you get to the point where, yeah, the AI can do your job, uh, including the most difficult parts and maybe it has to do that in a- a different way. Maybe it like thro- spends a ton more time thinking about, um, each step of a problem than you, uh, and that- that's- and that's delayed end. And the stronger these bottlenecks effects are, the more the economic returns, the scientific returns and such are end loaded towards getting sort of full AGI. The weaker the bottlenecks are, the more interim results-

Dwarkesh Patel 1:22:30 Yeah.

Carl Shulman 1:22:30 ... uh, will be really paying off.

Dwarkesh Patel 1:22:32 I guess I probably disagree with you on how much the sort of, the- the- the Ilyas of organizations seem to matter. I guess just from the evidence alone, like, h- h- how many of the big sort of breakthroughs that in deep learning in general was like that single individual responsible for, right? Um, and how much of his time is he spending doing anything that's an a- that like Copilot is helping him on? I'm guessing like most of it is just like managing people and coming up with ideas and, uh, yeah, trying to like understand systems and so on. Um, and if that is the, if like the five or 10 people who are like that at OpenAI or Anthropic or whatever are the, are basically the way in which the progress is happening, or at least the algorithmic progress is happening, then how much of better and better Copilot... I know Copilot is not the thing you're talking about with like 20% automation, but something like that. How much of, um, yeah. How- how much is that contributing to the sort of like core function of, uh, the- the research scientist?

Carl Shulman 1:23:29 Yeah. Not sure quantitatively how much we, uh, we disagree about the importance of sort of, uh, key research employees and such. I certainly think that, uh, some researchers, um, you know, add, you know, more than 10 times, uh, the average employee, uh, even much more. Uh, and obviously managers can add an enormous amount of value by proportionately multiplying the output of the many people, uh, that they manage. And so that's, uh, the kind of thing that we were discussing earlier, uh, when talking about well, if you had sort of full human level, um, AI or AI that had all of the human, uh, capabilities plus AI advantages, uh, it would be, you know, you- you'd benchmark not off of what the sort of typical human performance is but peak human performance and beyond. So yeah, I accept all that. Um, I do think it makes- makes a big difference, uh, for people, uh, how much they can outsource a lot of the tasks that are less wow, less creative and an enormous amount is learned by experimentation. ML has been-

Dwarkesh Patel 1:24:40 Mm-hmm.

Carl Shulman 1:24:40 ... um, you know, quite experimental, uh, field and there's a lot of engineering work in say, building large super clusters, um, making, um, yeah, har- hardware aware optimization and coding of these things. Uh, being able to, uh, do the parallelism, uh, in large models and the- the en- the engineers, um, are busy. Uh, and it's- it's not just only a- a big thoughts, uh, kind of area and then the- the other branch, um, is...... where will the AI advantages and disadvantages be? Um, and so one AI advantage is being omnidisciplinary and familiar with the newest things. So I mentioned before, there's no human who has a million years of tensor flow experience. Uh, and so to the extent that we're, we're interested in like the very, very cutting edge of things that have been developed quite recently, then AI that can learn about them in parallel and experiment and practice with them in parallel can learn much faster than a human potentially. Uh, and the, the area of computer science is one that is especially suitable for AI to learn in a digital environment. So it doesn't require like driving a car around that might kill someone, have enormous cost. You can, you can do unit tests, you can, you can prove theorems, um, you can do all sorts of operations entirely in the confines