Marc Andreessen and Ben Horowitz on the State of AI

Speaker 0:00 I think we don't yet know the shape and form of the ultimate products. Just one just obvious historical analogy is, you know, the personal computer from sort of invention in nineteen seventy-five through to, you know, basically nineteen ninety-two was a text prompt system. Seventeen years in, you know, the whole industry took a left turn into GUIs and never looked back. And then by the way, you know, five years after that, the industry took a left turn into web, web browsers and never looked back, right? And, you know, look, I'm, I'm sure there will be chatbots twenty years from now, but I'm pretty confident that both the current chatbot companies and many new companies are gonna figure out many kinds of user experiences that are radically different that we don't, we don't even know yet. Please join me in welcoming Marc Andreessen and Ben Horowitz with General Partner Erik Torenberg.

Ben Horowitz 0:41 Follow me into a solo. Get in the flow. And you can picture like a photo. Music makes mellow maintains to make melodies. For MCs, motivates the pranks. I'm everlasting, I do-

0:51 Thank you for the-

Speaker 0:51 Ben-

Ben Horowitz 0:51 ... Rakim, who did that. [laughs]

Speaker 0:53 Ben, Ben picks the mu-- Ben picked the music.

Ben Horowitz 0:55 [laughs]

Erik Torenberg 0:57 Marc, there's been a lot of talk lately about the limitations

Erik Torenberg 1:00 of LLMs, that they, they can't do true invention of, say, new science, that they can't do true creative genius, that it's just combining or packaging. You have thoughts here. What say you?

Speaker 1:13 Yeah. So, so for me, it's, yeah, so you get all these questions, and yeah, they usually come in either if, if, you, you know, sort of are, are, are language models intelligent in the sense of like, can they actually, um, you know, can they actually process information and have sort of conceptual breakthroughs the way that people can? And then there's, are, are language models or, or video models creative? You know, can they, can they create new art, um, actually have genuine creative breakthroughs? And of course, my, my, my answer to both of those is, well, can people do those things?

Erik Torenberg 1:36 [laughs]

Speaker 1:36 Um, and, um, I think there's two, two, two questions there, which is like, okay, even if some people are, quote, unquote, "intelligent" as in having original, uh, conceptual breakthroughs and not just, let's just say, regurgitating the training set, um, uh, or following scripts, um, uh, how, you know, what percentage of people can actually do that? And I'd say I've, I've only met a few. Some of them are here in the room. Um, but, uh, you know, not that many. Most people never do. And then creativity, I mean, how many people are actually genuinely creative, right? And so, you, you kind of point to a Beethoven or a, you know, Van Gogh or something like that, and you're like, "Okay, that's creativity." And yeah, that's creativity, and then how many Beethovens and Van Goghs are there? Obviously, not very many. So, so one is just like, okay, like, you know, if it's, if it, if, if, if these things clear the bar of, you know, ninety-nine point nine nine percent of humanity, you know, then that's pretty interesting just in and of itself. But then I-- you dig into it further, and you're like, "Okay, like, how many actual real conceptual breakthroughs have there ever been actually ever in human history a-as compared to sort of remixing, remixing ideas?" Um, and, you know, like, if you look at the history of technology, it's almost always the case that the big breakthroughs are the result of, you know, usually at least forty years of sort of work ahead of time, you know, four decades. Right. In fact, language models themselves are the culmination of eight decades, right, of, of previous work. And so there's remixing. And then in the arts, it's the exact same thing. You know, novels and, and music and everything, like, you know, there are clearly creative leaps, but, you know, there's tr- just tremendous amounts of influence that come in from, from people who came before. And even if you think about, like, somebody with the creativity of a Beethoven, like, there was a lot of Beethoven in Mozart and Haydn and in the composers that came before. And so there's just tremendous amounts of, of, of remixing and combination. And so it's, it's, it's a little bit of an angel's dancing on the head of a pin question, which is like, if you can get, if you can get, you know, within, you know, I don't know, one, you know, point oh one percent of, of kind of world-beating, uh, you know, generational creativity intelligence, like you're, you're, you're probably all the way there. So I-- so emotionally, I wanna, like, hold out hope that there is, you know, still something special a-about human creativity, and I, I certainly believe that, and

Speaker 3:32 I, and I, and I very much wanna believe that. But, um, I don't know. When I use these things, I'm like, "Wow, they seem to be awfully smart and awfully creative." So I, I'm, I'm, I'm pretty convinced that they're gonna clear the bar.

Erik Torenberg 3:42 Yeah. The-- I think that seems to be a common theme in your analysis when, when people talk about the limitations of LLMs of, you know, can they do transfer learning or, or just learning in general? You seem to ask, can people do, do that? [laughs]

Speaker 3:54 Yes. Can people do these things? Well, it's like lateral thinking, right? So yeah. So it's like reasoning in or out of distribution, right? And so it's like, okay, I know a lot of people who are very good at reasoning inside distribution. How many people do I actually know who are good at reasoning outside of distribution and doing transfer learning? And, and the answer is, like, I know a handful. Like, I, I know a few, I know a few people, um, where whenever you ask them a question, you get an extremely original answer, and usually, that answer involves bringing in some idea from some adjacent space and basically being able to bridge domains. Um, and so, you know, you'll ask them a question about, I don't know, you know, finance, and they'll, they'll bring you an answer from psychology, or you ask them a question about psychology, and they'll bring you an answer from biology, right, or whatever it is. And so I, I know, you know, I know, I don't know, sitting here today, probably three. I probably know three people who can do that reliably-

Erik Torenberg 4:34 [laughs]

Speaker 4:35 ... out of the, you know... I've, I've got, you know, I've got ten thousand in my address book, um, and so three out of ten thousand-

Erik Torenberg 4:41 Yeah

Speaker 4:41 ... is not, is not that high a percentage. By, by the way, I find this very encouraging. Like, I, I... Yeah, immediately the mood in the room has gone completely to hell.

Erik Torenberg 4:49 [laughs]

Speaker 4:50 Um, I find this very encouraging. I find this very encouraging because look at what humanity has been able to build, right, despite all of our limitations, right? And, and look at all the creativity that we've been able to exhibit, and all the amazing art and all the amazing movies and all the amazing novels and all the amazing technical inventions and scientific breakthroughs. And so we, you know, we've been able to do, you know, everything we've been able to do with the limitations that we have. Um, and so, you know, I think that, you know, like, you know, do you need to get to the thing where you are one hundred percent positive it's actually doing, you know, original thinking? I don't think so. Uh, I, I, I think it'd be great if you did, and I think ultimately we'll probably conclude that that's what's happening. Um, but, like, I... It's, it's not necess- it's not even necessary for, like, just tremendous amounts of improvement.

Erik Torenberg 5:28 Ben, we were just celebrating some-

Ben Horowitz 5:29 Yeah

Erik Torenberg 5:30 ... some hip-hop legends at your Paid in Full, uh, event last week, and so you think a lot about creative genius. H-how do you think about this question?

Ben Horowitz 5:38 Yeah, I mean, I think that, uh, I, I agree with Marc that it's... Whatever it is, it's very useful, um, even if it isn't all the way that level. Uh, I think that y- you know, there's something about the actual, like, real-time human experience that humans are very into, um, at least in art, whereYou know, with the current state of the technology, kind of the, the pre-training doesn't have quite the, the right data to, to, to get to, um, what you really wanna see. Uh, but it, you know, it's pretty good. [chuckles]

Ben Horowitz 6:20 It is.

Erik Torenberg 6:21 Uh, we're-

Ben Horowitz 6:21 It's pretty amazing.

Speaker 6:21 How many-- H- so how many true conceptual in-- So Ben, Ben, Ben's been-

Ben Horowitz 6:25 Yeah.

Speaker 6:25 One of Ben's nonprofit activities is something-

Ben Horowitz 6:27 Yeah

Speaker 6:27 ...called the Paid in Full Foundation, which is h-honoring and actually providing essentially a pension, a pension for, uh, you know, for sort of, uh, you know, the great innovators in, in, in rap and hip-hop. Um, uh, and, and so he has-- he knows and has many of m-

Ben Horowitz 6:38 Yeah

Speaker 6:38 ...we were just at the event, and he, you know, has many of the k- kind of leading lights of that field for the last fifty years, you know, perform, and, and it's really fun to meet them and talk to them. Um, but, like, how many people in that entire field over the course of the last fifty years would you classify as, like, a true conceptual innovator?

Ben Horowitz 6:53 Yeah, well, y-y-you know, it's interesting. It, well, it de- it depends how broadly you define it, but, you know, th- there were several [chuckles] of them there last, you know, on Saturday. So Rakim, I-

Erik Torenberg 7:05 Dr. Dre, Nas

Ben Horowitz 7:05 ...I think we, uh, Rakim, you'd certainly put in that category. Dr. Dre, you'd certainly put in that category. George Clinton, you'd certainly put in that category. Um, you know, in a narrower sense, like, Kool G Rap certainly had a new idea. Um, but, y-you know, it de- it depends. Like, a fundamental kind of musical breakthrough, you'd probably just say, like, Rakim and George Clinton. Um, but-

Erik Torenberg 7:33 Are, are they excited?

Speaker 7:34 So, so, so two out of...

Ben Horowitz 7:36 Uh, uh, uh, well, I mean, those are-

Speaker 7:38 Yeah

Ben Horowitz 7:38 ...of the guys who were there. [chuckles]

Speaker 7:39 Oh, yeah, yeah. Okay.

Ben Horowitz 7:39 Yeah. But, uh, yeah, it's a tiny percentage.

Speaker 7:42 Yeah.

Ben Horowitz 7:42 Tiny, tiny, tiny, tiny, tiny.

Erik Torenberg 7:44 We had, uh, Jared, at the Fireside last night with Jared Leto. He was talking about how many people in Hollywood are, are really scared or against this, um, what, what's happening here. Is, is it-- What do you see in, you know, when you talk to the Dr. Dres, the Nas, the Kanyes? Are they excited? Are they using it? Are they-

Ben Horowitz 7:59 Yeah, no, I, the... So everybody, uh, who I speak to, uh, there are definitely people who are scared in music, but, like, there are a lot of people who are very, very interested in it. And particularly the hip-hop guys are interested because, um, it, it's almost like a replay, uh, of what they did, right? The-- They just took other music [chuckles] and they kind of built new music out of it, and I think that, you know, AI is, uh, a fantastic creative tool for them. It, like, way opens up the palette. And then for, y-y-you know, a, a lot of what hip-hop is, is it's, um, kind of telling a very specific story of a specific time and place, which, um, having intimate knowledge and being trained just on that thing [chuckles] is, is actually an advantage, as opposed to being a, like, a generally smart, uh, music model.

Erik Torenberg 8:50 Um, people al-also use the same logic of, "Hey, whatever is more intelligent will rule whatever is le-less intelligent." And, and Marc, you recently, uh-

Ben Horowitz 8:58 [laughs]

Erik Torenberg 8:58 [laughs]

Ben Horowitz 8:59 Not, not, not said by anybody who owns a cat.

Erik Torenberg 9:02 Yeah, exactly. [laughs]

Speaker 9:03 [laughs]

Erik Torenberg 9:04 M-Marc, you, you recently tweeted, "A supreme shape rotator can only rotate shapes, but a supreme word cell can rotate

Erik Torenberg 9:10 shape rotators." And, and also-

Ben Horowitz 9:12 Mm.

Erik Torenberg 9:12 [chuckles] Someone's clapping. [chuckles] "And also high IQ experts work for mid IQ generalists. What means?"

Speaker 9:20 Yeah. What, what means? Uh, yeah, so, and so, so the PhDs all work for MBAs, right?

Erik Torenberg 9:24 [laughs]

Speaker 9:24 So it's like, you know.

Erik Torenberg 9:26 [laughs]

Speaker 9:27 Okay. So yeah, like, so yeah. Well, I just, you know, just take it up a level. It's just like when you look at the world today, do you think we're being ruled by the smart ones?

Erik Torenberg 9:35 [laughs]

Speaker 9:36 Right? Like, is that, is that your big conclusion from, like, current events, current affairs? Uh, right? Like, okay, we put the geniuses in charge. Like-

Erik Torenberg 9:43 You mean Kamala and Trump aren't the best, the-

Speaker 9:45 Well, uh, let's not even be specific towards the US.

Erik Torenberg 9:47 Yeah.

Speaker 9:47 Let's just look all over the world. Um, uh, you know, uh, yeah. And so, like, it's just like, I, it, it, there, there's this thing. Uh, so y- I think two, two things are true. One is we, we probably all kind of underrate the importance of intelligence. Um, and actually there, there's a whole kind of backstory here of, like, intelligence actually turns out to be this, like, incredibly inflammatory, you know, kind of topic for lots of reasons o-over the last hundred years, um, uh, which, which we could, we could talk about in great detail. But, like, it, it, and, you know, and it's, and the, e-even the just very idea that, like, some people are smarter than other people, you know, just, like, really freaks people out, and peop-people don't like to talk about it. It's, we really struggle with that as a society. And so, like, and, and then it is true that intelligence is like, in humans, intelligence is correlated to almost every kind of positive life outcome, right? And so, um, intelligence, uh, generally in the social sciences, what they'll tell you is what they call fluid intelligence or, or the G factor or IQ is sort of, it's sort of zero point four correlated to basically everything. Um, and so it's, it has zero point four correlation to, like, educational outcomes and, like, ec- you know, professional outcomes and, and, you know, income, and by the way, also, like, life satisfaction and by the way, non-violence, you know, being able to solve problems without physical violence and so forth. And so, like, on the one hand, like, we probably all underrate intelligence. Um, on the other hand, the people who are in the fields that involve intelligence probably overrate intelligence. Um, and you, you might even, you might even coin a term like maybe like intelligence supremacist or something like that, where it's just like, oh, like, intelligence is very important, and so therefore, maybe it's, like, the most important thing or the only thing. But, but then you look at reality and you're like, "Okay, that's clearly not the case."

Ben Horowitz 11:15 Yeah, it's still zero, only zero point four, right? Yeah.

Speaker 11:17 Well, so to start with, it's only zero point four, and, and, you know, in the social sciences, zero point four is a giant correlation factor, right? Like, most, most things that where you can correlate, whether it's, you know, genes or observed behavior or whatever, anything in the social sciences, the correlations are much smaller than that. So zero point four is, is tiny, but it's still only zero point four. So even if you're like a full on-If you-- Even if you're, like, a full-on genetic determinist, and you're just like, you know, g-genetic IQ just, like, drives all these outcomes, like, it, it still doesn't explain, you know, point six o-of the correlation, and so that leaves it. But, but, but that's just on the individual level. Then you just look at the collective level. Well, you just look at the collective level, and it's, it's like a famous, famous observation is you take a bu-- you take a bunch of s-- you, you take any group of people, you put them in a mob, and the mob is dumber [laughs]

Erik Torenberg 11:57 [laughs]

Speaker 11:57 ...right, than the average. A-and, and you put a bunch of smart people in a mob, and they definitely turn dumber. Like, and, and you see that all the time, right? Um, uh, and so y-you put people in groups, and they, they, they behave very differently. And then you, you create, and then you create these, you know, questions around, like, who's in charge, whether who's in charge at a, at a company or who's in charge of a, of a country. And, like, it, it's-- Whatever the filtration process, it's clearly not-- it's not, it's, it's not, it's certainly

Speaker 12:20 not only on IQ, and it may not even be primarily on IQ. And so, so therefore, it's just like this assumption that you kind of hear in some of the AI circles, uh, which is like inevitably the smart, you know, kind of thing is gonna govern the dumb thing. Like, I, I just think that's, like, very easily, uh, it's just sort of very easily and obviously falsified. Like, intelligence isn't sufficient. And then you just, you just, you just convey it. You know, we're all in this room lucky enough to know a lot of smart people, and you, you just kind of observe smart people. And, like, some smart people, you know, really figure out how to have their stuff together and become very successful, and a lot of smart people never do. Um, and so there, there's, there, there must be... There obviously are, and there, and there, in fact, must be many other factors that have to do with success, um, and have to do with, like, who's in charge, uh, than just raw intelligence.

Erik Torenberg 13:00 I-it begs the q- the follow-up question of what are, what are some examples of what that might be, you know, skills sort of outside of intelligence, and more particularly, specifically, why couldn't AI systems, you know, l-learn them?

Speaker 13:11 Yeah. So Ben, like, what, what, what-- Other than intelligence, what, what in your experience determines, for example, success in leadership or in entrepreneurship or in, in, uh, solving complex problems or organizing people?

Ben Horowitz 13:20 [laughs] Yeah. Well, there, there, there are many things. Um, you know, like, a lot of it is, uh, being able to have a confrontation in the correct way. And there-- like, there's some intelligence in that, but a lot of it is just under- really understanding who you're talking to, you know, being able to interpret everything about how they're thinking about it and just kinda generally seeing decisions through the eyes of the people working in the company, not through your eyes, is a skill that y-you know, you develop by talking to people all the time, understanding what they're saying, so forth, these kinds of things. And it's just, um, y-you know, it's certainly not an IQ thing and not that... L-like, I, I could imagine [laughs] an AI training on any individual and, like, figuring it all out and knowing what to say and so forth. Um, but then you also need that integrated with, uh, you know, like, whatever the business ought to be doing. So you're not, you're not trying to do what's popular. You're trying to get people to do what's correct even if they don't like it, and, you know, that's a lot of management. So, uh, it's not a problem anybody's working on [laughs] AI currently, but maybe they will.

Speaker 14:36 It's some co-- Right.

Ben Horowitz 14:37 Yeah.

Speaker 14:37 Some combination of, like, courage-

Ben Horowitz 14:38 Yeah

Speaker 14:39 ...some combination of motivation, some combination of, um, of emotional, uh, understanding, theory of mind.

Ben Horowitz 14:45 Yeah. What, you know, what do people want, like, you know, married to, you know, what needs to be done? And then, like, how talented are they? Like, which ones can you afford? Like, if they jump out the window, it's fine. You know, which one's not fine? You know, this kind of thing. It's a... Th-there's a lot of, like, weird subtleties to it, uh, and it's very situational. I think the hardest thing about it, uh, and why management books are so bad is because it's situational. Um, you know, like your company, your product, your people, your org chart is very, very different than, you know, you know, here are the five steps to building a strategy. It's like, well, that's the most useless fucking thing I ever read because i-it has n-nothing to do with you.

Speaker 15:29 So one of the interesting things o-on this, like, on... This is... Right. The co- the concept of theory of mind is really important, right? So theory of mind is can you in your head model what's happening in the other person's head, right? And, and, and you would think that maybe that, you know, maybe obviously people who are smarter should be better at that. It turns out that that may not be true, and I'll... Th-the reason to believe that that's not true, which, which is as follows. So the, so the, the, the U.S. military is- was the early adopter and has continued to be sort of the leading adopter in, in, in U.S. society of, of act- of actually IQ testing, and they, uh, they, they, they basically launder it through something called the ASVAB, which is their, like, a vocational aptitude battery test, but it's basically an IQ... It's, it's essentially an IQ test. Um, and, uh, so they, they, they, they still use basically explicit IQ tests, and they, they slot people, uh, into different specialties and, and roles, uh, you know, in, in part w-with, uh, according to IQ, um, i-including into leadership roles. Um, and, and so they, they know what everybody's IQ is, and they, they kind of organize, organize around that. And one of the things that they found over the years is if the leader is more than one standard deviation of IQ away from the followers, it's a real problem.

Erik Torenberg 16:25 [laughs]

Speaker 16:26 Um, and, and, and that's true in both directions, right?

Erik Torenberg 16:29 Yeah.

Speaker 16:29 Um, if, if the leader is not smart enough to be able to, right, manage the, you know, is, is... To, to be able to mo- You know, for somebody who is less smart to model the mental behavior of somebody who's more smart is inher-

Speaker 16:40 of course, inherently very challenging and maybe impossible. But it turns out the reverse is also true, which is if the leader is two standard deviations above the norm of the organization that he's running, he also loses theory of mind, right? It, it, it's, it's v- actually very hard for very smart people to, um, to, to model, uh, the internal thought processes of even moderately smart people. Um, and so th-there, there's actually li- there's actually a real, there's actually a real need to have a, a level of connection there that's not just... Right? A-and therefore, by inference, if you had a person or a, or a machine that had a, you know, a thousand IQ or something like it, it may just be... It would be so alien. Its understanding of reality would be so alien to the people or the things that it was managing that it wouldn't, it wouldn't even be able to connect in any sort of, any sort of realistic way. So again, this is a very good argument that, like, it... Yeah, this is... The world is gonna be far from organized by IQ for, yeah, [laughs] for centuries to come.

Ben Horowitz 17:27 Yeah, and Zuckerberg had a great line, which is, "Intelligence is not life." [laughs] And life, l-life has a lot of dimensionality to it that is independent of intelligence. I think that, you know, if you spend all your time working on intelligence, you lose track of that.

Erik Torenberg 17:43 We, we sometimes say [clears throat] about, uh, some specific people that they're too smart to properly model or, or, or, you know, um, too-- they sort of, sort of assume too much rationality on other people, or they just o-overthink things or over-rationalize them. Um, yeah, just to, to your point that it's, it's not everything.

Ben Horowitz 17:59 Yeah. Yeah. Yeah. [chuckles] People often... People seldom do what's in their best interest, [chuckles] I should say.

Erik Torenberg 18:04 Yeah.

Ben Horowitz 18:04 Yeah, yeah.

Speaker 18:05 You know, I also suspect, this kind of gets more into the biology side of things. I, you know, there's more and more scientific evidence that basically also that, like, human, human cognition, h-human cognition or human, I don't, whatever you wanna call it, self-awareness, information processing, decision-making sort of experience, um, uh, is, is not purely a brain. Like, the basically d- the do- the sort of mind, famous mind-body dualism is just not correct. Like, a-and again, this is an argument against sort of IQ supremacism or intelligence supremacism is it's not... You know, we, we-- Human beings didn't experience existence just through the rational thought, uh, and, and, and specifically not through just the rational thought of the brain, but rather it, it's a whole body experience, right? And there's, there's, there's aspects of our nervous system, and there's aspects of everything from our gut biome to, you know, to, to, you know, to, to smells, you know, to the olfactory senses and, and, um, you know, and hormones and, like, all, all kinds of, like, biochemical kind of aspects to life. Um, like, I su- If you just are gonna track the research, I suspect we're gonna find is human cognition is a full body experience, uh, much, much more than, much more than people thought. Um, and, and so therefore to actually... To-- And this, you know, this is like a... And this is, you know, one of the kind of big fundamental challenges in the AI f- AI field right now, which is, you know, the form of AI that we have working is, is the f- is the fully mind-body dual version of it, which is it's just a disembodied, you know, like a disembodied brain. You know, the robotics revolution for sure is coming when that happens, when we put AI in, in physical objects that move around the world. You know, you're, you're gonna be able to get closer to having that kind of, you know, inte-integrated intellectual, physical, you know, experience. You're gonna have sensors in the robots. They're gonna be able to, you know, gather a lot more, you know, real world data. And so maybe you can start to actually think about, you know, synthesizing-

Erik Torenberg 19:37 Interesting

Speaker 19:37 ... you know, a more advanced model of cognition. And, and, you know, maybe we're gonna actually discover more both about how the human version of that works and also how the machine version of that works. But it-it-it's just, to me at least reading the research, like, that all those ideas feel very nascent, and we have a lot of work to do to try to figure that out.

Erik Torenberg 19:51 Do you have a sense for h-h-how they are, LLMs are at theory of mind today? Um, or do you have a sense where the limitations are? You, you like to talk to them a lot. Are there any particular things that are particularly surprising to you as you do?

Speaker 20:03 Yeah. I would say generally they're really good. Um, yeah. And so, like, one of the, one, one of the more... I, I find one of the more fascinating ways, you know, to, to, to work with language models is actually have them create personas, um, and, uh, and then, you know, basically have... Well,

Speaker 20:14 actually, so the way... I like, I like, you know, I like basically, I like Socratic dialogues. I like when things are argued out in, like, a Socratic dialogue. And so you, you know, tell a, tell any advanced LLM today to create a Soc-Socratic dialogue, and it'll either make up the personas. You can tell what it is. It does a good job. It has this very, very annoying property, which is it wants everybody to be happy.

Erik Torenberg 20:30 [laughs]

Speaker 20:31 Um, and so it wants all of its personas to agree. Um, and so by default, it will have a, uh, it will have a briefly interesting discussion, and then it will sort of figure out-

Erik Torenberg 20:39 [laughs]

Speaker 20:40 ... you know, basically, it... Like you're watching, I don't know, a PBS special or something. It'll, it'll kind of figure out how to bring everybody in agreement.

Erik Torenberg 20:44 Mm-hmm.

Speaker 20:44 Everybody's happy at the end of the discussion. And of course, I fucking hate that.

Erik Torenberg 20:47 [laughs]

Speaker 20:47 Like, it drives me nuts. I don't want that.

Erik Torenberg 20:49 [laughs]

Speaker 20:50 So instead, I, I tell it. I'm like, "Make the conversation more tense," right?

Erik Torenberg 20:54 [laughs]

Speaker 20:54 "And, like, fraught with, like, anger and, like, you know, people sh- you know, gonna get, like, increasingly upset-

Erik Torenberg 21:00 [laughs]

Speaker 21:00 ... throughout the conversation." And then it starts to get really interesting. Um, and then I, and then I tell it, you know, "Bring it, you know, use-- introduce a lot more cursing."

Erik Torenberg 21:07 [laughs]

Speaker 21:07 Um, you know, really have them go at it.

Erik Torenberg 21:09 [laughs]

Speaker 21:09 Like, all the gloves come off. They're going for full, full, you know, you know, reputational destruction of each other.

Erik Torenberg 21:13 And you do a lot of these skits. You're like-

Speaker 21:14 Yeah, skits. And then I, I get carried away, and then I'm like, it turns out they're all, like, secret ninjas, and then they all start fighting.

Erik Torenberg 21:19 [laughs]

Speaker 21:19 And you've got Einstein, you know, you know, you know, hi-hitting, you know, Niels Bohr with nunch-nunchucks. And it, it...

Erik Torenberg 21:23 [laughs]

Speaker 21:23 And by the way, it's happy to do that too.

Erik Torenberg 21:25 [laughs]

Speaker 21:26 Um, so you do have to, you have to, you have to control yourself. But it is, it is very good at theor-at theory of mind. And then I'll, I'll give you another example. There's a, there's a startup actually in the UK, uh, in, uh, in, in the world of politics. And, and what, what they've found is that, um, they found that language models now are, are good enough, so specifically for, for politics, which is sort of a sub-subcategory where th-this, this idea matters. Um, so, you know, in politics, people do focus group. You do focus groups of voters all the time. And, and by the way, many businesses also do that. Um, you know, so you get a bunch of people together from different backgrounds in a room, and you kind of guide them through a discussion and try to get their, their points of view on things. And, and focus groups are often surprising. Like, politicians who... If you talk to politicians who do focus groups, they're often surprising. They're often surprised by the things that they thought voters cared about is actually not the things that voters care about, and so you can actually learn a lot by doing this. But focus groups are very expensive to run, and then there's a long lag time 'cause they have to be actually physically organized, and you have to recruit people and vet people and, and, and so forth. Um, and so it turns out that the, the, the, the state-of-the-art l-models now are, are good enough at this that they can actually... They can, they can correctly, um, accurately reproduce a focus group of real people, um, in-inside the model. Um, so, so, so they're good enough to clear that bar. In other words, you, you can basically have a focus group actually happening in the model where you create personas in the model, and then it actually accurately represents, you know, a college student from, you know, Kentucky as contrasted to a housewife from Tennessee as contrasted to a, you know, whatever, whatever. You, you just, like, specify this. And so, you know, they're good enough to clear, they're good enough to clear that bar, and, you know, we'll, we'll see how far they get.

Erik Torenberg 22:50 I wanna segue to the bubble conversation. Uh, Amin and G2, Jensen and Matt spoke about the enormous scale of physical infrastructure being built out. AI CapEx is

Erik Torenberg 23:02 1% of GDP. H-how should we understand and think about this bubble question?

Ben Horowitz 23:06 [chuckles] Well, I think the fact that it's a question means we're not in a bubble. That's the first thing to understand. I mean, bu- a bubble is a psychological phenomenon, um, as much as anything, and in order to get to a bubble, everybody has to believe it's not a bubble. That, that's sort of the, the, the core mechanic of it. And they, you know, we call that capitulation. Everybody just gives up, like, "Okay, I'm not gonna short these stocks anymore. I'm tired of losing all my money. I'm gonna go long." Uh, and we saw that actually in y-you know, and a little bit of question, like, really, what was the tech bubble? Um, but in the kind of dot-com era-Right as the prices went through the roof, Warren Buffett started inviting-- investing in tech. So like... And he swore he would never invest in tech because he didn't understand it. And so if he capitulated, nobody was saying it was a bubble when it became like a quote-unquote bubble. Now, if you look at that phenomenon, um, the internet clearly was not a bubble. Uh, you know, it was a real thing. It was in the short term, there was a kind of price dislocation that happened because, uh, the, the market, um, you know, there were just not enough people on the network to make those products go at the time. Uh, and then the prices kind of outran the market. You know, in AI, it's much harder to see that because there's so much demand in the short term, right [chuckles] ? Like, we don't have a demand problem right now, and like the idea that we're gonna have a demand problem five years from now, to me, seems quite absurd. Uh, you know, could there be like weird bottlenecks that, that appear? You know, like we just-- at some point, we just don't have enough cooling or something like that. You, you know, maybe. But like, like right now, if you look at demand and supply and what's going on and multiples, uh, against growth, it doesn't look like a bubble at all to me. Um, but [chuckles] I don't know. Do you think it's a bubble, Marc?

Speaker 25:08 Yeah. Look, I, I-- I would just say this. Yeah, like nobody know-- so nobody knows in the sense of like the experts. Like if you're talking to anybody at like a hedge fund or a bank or whatever, like they definitely don't know. Um, uh, generally, the CEOs don't know. So it, the, the, these-

Ben Horowitz 25:20 But by the way, a lot of VCs don't know. They just get upset. Like VCs get like emotionally upset when you guys have higher valuations.

Speaker 25:27 [chuckles]

Ben Horowitz 25:28 Like it, it makes them like, like angry.

Speaker 25:30 [chuckles]

Ben Horowitz 25:30 Uh, and, you know, and I, I get it all the time, and I'm like, "What are you mad about?" Like, "The shit is working, man. Be happy. Come on."

Speaker 25:38 [chuckles]

Ben Horowitz 25:39 But so, so like there's a lot of emotion around like people wanting it to be, be a bubble.

Speaker 25:45 Yeah. No- nothing's worse than passing on a deal than having the company become a great success. Like that's just, I was just saying, it puts you-

Ben Horowitz 25:50 Yeah. It's like, "That, that valuation's outrageous." [chuckles]

Speaker 25:52 You, you could be furious about that for thirty years in our business.

Ben Horowitz 25:54 [chuckles]

Speaker 25:54 It's, it's, it's amazing. Um, and you can find... Yeah, you come up with all kinds of reasons to cope and, and explain why it wasn't your mistake. But it's, you know, it's the world, it's the world that's wrong, not me, right? So there, there's a lot of that. Yeah. Uh, yeah. So I, I just-- I would just, I would just say, like I would always say, bring, bring the conversation back to ground truth fundamentals. And the, the two big ground truth fundamentals are, number one, does the technology actually work? Um, you know, can it deliver on its promise? And then number two is, are customers paying for it? Um, and if those, if those two things are true, then it's very hard to, it's very hard to, uh... Like as long as those two things stay grounded, um, you know, gener- generally, generally things are gonna, are I think are gonna be on track.

Erik Torenberg 26:31 Yeah. When Gavin was up here with DG, he said ChatGPT was a Pearl Harbor moment for Google, the moment when the giant wakes up. When, when we look at history and, and platform shifts, what determine whether the incumbent actually wins the next wave versus, versus new entrants? Or how should we think about that in, in AI?

Ben Horowitz 26:47 [sighs] Well, you know, reacting to it is important, um, but that doesn't mean... Like, like I-- It's a Pearl Harbor moment, I think. Uh, [chuckles] Google got their head out of their ass. That was the sound of it. [laughs]

Erik Torenberg 27:05 [laughs]

Speaker 27:06 [laughs]

Ben Horowitz 27:07 Uh, so you, you know, they're not gonna get completely run over, but nonetheless, like I, I don't think OpenAI is going away, so like they, they definitely let that happen. Um, yeah, some of it is speed, and then just look, it's execution over a long period of time. And, uh, you know, some of these very large companies, to varying degrees, have lost their ability to execute. And so if you're talking about a brand-new platform and you're talking about, you know, kind of building for a long time, it's like, you know, Microsoft got caught with their pants down on Google. Um, Microsoft's still like very strong, but they missed that whole opportunity. They also missed the opportunity... You know, Apple was nothing, and Microsoft fully believed that they were gonna own mobile computing. They completely missed that one. But they were still so big from their Windows

Ben Horowitz 27:58 monopoly, they could build into other things. So, you know, I think generally the new companies have won the new markets. Uh, and that doesn't mean the big company-- The biggest companies, the biggest monopolies from the prior generation just last a long time, is the way I would look at it.

Speaker 28:16 Yeah. I, I also think we don't quite know... Like it's all happened so fast. We, we actually don't... I think we don't yet know the shape and form of the ultimate products.

Ben Horowitz 28:24 Yeah.

Speaker 28:25 Um, right? And so, and so like... 'Cause it's, it's tempting, and this is kind of what, what always happens, it's kinda, it's kinda tempting to look, and I'm not saying what, you know, it's what these guys did on stage, but it's kinda tempting to look... You sometimes you hear the kinda reductive version of this, which is basically it's like, oh, there's either gonna be a chatbot or a search engine, right? The, the competition is between a chatbot and a search engine, and the, the problem Google has i- is the classic problem of disru- you know, disruption. Are you gonna disrupt the ten blue links model and swap in a, a, you know, ad, you know, sort of, uh, AI answers and, you know, potentially disrupt the advertising model? And then the problem OpenAI has is they have the, the full, you know, the full chat product, but, you know, they don't have the advertising yet, and they don't have the distribution, Google-scale distribution. And so, you know, you kinda say, okay, that's a fairly, it's a fairly cla- Like that'd be straight out of a, like, you know, the innovator's dilemma, you know, business textbook. Like this is just a very clear, you know, one, one versus one, you know, kinda dynamic. But that assumes that, you know, or the mistake that you can make in thinking that way is that assumes that the forms of the product in five, ten, fifteen, twenty years that, that, that are gonna be the, the main things that people use are gonna be either a search engine or a chatbot, right? Um, and, and, you know, the, I just... There's, you know, there's just obvious historical analogies. One just obvious historical analogy is, you know, the, the, the personal computer from sort of invention in 1975 through to, you know, basically 1992, you know, was, was a te- was a text prompt system, right? Um, you know, and at the time, by the way, an interactive text prompt was a big advance over the previous generation of like punch card systems, time-sharing systems. Uh, and then, you know, it was, you know, 1992, so it was what? Seven- seventeen years in, you know, the whole industry took a left turn into GUIs and never looked back, you know? And, and then by the way, you know, five years after that, the industry took a left turn into we-web browsers and never looked back, right? And so the very shape and form and nature of the user experience and how it, and how it fits into our lives, uh, you know, is, is, is I think still unformed. And so like, and, and, you know, look, I'm, I'm sure there will be chatbots twenty years from now, but I, I'm, I'm pretty confident that, um, you know, both the current chatbot companies and many new companies are gonna figure out many kinds of user experiences that are radically different that we don't, we don't even know yet.Yeah, and by, and by the way, that's one of the things, of course, that keeps the tech industry fun, which is it, you know, it's, especially on the, especially on the software side, you know, it's, it's not, it's not, it's not obvious what the shape and form of the products are, and there's just... I think there's just tremendous headroom for invention.

Erik Torenberg 30:32 As, as you're coaching en-entrepreneurs and the entrepreneurs in this room, what, what else feels different a-about this era or, or, or sort of other advice that you find yourself disp- whether it's around, uh, sort of the talent wars that are going on or oth-other aspects that feel unique to this era, what, what other advice do you wanna be leaving our en-entrepreneurs with?

Ben Horowitz 30:50 That's unique to this era. Well, like I, I, I actually think you said the right thing, which is this is a unique era. And so trying to

Ben Horowitz 31:01 learn the organizational design lessons of the past or trying to learn, um, kind of too much from the last generation is-- can be deceptive because things really are different. Like the way these, you know, the way your companies are getting built is, is quite different in, in many aspects. Uh, and, you know, the types of, [chuckles] you know, what the... Just like our observa-observation on like PhD AI researchers is just very different than like a traditional, um, engineer, full stack engineer or something like that. So, you know, I, I think you do have to think through a lot of things from first principles, uh, because it is different. Uh, and like, you know, ob-observing from the outside, it's really different.

Erik Torenberg 31:53 Yeah.

Speaker 31:53 Yeah, and I would just offer, look, I, I do think things are gonna change. So I already talked about I think the shape and form of products is gonna change. Um, uh, and so like I think there's still a lot of creativity there. I also think, and I, I, I was supposed to say, I think that, um, like in a, in a world of supply and demand, the thing that creates gluts is shortages, um, right? So like wh-when something becomes too scarce, there becomes a massive economic incentive to figure out how to unlock new supply. And so the, the, the, the current generation of AI companies are really struggling with, uh, particularly shortages of, of the really talented AI, AI researchers and engineers, and then they're really, you know, uh, uh, challenged with a shortage of, of infrastructure capacity chips and, and data centers and power. Um, I, I, I don't wanna call timing on this. There will come a time when both of those things become gluts. Um, a-and so I don't know, I don't know that we can plan for that, um, although I, I would just say the following. Number one, um, the, the, the researcher engineer side of things, it is striking, it is striking the degree to which, um, there are excellent, you know, outstanding models coming out of China now, um, and, you know, and, and, and, and mult- from multiple companies and, you know, specifically, you know, uh, DeepSeek and, and, uh, Qwen and Kimi. Um, it is striking how the teams that are making those are not, you know, the name brand. You know, for the most part, these are not like the name brand people with their names on all the papers. Um, a-a-and so like Ch-China is successfully decoding how to like basically take young people and train them up in the field.

Ben Horowitz 33:10 Well, and xAI to a large extent too.

Speaker 33:13 Yeah. And, yeah. And so I, I think that, I think there's gonna be... A-a-and look, it makes sense up until... It, it, it makes sense that for a while it's gonna be this super esoteric skill set, and people are gonna pay through the nose for it. But like, you know, there's no question the information is, right, being transferred into the environment. People are learning how to do this. Um, you know, college kids are figuring it out. Um, and so, um, you know, there's, there's... A-and I don't know that there's ever gonna be a talent glut per se, but like I, I think for sure there's gonna, there's gonna be a lot more people in the future who of course know, know how to build these things. Um, and then, and then by the way, also of course, you know, AI building AI, right? So the, the, the, the, the tools themselves are gonna be better, better at, at, at contributing to that. And so, a-a-and I think that, I think this is good 'cause I think that, you know, the current level of, of shortage of, of engineers and researchers is, is, is too constraining. And then, and then on the chip side, I don't, I don't wanna... I'm not a chip guy and I don't wanna call, call it specifically, but like it's never been the case, it's never been the case in the chip industry that there's ever, you know... E-every, every shortage in the chip industry has always resulted in a glut, uh, because the, the, the profit, the profit pool of a shortage, the, the margins get too big. The incentive, uh, for other people to come in and figure out how to commoditize the function get too big. And so, you

Speaker 34:14 know, N-Nvidia has like, you know, the best position probably anybody's ever had in chips, but notwithstanding that, I, I find it hard to believe that there's gonna be this level of pressure on infrastructure in five years.

Ben Horowitz 34:22 Yeah, and e-even if the bottleneck within the infrastructure moves, so if, if it becomes power, if it becomes cooling or, or, or anything else, then you'll have a chip glut for sure, yeah.

Speaker 34:32 Right. So, so I, I think over the... I w- I would just say this. It's likely the challenges that we, that we all have in five years from now are gonna be different challenges.

Erik Torenberg 34:39 Yeah.

Ben Horowitz 34:39 Yeah. Yeah, yeah. Like don't, [chuckles] don't... Definitely, this industry of all industries, don't look at us as static. Like, you know, the positions, uh, could change very, very fast.

Erik Torenberg 34:51 L-let's actually c-close on, on more of this macro note. Marc, Marc, you mentioned China. Last month we were in DC, and one, one of the big questions the senator has is how should we make sense of sort of the state of the AI race vis-à-vis China? Do you wanna share just the, the high level, um, summary of what, what you shared with them?

Speaker 35:07 Yeah. So my sense of things, and I, and I think the current, I think the current, if you just observe currently, specifically like DeepSeek, Qwen, and Kimi and, and these models coming out of China, I... My, my sense basically is like I would say the U.S.-... ju- specifically in the West generally, but, you know, more and more specifically the U.S., it... Like, the conceptual innovations are, are, you know, have, have been, you know, coming out, coming out of, coming out of the U.S., coming out of the West, you know, kind of the, the, the big kind of conceptual breakthroughs. Um, uh, China i-is extremely good at picking up ideas and implementing them and, and scaling them and, and commoditizing them and, and, you know, that's, the-they do that obviously throughout the manufacturing world. Um, and, and they're doing it, but now very, I think, successfully, uh, sort of i-in AI. Um, and so I would say they're, they're running, they're running the catch-up game, like, really well. Um, you know, and then there's, there's o- sort of always this question of, like, how much of that is, like, being done, let's just say, like, or, or authentically, um, uh, you know, through hard work, um, and smart people, and then how much is being done with maybe a little bit of help. Um, maybe a little USB stick, uh, in the middle of the night, uh, you know, kind of help.

Erik Torenberg 36:06 [laughs]

Speaker 36:08 Um, so, uh, you know, there's always a little bit of a question. But, like, either, either way, uh, you know, they're, they're, they're doing a great job. Uh, o- obviously they, they aspire to, you know, more than that, um, and there are many very smart and creative people in China. And so, you know, it'll be interesting now to see, you know, the level to which the conceptual breakthroughs start to come from there and whether they, whether they pull ahead. Um, and so s- but, like, I would say, like, w-what we tell people in Washington is, like, look, this, this is a f- this is now a f- this is a full-on race, it's a foot race, it's a game of inches. Like, we're not gonna have a five-year lead. We're gonna have, like, maybe a six-month lead. Like, we have to run fast. We have to win. Like, we, we have to, we have to do this. We, we can't... And then we can't put constraints on our companies that the, the, the Chinese, uh, government isn't putting on their own companies. And so, um, or, you know, we'll just lose. And, you know, do, do you really wa- do you really wanna wake up in the morning and live in a world, you know, really controlled and run by Chinese AI? Most of us would say, no, we, we don't wanna live in that world. Um, and so, um, so, s-so that-- so there's that. And, and I would say I feel moderately good about that, just 'cause I think the... I think we're, we're really good at software. Um, y-you know, the minute this goes into, you know, embodied AI in the form of robotics, I think things get a lot scarier

Speaker 37:10 and pro- you know, this is the thing I'm now spending time in DC trying to really educate people on, which is, you know, the Chi- Because the U.S. and the West have chosen to deindustrialize to the extent that we have over the last forty years, um, you know, th- China, China specifically now has this giant industrial ecosystem for building elec- you know, sort of mechanical, electri-electrical, and, um, uh, and, uh, semiconductor and now software, you know, devices of all kinds, including phones and drones and cars, um, a-and robots. Um, and so, uh, you know, there's gonna be a phase two to the A-AI revolution. It's gonna be robotics. It's gonna happen, you know, pretty quickly here, I think. Um, and when it does, like, even if the U.S. stays ahead in software, like, the robot's gotta get built, and that's not an easy thing. And it's not just, like, a company that does that, it's gotta be an entire ecosystem. Um, and it's, you know, it's gonna be... You know, it's gonna... You know, like, I mean, the, you know, the car industry was not three car companies. It was thousands and thousands of, of, of component suppliers building all the parts. And it's been the same thing for airplanes and the same thing for computers and everything else. Um, it's gonna be the same thing for robotics. Um, and, you know, by default, sitting here today, that's all [chuckles] gonna happen in China. And so even if they never quite catch us in software, they might just, like, lap us in hardware, and, and, and, and that'll be that. Um, you know, the, the good news is I, I think there's a growing awareness in... There, there's a growing awareness, I would say, across the politics- political spectrum in the U.S. that, like, deindustrialization went too far, uh, and there's a growing desire to kind of figure out how to reverse that. Um, and, um, you know, I'd say I'm guardedly optimistic that we'll be making progress on that, but I think there's a lot of work to be done.

Erik Torenberg 38:37 On that call to arms, uh, let's wrap. Uh, thank you, Marc and Ben. To, to wrap up, I'd like to welcome Jennifer-

Speaker 38:43 All right, thank you.

Erik Torenberg 38:43 Sorry about that.

Speaker 38:44 Thank you, everybody. [audience applauding]

Speaker 39:09 [upbeat music]