No PriorsNo Priors Ep. 95 | Best of 2024
EVERY SPOKEN WORD
55 min read · 11,199 words- 0:00 – 0:15
Introduction
- SGSarah Guo
(instrumental music plays) Hi, No Priors listeners. I hope it's been an amazing 2024 for you all. Looking back on this year, we wanted to bring you highlights from some of our favorite conversations. First
- 0:15 – 4:00
Jensen Huang on building at data-center scale
- SGSarah Guo
up, we have a clip with the one and only Jensen Huang, CEO of NVIDIA, the company powering the AI revolution. Since our 2023 No Priors chat with Jensen, NVIDIA's tripled in stock price, adding almost 100 billion of value each month of 2024, and entering the $3 trillion club. More recently, Jensen shared his perspective again with us, this time on why NVIDIA's no longer a chip company, but a data center ecosystem. Here's our conversation with Jensen. NVIDIA has moved into larger and larger, let's say, like unit of support for customers.
- JHJensen Huang
Mm-hmm.
- SGSarah Guo
So I think about it going from single chip to, you know-
- JHJensen Huang
Yeah.
- SGSarah Guo
... server to rack-
- JHJensen Huang
Yeah.
- SGSarah Guo
... NVL72. How do you think about that progression? Like what, what's next?
- JHJensen Huang
Uh-huh. That's great.
- SGSarah Guo
Like, could NVIDIA do a full data center?
- JHJensen Huang
Uh, in fact, we build full data centers. The way that we build everything, unless you're building... If you're developing software, you need the computer in its full manifestation. Um, we don't, we don't build PowerPoint slides and ship the chips and... We build a whole data center. And until we get the whole data center built up, how do you know the software works? Until you get the whole data center built up, how do you know your, you know, your fabric works and all the things that you expected the efficiencies to be? How do you know it's gonna really work at scale? And, and that's the reason why, that's the reason why it's not unusual to see somebody's actual performance be dramatically lower than their peak performance as shown in PowerPoint slides.
- SGSarah Guo
Mm-hmm.
- JHJensen Huang
And, and, and, and it's... Computing is just not used to... It's not what it used to be. You know, I say that the new unit of computing is the data center. That's, to us-
- SGSarah Guo
So that's what you have to deliver.
- JHJensen Huang
That's what we build. Now, we build a whole thing like that and then we... For every single thing that we... Every combination, uh, air cooled, x86, liquid cooled, grace, ethernet, InfiniBand, NVLink, no NVLink. You know what I'm saying? We build every single configuration. We have five super computers in our company today. Next year we're gonna build easily five more. So if you're serious about software, you build your own computers. If you're serious about software, then you're gonna build your whole computer, and we build it all at scale. This is the part that- Mm-hmm. ... that is really interesting. We build it at scale and we build it, uh, very vertically integrated. We optimize it, um, full stack, end-to-end, and then we disaggregate everything and we sell it in parts. That's the part that is completely, utterly remarkable about what we do. Mm-hmm. The complexity of, of that is just insane. And the reason for that is we wanna be able to graft our infrastructure into GCP, AWS, Azure, OCI. All of their control planes, security planes are all different, and all of the way they think about their cluster sizing, all different. And, um, uh, but yet we make it possible for them to all accommodate NVIDIA's architecture so that CUDA could be everywhere. That's really, really, in the end, the, the singular thought, you know, that we would like to have a computing platform that developers could use that's largely consistent, modulo, you know, 10% here and there because people's infrastructure are slightly optimized differently and... Modulo 10% here and there but, but everything they, they build will run everywhere. This is kind of the one of the principles of software that should never be g- given up. And it... And, and we, we, we protect it quite dearly. Uh, it makes it possible for our software engineers to build once, run everywhere. And, and that's because we recognize, uh, that the investment of software is the most expensive investment and it's easy to test. Uh, look at the size of the whole hardware industry and then look at the size of the world's industries. It's $100 trillion on top of this $1 trillion industry and that tells you something. The software that you build, you have to, you know, you basically maintain for as long as you shall live.
- SGSarah Guo
We, of
- 4:00 – 7:14
Andrej Karpathy on the AI exo-cortex, model control, and a shift to smaller models
- SGSarah Guo
course, have to mention our conversation with the lovely Andrej Karpathy, where we dig into the future of AI as an exocortex, an extension of human cognition. Andrej, who's been a key figure in AI development from OpenAI to Tesla to the education of us all, shares a provocative perspective on ownership and access to AI models, and also makes a case for why future models might be much smaller than we think. If we're talking about a, um, exocortex, that feels like a pretty fundamentally, um, important thing to democratize access to. How do you think, like, the current market structure of what's happening in LM research... You know, there's a small number of large labs that actually have a shot at the next generation progressing training. Like, how does that translate to what people have access to in the future?
- AKAndrej Karpathy
So what you're kind of alluding to maybe is the state of the ecosystem, right? So we have kind of, like, an oligopoly of a few closed platforms and then we have an open platform that is kind of, like, uh, behind. So like Meta, LLaMA, et cetera.
- SGSarah Guo
Mm-hmm.
- AKAndrej Karpathy
And this is kind of like mirroring the open source, uh, kind of ecosystem. I do think that when this stuff starts to... When we start to think of it as like an exocortex... Uh, so there's the, there's a saying in crypto, which is like, "Not your keys, not your, uh, not your tokens."
- SGSarah Guo
Not yours. Yeah.
- AKAndrej Karpathy
Like, is it the case that if it's like not your weights, not your brain?
- JHJensen Huang
That's interesting because a company is effectively controlling your exocortex and they're a form-
- AKAndrej Karpathy
Yeah.
- JHJensen Huang
... big part of your-
- AKAndrej Karpathy
It starts to feel kind of invasive. If this is my exocortex...
- SGSarah Guo
I think people will care much more about ownership, yes.
- AKAndrej Karpathy
Like you're... Yeah, you're... You realize you're renting your brain. Like, it seems-
- SGSarah Guo
Okay, but-
- AKAndrej Karpathy
... strange to rent your brain.
- SGSarah Guo
... the thought experiment is like are you willing to give up ownership and control to rent a better brain? Because I am.
- AKAndrej Karpathy
Yeah.
- SGSarah Guo
You know?
- AKAndrej Karpathy
So I think that's the trade-off, I think. We'll see how that works, but maybe it's possible to, like, by default use the closed versions because they're amazing but you have a fallback in various scenarios. And I think that's kind of like the way things are shaping up today even, right? Like, um, when APIs go down on some of the closed source providers, people start to implement fallbacks to, like, the open ecosystems, for example, that they fully control and they're en- en- uh, they feel empowered by that, right? So, so maybe that's just the extension it will look like for the brain, is you fall back on the open source stuff, um, s- should anything happen but most of the time you actually...
- SGSarah Guo
So it's quite important that the open source stuff continues to progress.
- AKAndrej Karpathy
I think so.
- SGSarah Guo
Yes.
- AKAndrej Karpathy
100%.And this is not, like, an obvious point or something that people maybe agree on right now. But I think 100%.
- EGElad Gil
I- I guess one thing I've been wondering about a little bit is, um, what is the smallest performant model that you can get to in some sense? Either in parameter size or however you want to think about it. And so I'm a little bit curious about your view-
- AKAndrej Karpathy
Yeah.
- EGElad Gil
... because you've thought a lot about the, uh, distillation, small models, you know.
- AKAndrej Karpathy
Yeah, yeah. I think it can be surprisingly small. And I do think that the current models are wasting a ton of capacity remembering stuff that doesn't matter. Like, they remember SHA hashes, they remember, like, the ancient...
- SGSarah Guo
'Cause the dataset is not curated at the best yet.
- AKAndrej Karpathy
Yeah, exactly.
- SGSarah Guo
Yeah.
- AKAndrej Karpathy
Like, and I think this will go away, and I think we just need to get to the cognitive core. And I think the cognitive core can be extremely small, and it's just this thing that thinks. And if it needs to look up information, it knows how to use different tools.
- 7:14 – 11:17
Bret Taylor on the agentic future of business interactions
- SGSarah Guo
with Bret Taylor, OpenAI board member and founder of Ciara, painted a really different picture of how we interact with businesses in the future. Here's a clip of Bret explaining company agents and why the website is going to take a backseat.
The other category, which is the area that my company Ciara works in, is what I call company agents. Um, and it's really, uh, less simply about automation or autonomy, but in this world of conversational AI, how does your company exist digitally? I always use the metaphor of if it were 1995, you know, if you existed digitally, it meant having a website and being in Yahoo! directory, right? In 2025, existing digitally will probably mean having a branded AI agent that your customers can interact with to do everything that they can do on your website. Um, whether it's, you know, asking about your products and services, doing commerce, doing customer service. Um, that domain, I think, is shovel-ready right now with current technology because again, like the persona-based agents, it's not boiling the proverbial ocean technically. You know, you have well-defined processes for your customer experience, well-defined systems that are your systems of record. And it's really about saying in this world where we've gone from websites to apps to now conversational experiences, what is the conversational experience you want around your brand? And it doesn't mean it's perfect or it's easy, otherwise we wouldn't have started a company around it. But it's at least well defined. Um, and I think that right now in AI, if you're working on artificial general intelligence, your version of agent probably means something different and that's okay. Um, that's just a different problem to be solved. But I think, you know, particularly in, you know, the areas that, that Ciara works and a lot of the companies that you all have invested in is it's saying, you know, there's some shovel-ready opportunities right now with existing technology and I absolutely think there are.
Can you describe the, um, like, shoveling cycle of building a company agent? Like, what is the gap between research and reality? Like, how do you... um, what do you invest in as an engineering team? Like, how do you understand the scope of different customer environments? Just like what are the sort of vectors of investment here?
I think-
- AKAndrej Karpathy
And, and maybe, sorry to interrupt.
- SGSarah Guo
Yeah, please.
- AKAndrej Karpathy
As a starting point, it may even be worth also defining like what are the products that Ciara provides today for its customers and then where do you want that to go and then maybe we can feed that back into, like, what are the components of that? 'Cause I think, uh, obviously you folks are really emerging as a leader in your vertical but it'd be great just for our broader audience to understand what you focus on.
- SGSarah Guo
Yeah, sure. I'll just give a couple of examples to make it concrete. So if you buy a new Sonos speaker or you're having technical issues with your speaker, you get the dreaded flashing orange light, you'll now chat with the Sonos AI, which is powered by Ciara to help you onboard, help you debug, whether it's a hardware issue, a Wi-Fi issue, um, things like that. If you're a SiriusXM subscriber, their AI agent is named Harmony, which I think is a delightful name.
- AKAndrej Karpathy
It's good to hear. Yeah.
- SGSarah Guo
And, uh, it's everything from upgrading and downgrading your subscription level to if you get a trial when you purchase a new vehicle, speaking to you about that. Broadly speaking, I would say we help companies build branded customer-facing agents. Um, and branded is an important part of it. It's, it's part of your brand, it's part of your brand experience and I think that's really interesting and compelling because I think just like, you know, when I go back to the proverbial 1995, you know, your website was on your business card. It was the first time you had sort of this digital presence and I think the same novelty and probably will look back at the agents today with the same sense of, "Oh, that was quaint." (laughs) Uh, you know, I remember, if you go back to the Wayback Machine and you look at early websites, it was either someone's phone number and that's it or it looked like a DVD intro screen with like lots of graphics. You know, a lot of the agents that customers start with are often around areas of customer service, which is a really great use case. But I do truly believe if you fast-forward three or four years, your agent will encompass all that your, your company does. I've used this example before but I like it, but just imagine an insurance company, all that you can do when you engage with them. Maybe you're filing a claim, maybe you're comparing plans, uh, we were talking about our kids earlier, maybe you're adding your child to your insurance premium when they get old enough to have a driver's license. All of the above. You know, all of the above will be, be done by your agent. So that's what we're helping companies
- 11:17 – 15:53
OpenAI’s Sora team on visual models and their role in AGI
- SGSarah Guo
build.
Next, we talked to the Sora team at OpenAI, which is building an incredibly realistic video AI generation model. In this clip, we talk about their research and how models that understand the world fit into the road to AGI. Is there anything you can say about how, um, the work you've done with Sora, uh, sort of affects the broader research roadmap?
- AKAndrej Karpathy
Yeah. So I think something here is about s- the knowledge that Sora ends up learning about the world just from seeing all this visual data. It understands 3D, which is one cool thing because we haven't trained it to. We didn't explicitly bake 3D information into it whatsoever. We just trained it on video data and it learned about 3D because 3D exists in those videos. And it learned that when you take a bite out of a hamburger that you leave a bite mark. So it's learning so much about our world and-When we interact with the world, so much of it is visual. So much of what we see and learn throughout our lives is visual information, so we really think that just in terms of intelligence, in terms of leading toward AI models that are more intelligent, that better understand the world like we do, this will actually be really important for them to have this grounding of like, "Hey, this is the world that we live in." There's so much complexity in it. There's so much about how people interact, how things happen, how events in the past end up impacting events in the future, that this will actually lead to just much more intelligent AI models more broadly than even generating videos.
- EGElad Gil
It's almost like you invented, like, the future visual cortex plus some part of the, uh, reasoning parts of the brain or something, sort of simultaneously.
- AKAndrej Karpathy
Yeah. And- and that's a cool comparison because a lot of the intelligence that humans have is actually about world modeling, right? All the time when we're thinking about how we're going to do things, we're playing out scenarios in our head. We have dreams where we're playing out scenarios in our head. We're thinking in advance of doing things. "If I did this, this thing would happen. If I did this other thing, what would happen," right? So we have a world model, and building Sora as a world model is very similar to a big part of the intelligence that humans have.
- SGSarah Guo
Um, how do you guys think about the, uh, sort of analogy to humans as having a very approximate world model versus something that is, um, as accurate as, like, let's say, a, uh, a physics engine in the traditional sense, right? Because if I, you know, hold an apple and I drop it, I expect it to fall at a certain rate, but most humans do not think of that as articulating a path with a speed as a calculation. Um, do you think that, uh, sort of learning is, like, parallel in, um, large models?
- GUGuest
I think it's a- a really interesting observation. I think how we think about things is that it's almost like a deficiency, you know, in humans, that it's not so high fidelity.
- SGSarah Guo
Mm-hmm.
- GUGuest
So you know, the fact that we actually can't do very accurate long-term prediction when you get down to a really narrow set of physics-
- SGSarah Guo
Mm-hmm.
- GUGuest
... um, is something that we can improve upon with some of these systems, and so we're optimistic that Sora will, you know, supersede that kind of capability and will, you know, in the long run enable it to be more intelligent one day than humans as world models.
- SGSarah Guo
Mm-hmm.
- GUGuest
Um, but it is, you know, certainly a- an existence proof that it's not necessary for other types of intelligence. Regardless of that, it's still something that Sora and- and models in the future will be able to improve upon.
- SGSarah Guo
Okay, so it's very clear that the trajectory prediction for, like, throwing a football is gonna be better-
- GUGuest
Yeah.
- SGSarah Guo
... um, the next, next versions of these models than mine is, let's say.
- AKAndrej Karpathy
If- if I could add something to that, this relates to the paradigm of scale and, uh, the bitter lesson a bit about how we want methods that as you increase compute get better and better, and something that works really well in this paradigm is doing the simple but challenging task of just predicting data. And you can try coming up with more complicated tasks. For example, something that doesn't use video explicitly, but is maybe in some, like, space that simulates approximate things or something, but all this complexity actually isn't beneficial when it comes to the scaling laws of how methods improve as you increase scale, and what works really well as you increase scale is just predict data, and that's what we do with text. We just predict text, and that's exactly what we're doing with visual data with Sora, which is we're not making some complicated, trying to figure out some new thing to optimize. We're saying, "Hey, the best way to learn intelligence in a scalable manner-
- SGSarah Guo
Yeah.
- AKAndrej Karpathy
... is to just predict data."
- SGSarah Guo
That makes sense in relating to what you said, Bill. Like, predictions will just get much better with no necessary limit that approximates-
- GUGuest
That's right.
- SGSarah Guo
... humans.
- 15:53 – 19:00
Waymo’s Dmitri Dolgov on bridging the gap to full autonomy and the challenge of 100% accuracy
- SGSarah Guo
We also sat down with Dmitri Dolgov, co-CEO of Waymo. Today, the company is scaling its self-driving fleet, completing over 100,000 fully autonomous rides per week in cities like San Francisco and Phoenix. It's my favorite way to travel. In this trip, Dmitri explains why achieving full autonomy, removing the driver entirely, and achieving 100% accuracy rather than 99.99% accuracy in self-driving is much harder than it might appear. Why is it breaking from, like, um, you know, uh, let's say, advanced driver assistance that i- seems to work in more and more scenarios versus, let's say, full autonomy?
- GUGuest
What's the- what's the delta?
- SGSarah Guo
Yeah.
- GUGuest
It's the number of nines, right? And it- it's the nature of this- this problem, right? If you think about, you know, where we started in 2009, uh, one of our first, uh, you know, uh, milestones, one of the goals that we set for ourselves was to drive, you know, uh, 10 routes. Each one was 100 miles long all over the Bay Area. Um, you know, freeways, uh, downtown San Francisco, around Lake Tahoe, you know, everything. And you had to do 100 miles with no intervention, so the car had to, you know, drive autonomous from the beginning, too. That's the goal that we created for ourselves. It was, you know, about a dozen of us. Took us maybe 18 months. We achieved that. 2009-
- SGSarah Guo
(laughs)
- GUGuest
... no ImageNet, no Continets, no transformers, no big models, com- tiny computers, you know? All this... Right? Very easy to get started. It's always been the property, and with every wave of technology, it's been al- you know, very easy, uh, to get started, but that... The hard problem, and this is kind of like- that- that early part of the curve has been getting, like, you know, even steeper and steeper.
- AKAndrej Karpathy
Mm-hmm.
- GUGuest
But that's not where the complexity is. The complexity is in the long tail of the many, many, many nines, and you don't see that if you go, you know, uh, for a prototype, if you go for, you know, a driver assist system, uh, and this is where, you know, we've been spending all of our... That's the only hard part of the problem, right? And I guess, you know, nowadays, it's always been getting easier with every technical, uh, uh, kind of, uh, cycle. So nowadays, you can take with, like, all of the advances of an AI and especially in the, you know, generative AI world and the LLMs and VLMs, you can take kind of an almost off-the-shelf... Uh, you know, transformers are amazing.
- SGSarah Guo
Mm-hmm.
- GUGuest
VLMs are amazing. You can take, uh, kind of a VLM, uh-... that, you know, can accept, uh, images or video, and is, you know, has a decoder where you can give it, you know, text prompts and it'll output text. And you can fine-tune it, you know, with just a little bit of, you know, data to go from, let's say, camera data on a car to, instead of words, to trajectories, or, you know, whatever decisions you want to make. You just, you know, take the thing as a black box-
- SGSarah Guo
Mm-hmm.
- GUGuest
... yeah, you take, um, whatever's been trained for you know, let me fine-tune it a little bit. And like not without, you know, 'cause I think if you ask any good grist in, in computer science to build, you know, NAV today, this is what they would do.
- SGSarah Guo
Yeah.
- GUGuest
And out of the box-
- SGSarah Guo
That's amazing. Yeah.
- GUGuest
... you get something that ... It, it, it's amazing, right?
- SGSarah Guo
Yeah.
- GUGuest
Like, the power of transformers, the power of is mind-blowing, right?
- SGSarah Guo
Hm.
- GUGuest
So with just a little bit of effort, you get something on the road, and it works. You can, you know, drive 100, 10s, 100s of miles and it will just be, it will blow your mind. But then is that enough? Is that enough to remove the driver and drive, you know, millions of miles and have a safety record, you know, that is demonstrably better than humans? No. Right? And I guess this is, you know, with every tech, you know, evolution, technology and, uh, breakthrough in AI, we've seen like appreciated.
- 19:00 – 23:29
Figma’s Dylan Field on the future of interfaces and new modalities
- GUGuest
- SGSarah Guo
Up next, we have my dear friend, Dylan Field, CEO of Figma. Dylan shares his prediction for how user interfaces will evolve in an AI-driven world. While many predict a shift toward conversational or agent-based interfaces, Dylan suggests that new interface paradigms will complement existing ones. He also highlights the exciting potential of visual AI and intelligent cameras as the next frontier in input methods.
- EGElad Gil
How do you think about the shift in UI in general that's gonna come with AI? A lot of things are kinda collapsing in the short run into chat interfaces. There's a lot of people talking about a future agentic world which does away with most UI altogether, and it's just all programmatic stuff happening in the background. Um, how do you think about where UI is, is going in general right now?
- GUGuest
I mean, I kinda think this comes back to the rabbit point I was making earlier. Yes, there's a lot of, uh, innovation happening in terms of agents, but I think, like, in terms of the way that we use UI to interact with agents, we're just beginning. And I think that the interfaces will get more sophisticated, um, but also even if they don't, I suspect that it's just like any new media type. When it's introduced, it's not like the old media types go away, right? Just 'cause you have TikTok doesn't mean that you, uh, you no longer watch YouTube. Even if it's true that, uh, a new form of interaction is via chat interfaces, which I'm, I'm not even sure I believe, but if, even if we take that as a prior on the No Priors podcast, then I, I think that you still have, uh, UI, and actually, I think you have more UI and more software than before.
- EGElad Gil
Do you have any predictions in terms of multimodality? Like, do you think there's more need for voice? Like, so, uh, you know, a lot of the debate people have is like, when are you gonna use voice versus text, uh, versus other types of interfaces. And, uh, y- you know, you could imagine arguments in all sorts of directions in terms of, you know, when do you use what and things like that. And a lot of people are, or not a lot, some people are suggesting because of the rise of mo- multimodal models, you'll have, like, more voice input or more things like that because you'll be able to do real-time sort of smart, uh, contextual semantic understanding of, like, a conversation. And so you have more of a verbal conversational UI versus a tech space UI or ... And so it kinda changes how you think about design. So I was just curious if, if you have any thoughts on that, that sort of future-looking stuff.
- GUGuest
There's all sorts of contexts where a voice UI is really important, and I think that, uh, uh, it might be that we find that voice UIs start to map to more traditional UIs, um, because it's something that, like, you could obviously do, uh, in a more generalized way. But yeah. I mean, personally, I don't want to navigate the information spaces that I interact with every day all day, uh, via voice. I also don't wanna do it in Minority Report style on the Vision Pro exactly either. Uh, maybe with a keyboard and mouse and, like, an amazing Vision Pro monitor setup or Oculus. Like, that could be cool, but I don't wanna do the Minority Report thing. And so it's, it's interesting 'cause I think that we get these new glimpses at interaction patterns that are really cool, and the natural inclination is to extrapolate and say they're gonna be useful for everything. And I think that they have, like, sort of their role, and, um, it doesn't mean that they're gonna be ubiquitous across every interaction we have. Uh, but that's a natural cycle to be in, and I think it's good. Uh, it's healthy to have sort of that almost mania around what can it do, because if you don't have that, then you don't get to find out. And so I, I, I'm supportive of, of people exploring as much as possible, uh, 'cause that's how you kind of progress on HCI and, and figuring out how to use computers and, to the fullest potential that that could be possible.
- SGSarah Guo
One of the things I am really bullish on is, I mean, maybe you just think of it as an input mode or a peripheral, but, um, it's really hard for people to describe things visually. And so the idea of intelligent cameras, even in the, like, most basic sense-
- GUGuest
Oh, it worked. (laughs)
- SGSarah Guo
Yeah. I, I ... It worked. I think that's actually a really fun space to be, as you said, like, exploring, um, because I, I actually think that will be useful, and, um, it's something that every user is capable of, right? Taking pictures, capturing video. And so I, I think that'll be ... I'm pretty bullish on that.
- 23:29 – 26:29
Scale AI’s Alexandr Wang on the journey to AGI
- SGSarah Guo
To wrap up our favorite moments from 2024, we have Scale CEO Alexandre Wang. In this clip, he shares his bold take on the road to AGI. Alex also dives into why generalization in AI is harder than many think, and why solving these niche problems and more data in evals is key to advancing the technology. Something you believe about AI that other people don't.
- GUGuest
My biggest belief here is that the, the path to AGI is, uh, is one that looks a lot more like curing cancer than, uh, than developing a vaccine. And what I mean by that is I think that the, the path to build AGI is going to be, um, in-... in- you know, you're going to have to solve a bunch of small problems that where you don't get that much positive leverage between, um, solving one problem to solving the next problem, and there's just sort of... You know, it's like curing cancer, which is you have to then zoom in to each individual cancer and solve them independently, and eventually, over a multi-decade timeframe, we're gonna look back and realize that we've, we've, you know, built AGI, we've cured cancer. But the, the path to get there will be this, like, you know, quite plodding road of, of solving individual capabilities and building individual sort of, um, data flywheels to support this end mission. Whereas, I think a lot of people in history paint the path to AGI as like, you know, eventually we'll just, boop, we'll get there. We'll like, you know... (laughs) we'll, we'll, we'll, we'll like, uh, we'll solve it, uh, in one fell swoop. And, um, I think this has a lot of implications for how you actually think about, you know, the technology arc and, and, and how the- how society is gonna have to deal with it. I think it's actually a pretty bullish case for society adapting the technology because I think it's going to be, you know, consistent slow progress for quite some time, and society will have time to fully sort of, uh, uh, acclimate to the technology it develops.
- SGSarah Guo
When you say solve, like, a problem at a time, right, if we just, like, pull away from the analogy a little bit, should, uh, should I think of that as, um, generality of multi-step reasoning is really hard, as, you know, Monta College Research is not the answer that people think it might be, um, we're just gonna run into scaling walls? Like what- sort of what are the dimensions of, like, solving multiple problems?
- GUGuest
I think the main thing, fundamentally, is I think there's, there's very limited generality that we get from these models, um, and even from multimodality, for example. Uh, my understanding is there's no positive transfer from learning in one modality to other modalities. So like training off of a bunch of video doesn't really help you that much with your text problems and vice versa. And so, I think what this means is like each, um, sort of, uh, each niche of capabilities or each area of capability is re- going to require separate flywheels, data flywheels, to be able to, to push through and drive performance.
- SGSarah Guo
You don't yet believe in video as basis for world model that helps?
- GUGuest
I think that's-
- SGSarah Guo
Far less reason.
- GUGuest
I think it's great narrative, I don't think there's strong scientific evidence of that yet. Maybe there will be eventually, um, but I think that this is the, uh... I think the base case, let's say, is one where, you know, there's not that much generalization coming out of the models, and so we actually just need to slowly solve lots and lots of little problems to ultimately result in AGI.
- 26:29 – 27:06
Outro
- GUGuest
- SGSarah Guo
Thank you so much for listening in 2024. We've really enjoyed talking to the people reshaping the world for AI. If you wanna more deeply dive into any of the conversations you've heard today, we've linked the full episodes in our description. Please let us know who you want to hear from and what your questions are for next year. Happy holidays. Find us on Twitter @nopriorspod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.
Episode duration: 27:07
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode gmllIye7BBY
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome