David Luan: Why Nvidia Will Enter the Model Space & Models Will Enter the Chip Space | E1169

What OpenAI realized before basically everybody but DeepMind was that the next phase of AI after Transformer was not going to be about research paper writing. It was going to be about, "Let's choose a major unsolved scientific problem and just try to solve it." The second way of improving model performance is just starting to be tapped now, and that's also going to absorb a boatload of compute. So because of that, I actually am not worried about diminishing returns to compute over time. I think every tier one cloud provider existentially needs to win here.

Ready to go? (instrumental music plays) David, I am so excited for this. I've wanted to do this one for a long time. I've heard so many good things. So first, thank you so much for joining me today.

Yeah. Thanks, Harry, for having me on it. I've, um, got to watch some of your cool previous episodes, so it's a, it's a real honor to be on here.

And that's very, very kind of you. I really do appreciate that, man. But you, you've been at some incredible companies as a training ground, so to speak, one of which was Google Brain, and I just wanted to start there. When you think about your biggest takeaways from your time with Google Brain, what would you say ones two are, and how do you think that shaped how you think about building Adept today?

Google Brain, um, uh, uh, was, and also now is part of DeepMind, is a really magical place. It's, uh, it's the, I think during the, uh, peak days of AI, AI progress on the research side, right, where every day there was a new paper that came out that would just change the world, that, like, 2012 to 2018 or so era of Google Brain was just, like, incredibly dominant. They did an amazing job picking talent, like the people who invented Transformer, the people who invented the diffusion model, people who did, um, all of these new optimization techniques that we all take for granted today. They were all at Brain at the same time. Like, truly the Bell Labs of, of the era. And I think I learned a lot about how to make, um, how to make pure bottom-up, like to see what good pure bottom-up basic research looks like at Google Brain.

What does that mean, what pure bottoms-up good basic research is?

Uh, yeah. So, so I have this worldview, um, of, of AI progress as being a part of a couple different phases, right? And I, I like to think about pre-2012 as basically being pre-history. Of course, um, all of the, uh, OGs in the field would probably not like that if I characterized it that way, but before 2012, like, most of the things we tried just didn't really work, right? Like, you, you had things like, um, uh, a sheep being identified as cats and dogs, and, uh, chatbots that, um, barely said anything coherent, et cetera. But I think, like, there was a p- uh, there was a period between 2012 to, like, 2017 or 2018, where deep learning went from something that, um, people didn't believe in to being, like, the dominant paradigm in the field. Right? And so during that 2012, 2018 era, the way people made progress, um, what I mean by bottom-up basic research is you hire the most brilliant scientists. They come to work every day with, like, no, um, near-term objective they're being held accountable to, um, and they just work together and they think about, you know, "Hmm," like, "I wonder what it'd be like if we could solve this, like, open, uh, technical problem in AI." Like, "How do we go, um, how do we create a model that better understands how to generate images?" And they just go work on that out of their own curiosity and drive and maybe some interest in glory and fame through papers. And they do that for, uh, they, they do that for, like, six months or so, and then out pops out this, like, research paper that gets posted to arXiv and goes to a p- a, a journal that, um, that, like, just solves the problem. Like, that's, that's huge. Right? And so, m- the reason I call it bottom-up is because it's just driven by the natural, uh, interactions between all, all these researchers in a, in a setting, and they figure out what they wanna do.