No Priors Ep. 7 | With Stanford Professor Dr. Percy Liang

(music plays) Thanks, Percy.

Great, welcome.

Um, so I think just to start, can you tell us a little bit about how you got into, uh, the machine learning research field and your personal background?

Yeah. So I've been in the field of machine learning and natural language processing for over 20 years. I started getting into it in undergrad. I was an undergrad at MIT. I liked theory. I had a fascination with languages. I was fascinated by how humans could just, um, be exposed to just strings of, uh, text, uh, I mean, uh, speech, and somehow acquire very sophisticated understanding of the world and also syntax, and learn that in a fairly, uh, unsupervised way. And my dream was to get computers to do the, the same, so then I went to grad school, uh, at Berkeley, and then after that started at, uh, Stanford. And ever since, I've been in pursuit of, uh, you know, developing, uh, systems that can really truly understand natural language. Um, and of course, in the last four years, um, this once upon a time kind of dream has really kind of taken off in a s- in a sense. Um, maybe in a, not a way that I would necessarily ex- expect, uh, but with the coming out of, uh, large language models such as GPT-3, it's truly kind of astonishing how much of the structure of language and the world that these models can, can capture. Um, in some ways, it kind of hearkens back when I actually first started in NLP. I was, uh, training language models, but of a very different type. It was based on, uh, Hidden Markov Models. And there the goal was to discover hidden structure in, in text, and we were... I was very excited by the fact that it could learn about, um, tease apart what words were like city names versus days of the week and so on. Um, but now it's kind of on a d- completely different l- level.

Was there a moment, since y- you know, you've worked on multiple generations of NLP at this point, you know, pushing the forefront of semantic parsing, was there a moment at which you, um, decided that, you know, you were gonna focus on foundation models and large language models?

Yeah. There was a very decisive moment, and that moment was when GPT-3 came out.

Okay.

That was in the middle of the pandemic. Um, and it wasn't so much the capabilities of the model that, um, shocked me, but it was the way that the model was trained, which was basically taking a massive amount of text and asking the model to predict the next word over and over again, you know, billions of times. And just that simple, uh, objective and a very simple principle, what r- rose from it was not only a model that could generate fluent text, but also a model that could do in-context learning, which means that you can prompt a language model with instructions, uh, for example, summarize this document, give it some examples, and have the model on the fly in context figure out what the task was. And this was a paradigm shift in, in my opinion because it changed the way that we conceptualize machine learning and NLP systems from these bespoke systems where you're, it's trained to do question answering, to train to do this, to just a general, um, substrate where you can ask the model to do various things. And then the idea of a task which is so central to AI I think s- begins to dissolve, and I find that extremely exciting. Um, and that's the reason later, um, in 2021, uh, we founded the Center for Research on Foundation Models. We coined the term foundation models because we thought it, there was something that was happening in the world that was, that somehow large language models didn't really capture the significance. And it was not just about language, it was about images and multimodality, it was a more general phenomenon, and we coined the term foundation models and then, um, then the center started and it's been sort of, you know, a kind of a roller coaster ride, uh, ever since.