John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI

I think even in one or two years, you could imagine having the models carry out a whole coding project, moving away from using the model of like a search engine and more towards having a whole project that I'm, like, doing in collaboration with the model. We might not wanna jump to having AIs run whole firms immediately, even if the models are good enough to actually run a successful business themselves.

D- if there's no other bottlenecks next year or y- or something, you got AGI, wh- what- what's the plan? Today, I have the pleasure to speak with John Schulman, who is one of the co-founders of OpenAI and leads the post-training team here. And, um, he also led the creation of ChatGPT and is the author of many of the most important and widely cited papers in AI and RL, including PPO and many others. So John, really excited to chat with you. Thanks for coming on the podcast.

Thanks for having me on the podcast. I'm a big fan.

Oh, thank you. Tha- thank you for saying that. Um, so the first question I had is, we have these distinctions between pre-training and post-training. Beyond what is actually happening in terms of loss, function, and training regimes, I'm just curious, taking a step back conceptually, like, what kind of thing is pre-training creating? What does post-training do on top of that?

In pre-training, you're basically training to imitate all of the content on, on the internet or on the web, um, including websites and code and so forth. Uh, so you get a model that can basically, um, generate, uh, content that looks like, uh, random web pages from the internet. And, um, th- the model is also trained to maximize likelihood, where it has to put a probability on everything. So it's, um... The objective is, uh, basically predicting the next token given the previous tokens. Tokens are, like, words or parts of words. And, uh, since the model has to put a probability on it, uh, and it's, we're training with, um... to maximize log probability, it ends up being very calibrated. So it can not only generate all of this, uh, the content of the web, it can also assign probabilities to everything. So, so the base model can effectively take on all of these different personas or generate, um, all these different kinds of content. And then, uh, when we do post-training, uh, we're usually targeting a narrower, um, range of behavior where we basically want the model to behave like this kind of chat assistant. And, uh, it's a, it's a more specific persona where it's, um, th- trying to be helpful. It's not trying to imitate a person. It's, um, answering your questions or doing your tasks. Um, and, uh, we're optimizing on a different objective, which is more about producing outputs that humans will like and find useful, as opposed to just trying to imitate, uh, this raw content from the web.