Joe Rogan Experience #1934 - Lex Fridman

(drumming music) Joe Rogan podcast. Check it out.

The Joe Rogan Experience.

Train by day, Joe Rogan podcast by night. All day. (rock music) What's up, brother? How are you? Good to see you, my friend.

It's good to see you.

Hey, uh, what have your people done? Your, your AI people with this fucking ChatGPT shit? This scares the fuck out of me.

Your people?

It scares me.

What do you mean?

Your AI people.

Your people? (laughs)

Your, your wacky coders. What have you done?

Yeah, it's super interesting.

Fascinating.

Language models, I don't know if you know what those are, but that's the general, uh, systems that, uh, underlie ChatGPT and GPT. They've been progressing over the past maybe four years aggressively. There's been a lot of development, GPT-1, GPT-2, GPT-3, uh, GPT-3.5. And ChatGPT, there's a lot of interesting technical stuff that maybe we don't wanna get into, but-

Sure, let's get into it.

Well, there was-

I'm, I'm fascinated by it.

So ChatGPT is based on fundamentally on 175 billion, uh, parameter neural network that is GPT-3, and the rest is what data is it trained on and how is it trained. So you already have like a brain-

Mm-hmm.

... a giant neural network, and it's just trained in different ways. So Chat... Uh, GPT-3 came out about two years ago, and it was like impressive but dumb in a lot of ways. It was like you would expect as a human being for it to generate certain kinds of texts, and it was like saying kind of dumb things that were off. And you're like, "All right, this is really impressive, but it's not quite there." You can tell it's not intelligent. And what they did with, uh, GPT-3.5 is they started adding more and different kinds of datasets there. One of them, probably the smartest neural network currently, is Codex, which is fine-tuned for programming. Like, it was, it was, uh, trained on code, on programming code. And when you train on programming code, which Chat, ChatGPT is also, you're teaching it something like reasoning, 'cause it's no longer, uh, information and knowledge from the internet. It's also reasoning. You can like logic. Even though you're looking at code, programming code is- (laughs) You're looking at me like, "What the fuck is he talking about?"

(laughs) Oh, Jesus. No, no, no, no, that's not what I'm looking at.

But, so-

I'm looking at you like, "Oh, my God."

But reasoning is a... In order to be able to stitch together sentences that make sense, you not only need to know the facts that underlie those sentences, you also have to be able to reason.

Yeah.

And, and we think of it, we take it for granted as human beings that we can do some common sense reasoning. Like s- like this war started at this date and ended at this date, therefore it means that, uh, like the start and the end has a meaning. There's a temporal consistency. There's a cause and effect. All of those things are inside programming code. By the way, a lot of stuff I'm saying we still don't understand. We're like intuiting why this works so well.