EVERY SPOKEN WORD
70 min read · 13,647 words- 0:00 – 1:50
Introduction
- EGElad Gil
(digital music) Noam, welcome to No Priors.
- NSNoam Shazeer
Hey, Elad. Th- uh, thanks for having me on. Uh, hi, Sara.
- EGElad Gil
Good to see you. Yeah, thanks for joining. So, um, you've been working on the NLP and AI for a long time. So I think you were at Google for something like 17 years off and on. And I think even your Google interview question was something around spellchecking, an approach that eventually got (laughs) implemented there. Um, and when I joined Google, one of the main, um, systems being used at the time for ads targeting was like fill, and fill clusters, and all the stuff which I think you wrote with George Herrick. And so it'd just be great to get kind of your history in terms of working on, um, AI, NLP, language models, how this all evolved, what you got started on, and what sparked your interest.
- NSNoam Shazeer
Oh, thanks, Elad. Yeah, uh, just, uh, always was naturally drawn to AI, you know? Wanted to make the computer do something smart. Seems like pretty much the most, uh, fun, uh, fun game around. Um, so, uh, yeah, was, uh, lucky to find, uh, Google early on, and, uh, it really is, uh, an AI company. So, um, yeah, I got, uh, got involved in a lot of the, uh, early projects there that, uh, that maybe you wouldn't call AI now but, uh, but seemed pretty smart, uh, at the time. Um, and then more recently was on the Google, uh, Brain team starting in 2012. It looked like a really smart group of people, uh, doing something interesting, and never... Uh, I had never done deep learning before, or neural networks I guess as it was called then, or whatever. I forget when the rebrand happened-
- EGElad Gil
(laughs)
- NSNoam Shazeer
... but, uh (laughs) -
- EGElad Gil
Yeah.
- NSNoam Shazeer
... but, uh, yeah, it turned out to be really fun.
- EGElad Gil
That's cool. And then, you know, you were one of the main people working on the transformer paper and design in 2017, and then you worked on Mesh TensorFlow, I think, um, sometime within the following year. Could you talk a little bit about how
- 1:50 – 7:13
Noam’s early AI projects at Google
- EGElad Gil
all that got going?
- NSNoam Shazeer
Yeah, I, I mean, I, um, messed around a few years, um, on the Google Brain team and, like, utterly failed at a bunch of stuff, uh, till I kinda got the hang of it. Um, really the key insight is that what makes deep learning work is that it is, um, really well-suited to, uh, to modern hardware, um, where, you know, you have the, uh, the current generation of, uh, of chips that are great at, um, at matrix multiplies and, you know, o- other forms of things that require large amounts of, um, computation relative to communication. So, uh, so basically deep learning, like, really took off because, you know, it runs thousands of times faster than anything else. And as soon as I got the hang of that, started designing things that actually, uh, were smart and, uh, and ran fast. Um, but, you know, the most, most exciting, uh, problem out there is language modeling. It's just, like, i- i- it's like the best problem ever, because, like, there's, like, an infinite amount of data, you know, just scrape the web and you've got, like, all the training data you could ever, uh, ever hope for. And, like, the problem is super simple to define. It's just, like, um, predict the next word. The fat cat sat on the, you know, like, okay, what, you know, what comes next?
- EGElad Gil
(laughs)
- NSNoam Shazeer
Like, it's, it's, it's extremely easy to define, and if you can do a great job of it, like, you know, then you get everything that you're seeing right now, uh, and more. You can just talk to, uh, talk to the thing and it's, it's really AI-complete. And so, uh, so got started, uh, around, like, 2015 or so, uh, w- you know, working on language modeling and, uh, messing with the recurrent neural networks, which was, uh, what, uh, what, what was great then, and then, uh, transformer kinda came about as, as, uh, you know, uh, someone had the bright idea. It was, uh, Jakob, uh, it was great, the, that, "Hey, like, these RNNs are just annoying. Let's, uh," (laughs)
- EGElad Gil
(laughs)
- NSNoam Shazeer
"let's try to replace 'em with something better, uh, you know, uh, uh, uh, in attention." And, uh, you know, then over- heard a couple of colleagues, uh, talking about it, and, uh, you know, in the, in the next cube over I was like, "That sounds great. Let me help you guys. Um, you know, these RNNs are annoying and we could, uh, it's gonna be so much more fun."
- EGElad Gil
Can, can you quickly describe sort of the difference between an RNN and, uh, an, and a transformer-based or attention-based model?
- NSNoam Shazeer
Yeah, sure. Um, so, um, okay, so, like, the recurrent neural network is like the sequential computation where every word, you know, you read the next word and you, you kind of compute your current, you know, state of, uh, of y- of your brain based on the old state of your brain and the, um, you know, and, and what this n- this next word is, and then you, you predict the next word. So you have this very long sequence of computations that has to be, um, you know, executed in order. So that, you know, the magic of transformer, kind of like convolutions, is that you get to process the whole sequence, uh, at once. I mean, it, it still talks, you know, it, it's still a function of, like, the, you know, the, the predictions for the later words are, uh, dependent on what the earlier words are, but it happens in, like, a constant number of steps where you, where you get to, to, to take advantage of, like, this parallelism of you can look at, like, the whole thing at once and, like, that's what modern hardware's good at is parallelism, and now you can use the length of the sequences, your parallelism, and everything works, you know, super well. Um, attention itself, it's kind of like you're, you're creating this big, um, like, key value associative memory where y- you're, you're like building this big table, like, with one entry for every, for every word in the sequence, and then you're kind of looking...... looking things up in that table, and it's- it's all, like, fuzzy and differentiable and, you know, a, a big, uh, differentiable function that you can, you can backprop through. And people have been using this for, for problems where there are two sequences, where you've got, like, machine translation, you're, like, translating, um, like, English to French. And so while you're producing the French sequence, you are, like, looking over the English sequence and trying to pay attention to the right place in that, in that sequence. But, like, the insight here was, hey, you can use the same attention thing to, like, look back at the past of the sequence that you're, that you're trying to produce and, you know, the beauty is that, uh, that it runs great on, uh, on GPUs and TPUs and, um, it's kind of parallel to, like, how deep learning is, uh, like, has taken off, because it's, like, great on the hardware that, that exists, and- and this sort of brought the same thing, you know, uh, to, uh, to sequences.
- EGElad Gil
Mm-hmm.
- SGSarah Guo
Yeah, I think the classic, the classic example to help people picture it is, like, you know, saying the same sentence in French and English. Like, the ordering of the words is different, uh, so you can't, you're not mapping, like, one-to-one in that sequence, and to figure out how to do that with parallel computation without information loss was, like, a really elegant thing.
- NSNoam Shazeer
Yep.
- EGElad Gil
Yeah. Uh, it seems like the technology's also been applied in a variety of different areas. The obvious ones are,
- 7:13 – 11:13
Noam’s focus on language models and AI applications
- EGElad Gil
um, these multimodal language models, so it's things like ChatGPT or what you're doing at Character. Um, I've also been surprised by some of the applications into things like AlphaFold, the protein folding efforts at Google that w- it actually worked in an enormously performant way. Are there any application areas that you found really unexpected, relative to how transformers work and relative to what they can do?
- NSNoam Shazeer
Oh, um, I've just had my head down in, uh, (laughs) in language-
- EGElad Gil
Yeah.
- NSNoam Shazeer
... like th- like, y- here you have, like, something that, like, a, a problem that, like, can do, like, anything. Like, I want this thing to be good enough, so I just ask it, like, "How do you cure cancer?" And it, like, invents a solution. Um, a- and, you know, like, so, so I've been totally ignoring, like, what everybody's been doing in, uh, in all these other modalities where, like, I think a lot of the early successes in, in deep learning ha- have been, like, in images and people are, like, all excited about images, and I kind of, like, completely ignored it 'cause, like, you know, an image is worth a thousand words but it's, like, a million pixels. So, like, the text is, like, a thousand times as dense, so, like, kind of big, uh, big, uh, text, uh, text nerd here. But, um, you know, it's very exciting to see it, uh, it take off in, you know, in all these other modalities as well and, you know, tho- those things are gonna be great. It's, uh, like, super useful for, uh, building products that people wanna use, uh, but I think the, a lot of the core intelligence is going to come from, uh, from these text models.
- EGElad Gil
What do you think creates the асымтоитthat all this is being built against? Because people often talk about just scale, like, you just throw more compute and this thing will scale further. There's data and different types of data that may or may not be available. There's algorithmic tweaks. Um, there's adding new things like memory or loopbacks or things like that. Like, what- what do you think are the big things that people still need to build against, and where do you think this sort of taps out as an architecture?
- NSNoam Shazeer
Yeah, I don't know that it taps out. I mean, we haven't seen it tap out yet. The amount of, of work that has gone into it is probably nothing compared to, like, the amount of work that will go into it, so quite possibly there will be all kinds of, like, factors of two inefficiency that people are gonna get through better, uh, better training algorithms, better model architectures, um, you know, better ways of, uh, building chips and using quantization and, like, all of that. And then there are going to be, you know, factors of 10 and 100 and 1,000 of, um, just, like, you know, scaling and money that people are just gonna throw into the thing because, you know, hey, like, everyone just realized this thing is phenomenally valuable. So ... And, and at the same time, like, I don't think anyone's seen a wall in terms of how good this stuff is, so I think it will just, it's just gonna keep getting better and I don't know what stops it.
- SGSarah Guo
What- what have y- what do you think about this, um, sort of idea that we can increase compute but the largest models are undertrained? We've used all the text data on the internet that's easily accessible, like, you know, we- we have to go improve the quality, we have to go do human feedback. Like, how do you think about that?
- NSNoam Shazeer
Yeah. I mean, in terms of getting some more data, like, there are a lot of people talking all the time. I mean, the- there's (laughs) -
- SGSarah Guo
(laughs)
- NSNoam Shazeer
... you know, like, uh, you know, like-
- SGSarah Guo
Why do you think we do this podcast? (laughs)
- NSNoam Shazeer
Right. Like, there's, like, order of, like, 10 billion people, like, uh-
- SGSarah Guo
(laughs)
- NSNoam Shazeer
... you know, producing, like, a thousa- you know, like, I don't know, 10,000 words a day, each. I mean, that's, like, that's, like, a lot of words that, you know ... And pretty soon a lot of, uh, many of those people will be doing a lot of that, the, that talking, uh, to AI systems, so I- I have a feeling, like, a lot of data is going to, um, find its way into some AI systems. Um, but, uh, I mean, in privacy-preserving ways, I- I would, uh, I would hope. And then the, um ... You know, the, the data requirements tend to go up, like, with the square root of the amount of computation 'cause you're gonna train a bigger model and then you're going to, um, throw more data at it. So, uh, you know, uh, I- I- I think I'm not that worried about coming up with data, and
- 11:13 – 13:53
Character’s co-founder Daniel de Freitas Adiwardana work on Google’s Lambda
- NSNoam Shazeer
I feel like we could probably, like, just generate some more with the AI. Like ...
- SGSarah Guo
(laughs)
- EGElad Gil
Yeah. (laughs) And then what do you think are the main things to solve for these models going forward? Is it hallucinations? Is it memory? Is it something else?
- NSNoam Shazeer
I don't know. I kind of like hallucinations. Um ... (laughs)
- EGElad Gil
(laughs)
- NSNoam Shazeer
It's-
- SGSarah Guo
Also a feature, yeah.
- NSNoam Shazeer
They're fun. Yeah.
- EGElad Gil
Yeah.
- NSNoam Shazeer
Well, I'll call it a feature. Um ... Yeah. I mean, yeah. Some of the things we wanna work on the most are, like, um, memory, 'cause, you know, our users definitely want their virtual friends to remember them. Uh, (laughs) you know, they- there's, like, you know, there's so much you can do with, uh, you know, personalization and, you know, you wanna dump in a lot of data and use it effi- uh, efficiently. Um ... Yeah. I- I think, yeah, there's, like, a ton of great work going on in the, you know, in, uh-... trying to, uh, figure out what's real and what's hallucinated, of course. I think we'll solve this.
- EGElad Gil
Do you wanna talk a little bit about LaMDA and your role with it and, you know, how that led eventually to Character?
- NSNoam Shazeer
Yeah. Uh, sure. Yeah. I guess, um, yeah, my co-founder Daniel, uh, Daniel de Freitas, he's like the, you know, scrappiest, uh, most, uh, you know, hardworking, really s- you know, smartest guy, you know. He's kind of been on this lifelong mission to build, um, chatbots, like he wa- like, since, like, he was, like, a kid in Brazil. He's, like always es- you know, been trying to build chatbots. So, like, he, uh, came to join us at Google Brain because I think he had read some, some papers and, uh, figured that this neural, um, neural language model technology would be, like, you know, something that could actually generalize and, you know, uh, and build something truly open domain. So... And, and like he, um, yeah, didn't, did not get a lot of headcount. Like, he started the thing as like a 20% project where like people are encouraged to, uh, spend 20% of their time like doing whatever they want. So, um, and then he just like recruited like an army of like 20% helpers who were like ignoring their, uh, their day jobs and like actually just, uh, you know, help- helping him with the system, and he like went as far as like, um, going around and like, uh, panhandling people's TPU quota. Like, and he called this Project Mina 'cause he, uh, I- I guess it came to him in a dream and like at some point, um, like looking at the scoreboard and was like, "What is this thing called Mina and why does it have 30 TPU credits?" And it was like, yeah, like (laughs) just gotten a bunch of people to contribute. And then he was like really successful at this because, um, you know, i- in, in building something really cool that, uh, that actually worked where like a lot of other, uh, systems were just like totally failing either because people were not,
- 13:53 – 18:47
The origin story of Character.AI
- NSNoam Shazeer
um, you know, just weren't scrappy enough or were, uh, like going for like rule-based systems that were just never going to, uh, generalize. So like at, at some point I was like, "Okay, there's so many ways we can make this technology better by like factors of two, but like the biggest thing is like just convince everyone that this is like worth trillions of dollars by like demonstrating some application that is clearly super valuable to like billions of people." So-
- EGElad Gil
And LaMDA was this, um... I believe is the, is the internal chatbot pre-GPT at Google that, um, was famously in the news because an engineer thought it'd become sentient, right? Uh-
- NSNoam Shazeer
Yeah. Yeah, yeah. So that was like a renaming of, uh, Mina. So I guess I went and helped Daniel on Mina. We got it on some giant, uh, language models and then kind of became like an internal, like, uh, viral sensation, and then, then got renamed to LaMDA and, um... Yeah, we had left before th- uh, before the, the business of how somebody thought it was sentient. (laughs) and Father.
- EGElad Gil
(laughs)
- SGSarah Guo
Can you talk a little bit about just what, um... like, why it wasn't released? Like, what some of the concerns were?
- NSNoam Shazeer
I mean, I, I, I think just like large companies have concerns around, like, launching products that can say anything. (laughs) Like, I, I mean, it's, it's probably just a mat- uh, like, uh... I would, I would guess it's just like a matter of, uh, how much you're risking versus, uh, how much you, you, you have to gain from it. Um, you know, so figured hey, startup. Uh, startup seems like the, the right idea in that you can kind of just, uh, move faster.
- SGSarah Guo
Yeah. So tell us about, uh, Character. Like, what's the origin story there? Uh, did you and Daniel look at each other one day and were just like, "We have to get it out there."
- NSNoam Shazeer
Yeah, pretty much. We were like, "Yeah." And kinda noticed, hey, you know, there are people who like just go out and, you know, get, uh, get some investors and, and start doing something. So we've just like, "Okay, now we, uh... Let's just like, let's just like build this thing and launch it as fast as we can." So hired like a, you know, total rock star team of, uh, of engineer researchers and um, um, you know, got some compute.
- EGElad Gil
Have, uh... Uh, one, one thing that comes up a lot is people say that you all have, you know, one of the truly extraordinary teams in the AI world. Uh, are there specific things that you recruited against, or how did you actually go about finding these people?
- NSNoam Shazeer
Um, you know, some people, some people we knew from Google, uh, happened to get introduced a mile out from, uh, for- formerly from, uh, Meta, who's, uh... like was, uh, had, uh, built a lot of their large language model stuff and, uh, and, uh, their, uh, you know, sort of neural language model infrastructure. And, uh, you know, he, uh... a, a bunch of other Meta people followed him and, uh, you know, they were great.
- EGElad Gil
Is there anything specific that you would like look for in people or ways to test for it or was it just standard-
- NSNoam Shazeer
Yeah.
- EGElad Gil
... interviewing approaches?
- NSNoam Shazeer
Uh, I mean, you know, a, a lot of it was, um, just kind of motivation. I think, uh, Daniel tends to, um, like very, very highly value like motivation. I think he's looking for something between like burning desire and childhood dream. So like-
- EGElad Gil
(laughs)
- NSNoam Shazeer
... there were a lot of great people that w- that we did not hire, uh, because they d- uh, didn't quite meet that bar, but the- then we got a bunch of people who were uh, kinda up for uh, up for, uh, joining a startup and, uh, and really talented and, uh, and highly motivated.
- SGSarah Guo
I mean, speaking of like childhood dreams, like s- do you wanna describe the product a little bit? Like you can... You have these bots. Like, you know, they, they can be user created, they can be character created, um, can be public figures, fictional figures, anybody with like a corpus that you can make up or historic figures. Like, how'd you even, uh, arrive there as like the right form for this?
- NSNoam Shazeer
Yeah. I mean, like basically this is kind of a technology that, um, that's so accessible that like billions of people can just invent use cases, you know? And, and y- like, it's so flexible that, you know, you really just wanna put the user in control because often they know way better than you do what, what they wanna use the thing for. And I guess we had kind of seen, um...You know, s- some of the, you know, we... Like, like, uh, assistants from, uh, sort of assistant bots from, uh, you know, from large companies. You know, you've got whatever, you've got, uh, Siri and Alexa and Google Assistant. And, uh, and, you know, like some of the problems there are that, like, when you're just projecting one persona to the world, like people will A, you know, expect you to, like, be very consistent in, say, like, your likes and dislikes, and B, just like not be offensive to anyone and, like, not really have
- 18:47 – 26:51
How AI can express emotions
- NSNoam Shazeer
an opinion. It's kinda like, you know, like you're the Queen of England and you can't, like, say something that's, that, you know, that's going to disappoint someone or, like, I don't know. Like, I remember, like, I think it was, like, George H.W. Bush said he didn't like broccoli, and then, like, the broccoli farmers-
- SGSarah Guo
(laughs)
- NSNoam Shazeer
... were, like, all mad at him or something. So, like, you can't, like, uh, like, if you're, like, such a public, you know, trying to present, like, one public persona that everyone likes, you're going to end up just being boring, essentially. And people just don't want boring. P- You know, people want, like, the, you know, uh, wanna interact with, uh, something that feels human, you know? So, so basically you need to go for, like, you know, multiple personas. You know, like, let people-
- SGSarah Guo
Mm-hmm.
- NSNoam Shazeer
... invent personas as much as they want, and kind of... I like the name character 'cause it's like, okay, you know, it's, it's got a few different meanings, you know? There's character, like, you know, ASCII character, uh, it's like unit of text character, like, uh, like, you know, like a persona or a character like, you know, good morals. But, um, but anyway, so I, I think that's just how people like to relate to this stuff. It's like, okay, I kinda know, know what I, what to expect, you know, like from an experience if I can kinda define it as a person or a character. Maybe it's someone I know. Maybe it's just, like, something I invent. Um, but it, it kind of helps people, like, kinda use their imagination.
- SGSarah Guo
So, what do people, what do people want? Like, do they do, like, their friends? Do they do fiction? Do they do entirely new things?
- NSNoam Shazeer
Yeah, I mean, there's, like, a lot of, um... You know, there, there's a lot of, you know, role playing, like role playing games are big. You know, like, you know, like text adventure where it's just, like, making it up, uh, uh, as it goes. There's a lot of, like, video game characters and anime, and there's, um, you know, uh, you know, some amount of people talking to public figures and influencers and, like, you know, like... Uh, I think a lot of people have these existing parasocial relationships where they- there's... You know, they've got characters they're following, like, on, uh, on TV or some, uh, you know, or internet or influencers or, or whatever. And, um, and so far, they just have not had the experience of, "Okay, the- now this character responds," 'cause like, you know, the... It's, it's always something you can watch or maybe you're in like a, you know, thousand on one, like, fan chat or something where, like, this VTuber will, like, get right back to you, like, once, you know, o- once in an hour or something. But, like, now they get the experience of, "Oh, like, I can just create a version of this privately and just, like, uh, you know, t- just like, uh, talk to it," and it, and it, uh, it's pretty fun. We also see, like, a lot of, um, you know, people using it 'cause they're, you know, they're, they're lonely or, uh, or, uh, or troubled and need someone to talk to. Like, so many people just don't have someone to talk to, so, like, you know. And a lot of use kind of crosses all of these boundaries, like some, you know, somebody will post, "Okay, this video game character is my new therapist," or something.
- SGSarah Guo
(laughs)
- NSNoam Shazeer
So, like, it, it's a huge mix of, like, fun and people who need a friend and, like, um, connecting with, you know, you know, game playing, all, all kinds of stuff.
- SGSarah Guo
How do you, how do you think about, um, emotion, like both ways, right? Like people's relationships with characters or, like, how, you know, like, what level we are at in expressing c- coherent emotion and how important that is?
- NSNoam Shazeer
Oh, yeah. I mean, probably you don't need that high an, like, level of intelligence, like, to, you know, to do, uh, to do emotion. I mean, emotion is great and it's, it's super important, but, like, you know, like, a dog probably does emotion pretty well, right? Like, it's, like... I mean, I don't have a dog but I- I've heard that, like, a dog is great for, like, emotional support and it's, like, got pretty lousy linguistic capabilities. But, um-
- SGSarah Guo
(laughs)
- NSNoam Shazeer
... but, you know, the emotional use case is huge and people are using the stuff, uh, you know, uh, you know, for, you know, for, for all kinds of, uh, uh, emotional support or relationships or, or whatever, wh- which is, which is just terrific. Um...
- SGSarah Guo
How do you think the, the behavior of the system will change as you kinda scale things up? Because, you know, I think the original model was trained on not a ton of money. Like, on a relative basis-
- NSNoam Shazeer
Yeah.
- SGSarah Guo
... you folks were incredibly frugal. Um...
- NSNoam Shazeer
Yeah. Um, I, I think, uh, yeah, we, we should be able to make it smarter in all kinds of ways, both, uh, both algorithmically and, uh, and just, uh, scaling, you know? Uh, get more, uh, get more compute and, uh, train a bigger model and train it for longer. Um, yeah, it should just get more brilliant and more knowledgeable and better attuned to, like, what, what people want and what p- what people are looking for.
- SGSarah Guo
You have some users that are on the service, like, many hours a day. Um, like, uh, how do you think about your target user over time and, like, uh, you know, what the usage patterns do you expect they are?
- NSNoam Shazeer
Um, we're gonna just leave that up to the user. Like, uh, (laughs) our, our, our aim has always been, like, get something out there and let users decide, you know, what th- what they think it's, uh, what they think it's good for. And, you know, we see, like...... yeah, yeah, like, uh, as somebody who's on the site today, is active for about two hours on average today, uh, th- i- that's of people who send a message today, which is, which is pretty-
- SGSarah Guo
That's wild, though.
- NSNoam Shazeer
... yeah, that's pretty wild. But-
- EGElad Gil
Yeah.
- NSNoam Shazeer
... like-
- SGSarah Guo
It's amazing.
- NSNoam Shazeer
... um, yeah, it's, it's def- you know, it's a great metric, like, that people are finding some sort of value in it, and as I said, it's really hard to pin down exactly what that value is, you know, but, because it's, it's really, like, uh, you know, a big mix of things. But, like, our goal is, like, make, make this thing more useful to people and let people kind of customize it and decide, uh, what they want to use it for, if it's, uh, if it's, you know, brainstorming or help or information or, uh, or fun or, like, emotional support, like, um, you know, let's, let's, uh, just get it, get it into users' hands and, and see what happens.
- SGSarah Guo
How, how do you think about commercialization?
- NSNoam Shazeer
Oh, oh, we're just going to, um, like, lose money on every user and make it up in volume. (laughs)
- EGElad Gil
(laughs)
- SGSarah Guo
Oh, good. That's a good strategy.
- NSNoam Shazeer
No, I'm joking. No. (laughs)
- 26:51 – 35:36
What Noam looks for in new hires
- NSNoam Shazeer
people. Priority number one is just, just get it available to the, you know, to the general public. Um, you know, definitely. It would be fun to, like, uh, launch it as customer service bots when we're able. Like, people would just stay on customer service all day, like... (laughs)
- SGSarah Guo
(laughs)
- EGElad Gil
Yeah. You're like chatting with a friend effectively, so... Let's start with the customer support and... That actually happened apparently on some old e-commerce sites. Like, eBay apparently was effectively a social network really early on as people were buying and selling things and just kind of hanging out 'cause there weren't that many places to hang out online. So I always think it's kind of interesting to see these emergent social behaviors on different types of almost like commercial products or sites. But it makes a lot of sense.
- SGSarah Guo
So, you said one of the, I mean, one of the obvious reasons, um, LaMDA didn't ship immediately at Google was safety. Like, how do you guys think about that? Like, remember, everything characters say is made up.
- NSNoam Shazeer
Exactly, right. Like, uh, just let the, make sure the users are aware that this is fiction. If there's any, uh, anything factual that you're, that you're trying to extract from it at this point, it's best to go, uh, look it up somewhere that you find reliable. (laughs) Um, you know, I, I mean, there are other things, th- types of, uh, filters we've got there. Like, you know, we don't wanna encourage people to, like, harm, you know, hurt themselves or hurt other people or, uh, you know, we're, we're, we're blocking porn. Um, there's been a bit of protest around that. But, uh... (laughs)
- EGElad Gil
Yeah. And do you view all this as a path to AGI or sort of super intelligence? 'Cause, 'cause-
- NSNoam Shazeer
Um, sure, yeah.
- EGElad Gil
And is that part of the goal? For some companies, it seems like it's part of the goal, and for some companies, it seems like it's either not, e- it's ex- explicitly an end title or if it happens, it happens, and, you know, the, the, the thing people are trying to build is just something useful for people.
- SGSarah Guo
What a flex. AGI's a side effect.
- NSNoam Shazeer
Yeah. Well, I mean, that was a lot of the, uh, the motivations here 'cause, like, I mean, my main motivation for working on AI other than that it's fun... Well, I mean, fun is secondary. Like, the, the real thing is, like, I want to drive technology forward. Like, there are, there are just so many, um, you know, technological problems in the world that could be solved, like, for example, like, all of medicine. Like, there are all these people who die from all kinds of things that we could come up with technological solutions for. I would like that to happen, like, as soon as possible, which is why I've been working on, you know, on AI because, okay, rather than working on, say, medicine directly, like, let's work on AI, and then AI can be used to, uh, you know, to, to ex- accelerate some of these oth- other things. So, I mean, basically that's why I'm working so hard on the, the AI stuff. So, I wanted to have a company that was both, like, you know, AGI first and product first because, like, you know, product, product is great. It lets you build a company and, like, motivates you. And so, like, the way you have a, a company that's, that's both AGI first and product first is that you make the, make your product depend entirely on the quality of the AI. Like, the, the most, the biggest determining factor in the, uh, in the quality of our product is, is how smart the thing's gonna be. So now, we're like fully motivated, A, to make...... the AI better and, you know, a- a- and, and to make the product better.
- EGElad Gil
Yeah. It's, it's a really nice, um, sort of virtuous feedback loop because, to your point, as, as you make the product better, people, more people interact with it, and that helps make it a better product over time. So, um, it's a really smart approach. Um, how far away do you think we are from, um, AIs that are as smart or smarter than people? And obviously they're smarter than people on certain dimensions already, but I'm just thinking of something that would be sort of equivalent.
- NSNoam Shazeer
Yeah. I guess we, we just always get surprised at, like, what dimensions the AI (laughs) gets better than people. Pretty cool that some of these things can now, like, do your homework for you.
- EGElad Gil
Yeah.
- NSNoam Shazeer
I wish I had that as a kid.
- EGElad Gil
What advice would you give to people starting companies now who come from backgrounds similar to yours? Like, what are things that you learned as a founder that you didn't necessarily learn while working at Google or other places?
- NSNoam Shazeer
Oh, good question. Basically, like, you learn from, like, horrible mistakes. Um, but, like, um, ha- I don't feel like we've made really, really bad ones so far, or at least we've kind of, uh, like, recovered. Uh (laughs) . But, uh, uh, I guess, um... Yeah, just, like, build the thing you want, uh, really fast and, like, um, hire people who are just, like, really motivated, you know, to do it, you know?
- SGSarah Guo
Um, so one, one quick question just for users, like, what's the secret to making a good character? Like, if I'm gonna go make a copy of Elad instead of rubber ducking with myself, like, what do I need? Just, like, my text chat with Elad?
- NSNoam Shazeer
Oh-ho.
- SGSarah Guo
Yeah, stop disappearing in the chat, Elad. (laughs)
- EGElad Gil
I'm just trying to protect myself from becoming a character, you know? So...
- SGSarah Guo
(laughs)
- NSNoam Shazeer
(laughs) Yeah. Uh, I mean, so you can do it, uh, just as simply as, like, put in a greeting, a name and a greeting is, uh, is all you need typically for, um, you know, for famous, uh, like famous characters 'cause, like... or famous people 'cause the, um, the model probably already knows, uh, what, uh, what they're supposed to be like. Um, if it's, uh, you know, something that the model is not going to know about because it's, uh, uh, a little less famous, then you can, uh, create a, uh, an example conversation to, like, uh, show it, uh, what, um, you know, how the, how the character is supposed to act.
- SGSarah Guo
It's insane that Character is only 22 people. Like, you're hiring. What are you hiring for? Wha- what are you looking for?
- NSNoam Shazeer
Um, so, so far, 21 of the 22 are, uh, engineers. So, uh, we are, w- we're gonna hire more engineers. No, I'm (laughs) joking. We are, we are gonna-
- SGSarah Guo
(laughs)
- NSNoam Shazeer
... hire more engineers.
- SGSarah Guo
Shocked.
- NSNoam Shazeer
Um, you know, bo- both, uh, you know, uh, you know, in, uh, deep learning but also, like, uh, you know, front and back end, um, uh, definitely hire more pe- people on, like, the business and product, uh, side. Um, yeah, we've got a recruiter, uh, starting on Monday (laughs) . Um...
- EGElad Gil
Okay.
- SGSarah Guo
Hard requirement, burning desire or childhood dream-
Episode duration: 35:36
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode emCoG-hA7AE
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome