Inside image generation’s Renaissance moment — the OpenAI Podcast Ep. 19

1. 0:00 – 0:36

   Intro

   1. AMAndrew Mayne0:00

      Hello, I'm Andrew Main, and this is the OpenAI Podcast. On today's episode, we're talking about Images 2.0 with researcher Kenji Hata and product lead Adele Li. They'll discuss why the new model represents such a major leap forward, the evaluations that mattered most during development, and what people are creating with it now that it's widely available.

   2. ALAdele Li0:19

      [instrumental music] If DALL-E was the Stone Ages, Image Gen 2.0 is the Renaissance. It's not only great artistically and aesthetically, but it also incorporates, you know, science, art, architecture, all in one image.

   3. KHKenji Hata0:31

      We looked at it and we're like, "All right, this is better than Images 1." [laughs]

2. 0:36 – 2:27

   How Adele and Kenji came to work on Images

   1. AMAndrew Mayne0:36

      A- Adele, tell me a little bit about how you became a product manager here.

   2. ALAdele Li0:40

      So I joined OpenAI a little over two years ago, and before OpenAI, I was an investor my entire career.

   3. AMAndrew Mayne0:45

      Oh, wow.

   4. ALAdele Li0:45

      So I was in private equity and spent three years at Ve- Redpoint Ventures investing in AI and software companies. And when I first joined OpenAI, it was for a completely different role. I was thinking about, how do we build out our data and compute infrastructure? And over time, made my way over to the product side, and for the last six months have been working on Image Gen.

   5. AMAndrew Mayne1:04

      It, it's, it's interesting how you saw yourself going from one role then finding yourself into this space here, which is kind of cool, you know, to think about the idea that you have this sort of, you know, ability to be useful in different ways.

   6. ALAdele Li1:15

      Absolutely, and I think the role of a product manager is just to do the job that needs to be done-

   7. AMAndrew Mayne1:20

      Mm-hmm

   8. ALAdele Li1:20

      ... no matter what it is. And for Image Gen in particular, it's been really awesome to flex a lot of different muscles when it comes to, uh, building products, working with researchers like Kenji, but also thinking about, like, what is the gap in the market today that we wanna fill, and what is the opportunity that we wanna grasp here? It's not the same market that it was a year ago-

   9. AMAndrew Mayne1:40

      Mm-hmm

   10. ALAdele Li1:40

       ... when we first released Image Gen 1.0. Now it's a very different landscape. There are multiple image generation makers out there. Um, and ChatGPT is a very different company and, uh, product itself too, and so, um, really thinking about the evolution of Image Gen and its role within ChatGPT has been really, really exciting to me.

   11. AMAndrew Mayne1:58

       Kenji, how did you end up working on Images?

   12. KHKenji Hata2:01

       Uh, actually, like, when I first started at OpenAI, I also started about two years ago.

   13. AMAndrew Mayne2:05

       Mm-hmm.

   14. KHKenji Hata2:06

       Um, I was working on, like, some random audio project initially. Just-

   15. AMAndrew Mayne2:11

       Mm-hmm

   16. KHKenji Hata2:11

       ... this was my first project, and then at the time, I just found my way just working on helping them work on, uh, Images 1.0, the-

   17. AMAndrew Mayne2:20

       Mm-hmm

   18. KHKenji Hata2:20

       ... prior to the launch. Um, and so gradually I moved more and more onto the project, and then I became full-time on it, basically.

3. 2:27 – 5:25

   Images 2.0 launch reception

   1. AMAndrew Mayne2:27

      What has the reception been like right now for the model?

   2. ALAdele Li2:29

      In the last two weeks since we launched the model, usage is up more than 50%. More than 1.5 billion images are generated every week on ChatGPT.

   3. AMAndrew Mayne2:39

      Wow.

   4. ALAdele Li2:39

      And we've seen viral trends emerge across the world, um, all the way from trends in Asia for color analysis and stickers to US, where crayon and scribble are going viral. Um, but also a lot of people exploring emergent use cases, and I think it shows the dynamic range of the model but also how people are able to visually grasp the advancement of the model almost immediately. I think the visual, uh, communication and reaction that we've seen from our users, for them to say, "Hey, this is the best, highest fidelity, highest quality and aesthetic model that we've seen," has been really awesome.

   5. AMAndrew Mayne3:16

      This felt like a really big shift, almost worthy of maybe not even being, uh, uh, Images 2, but almost, like, just a new paradigm because just the capabilities are through the roof. What made that possible?

   6. ALAdele Li3:28

      When we started working on this project, I think we sat down and we discussed, what is the step change of capability and use cases that we wanted to build towards? Um, and we believe that image generation has the ability to do so much more than-

   7. AMAndrew Mayne3:42

      Mm-hmm

   8. ALAdele Li3:43

      ... it, what it does today. You could distill every single output, uh, or visual content that you see today into an image, and so that was the mandate that we sought out to improve. Um, and f- with this 2.0 model, we've improved on various different dimensions. Um, one is text rendering.

   9. AMAndrew Mayne4:01

      Mm-hmm.

   10. ALAdele Li4:01

       The ability for text on a page is so much better fidelity. The language and words actually make sense, and they're actual words. Um, the second of all is multilingual.

   11. AMAndrew Mayne4:12

       Mm-hmm.

   12. ALAdele Li4:12

       So we've really focused on making this model work in various different languages, and we're already seeing that people across the world in Asia and Europe are really resonating with these advancements. Um, the third is photorealism. I think we really saw a lot of feedback from our previous models that, uh, the output wasn't very realistic or altered their face or their bodies, and so one of our mandates was, how do we actually make the image feel like more like yourself? And so all the things that you think that the model knows, it does because it has imbued the knowledge of the world into, um, its conscience and is able to visually communicate that back to you as a user. And so putting that all together, I think we really get a state-of-the-art image generation model that is the best aesthetic model out there on the market right now, um, that really represents a new paradigm for image generation, um, which is a huge part of, I think, AI progress at large, uh, that, that we have an opportunity to work on here.

   13. KHKenji Hata5:12

       We often listen back, uh, listen to feedback on social media too.

   14. AMAndrew Mayne5:15

       Mm-hmm.

   15. KHKenji Hata5:15

       So we kind of just take all these things and basically are just aware of it and try to make sure that they're mitigated or completely fixed in some cases in, in the next iteration.

4. 5:25 – 9:34

   Productivity use cases and and 360 images

   1. AMAndrew Mayne5:25

      What kind of use cases are you seeing? What are you seeing people do with this now?

   2. KHKenji Hata5:28

      I think one that's particularly close to, like, the research team as a general is, like, infographics, text. Um, I think text in images is, like, so much better nowadays.

   3. AMAndrew Mayne5:38

      Mm-hmm.

   4. KHKenji Hata5:40

      So, um, I think it just opens up a lot more productive use cases, and at, from, like, the research side, we th- kind of think, you know, image generation used to al- always be about fun and maybe, like, unproductive things.

   5. AMAndrew Mayne5:53

      Mm-hmm.

   6. KHKenji Hata5:53

      But now we're really seeing steps forward into productivity and, uh, image generation for any type of use case that you can imagine it for.

   7. AMAndrew Mayne6:00

      So you mentioned text. I remember the early modelsUh, no disrespect to chimpanzees, but getting it to spell, like, OpenAI even looked like a chimp did it. And then now I'm looking at pages of text and finely detailed stuff, and I know that as models get smarter, variable binding, the ability to put things next to each other improves, but this was just a big improvement.

   8. KHKenji Hata6:21

      Yeah. But I don't think it's, like, completely unexpected.

   9. AMAndrew Mayne6:24

      Mm.

   10. KHKenji Hata6:24

       I think you, you see a lot of growth in between-

   11. AMAndrew Mayne6:27

       Mm

   12. KHKenji Hata6:27

       ... uh, well, first you see between DALL-E 3 and, you know, GPT Images 1.

   13. AMAndrew Mayne6:32

       Mm-hmm.

   14. KHKenji Hata6:32

       There was, um, if you ask for a grid of random objects, you, you go from maybe, like, five to eight in DALL-E 3 to maybe around 16 in Images 1, and then with 1.5, we went to about 25 to 36-

   15. AMAndrew Mayne6:49

       Mm-hmm

   16. KHKenji Hata6:50

       ... um, consistently. And I think now we could probably do over 100, I think.

   17. AMAndrew Mayne6:55

       Wow.

   18. KHKenji Hata6:55

       This is, like, a test that we might do internally is just, um, we just ask ChatGPT gen- give me a list of 100 random objects, right?

   19. AMAndrew Mayne7:02

       Yeah.

   20. KHKenji Hata7:02

       And then we just send that to our image generator and s- see how, how many are correct. And usually, you know, it'll get almost all 100 correct. Uh, and that's... But you see the, the constant growth-

   21. AMAndrew Mayne7:15

       Mm-hmm

   22. KHKenji Hata7:16

       ... over time. Um, so I don't think it's, like, completely unexpected. It's just a steady pace.

   23. AMAndrew Mayne7:20

       That was a test I used to use for, like, the really old models back with, like, Ada, Babbage, and Curry, like, list 100 science fiction books.

   24. KHKenji Hata7:26

       Yeah.

   25. AMAndrew Mayne7:26

       And then some of them would get, by the time it got to, like, 22, would just start repeating stuff-

   26. KHKenji Hata7:30

       Yeah, yeah

   27. AMAndrew Mayne7:30

       ... as it was, the model reached the end of it. So we've seen stuff too, like 360, 360-degree panoramas. How did that happen?

   28. ALAdele Li7:36

       Yeah, that really came from the emerging capability of the model, which is the ability to render images in any aspect ratio.

   29. AMAndrew Mayne7:43

       Mm-hmm.

   30. ALAdele Li7:43

       We discovered that people were generating really long, amazing panoramics, you know, skinny bookmarks as well, and one of the cool capabilities with the model is that not only were you able to generate images in this panoramic aspect ratio, but you'd also render images in the style of 360.
5. 9:34 – 10:51

   Viral trends, authenticity, and imperfection

   1. AMAndrew Mayne9:34

      funny too how one of the things that was trending was taking popular images or photos of people and then having the model make, like, kinda janky looking Microsoft Paint versions of them.

   2. ALAdele Li9:44

      Yes. Yeah.

   3. AMAndrew Mayne9:46

      And did you think that was something you would see, was that people are gonna use this [laughs] incredibly capable tool to then go make, you know, these silly looking things?

   4. ALAdele Li9:53

      Yeah. It's funny 'cause it takes a lot of intelligence to actually create something that is imperfect.

   5. AMAndrew Mayne9:58

      That's what I tell people all the time.

   6. ALAdele Li9:59

      Yeah. And it's definitely very interesting in the viral trends that we're seeing online right now. Um, one thing that I think people are really striving for is authenticity-

   7. AMAndrew Mayne10:09

      Mm

   8. ALAdele Li10:10

      ... imperfection, nostalgia. We're seeing that in the MS Paint prompt-

   9. AMAndrew Mayne10:14

      Mm

   10. ALAdele Li10:15

       ... crayons, um, all different kinds of generations that people are creating, and that really feels like the theme of consumers, is they wanna interact with AI in a very authentic, imperfect way. They wanna show their imperfections and use AI to help make them look good, but also show a more fun and goofy side of themselves, and I think that's self-expression via AI is something that we're really excited about. And, you know, I think it's really part of our mission as a company to make it easier for people to learn more and distribute that intelligence, but also, um, letting them express a version of themselves that maybe wasn't possible before.

6. 10:51 – 14:06

   Training breakthroughs and photorealism

   1. AMAndrew Mayne10:51

      Kenji, was there a moment with this model where you're saying to yourself, "Wow, I think, think this is ready to go"?

   2. KHKenji Hata10:57

      You know, as it's training, we take a checkpoint, and then, like-

   3. AMAndrew Mayne11:00

      Mm

   4. KHKenji Hata11:00

      ... we just sample from it, right? And just see, okay, how good is this thing? And I think, like, we just sampled a checkpoint, a model, uh, an image, and we looked at it and we're like, "All right, this is better than Images 1." [laughs]

   5. AMAndrew Mayne11:11

      [laughs]

   6. KHKenji Hata11:12

      Like, we were just like, "Okay."

   7. AMAndrew Mayne11:13

      I remember watching the iteration of one of the early versions of DALL-E-

   8. KHKenji Hata11:18

      Yeah

   9. AMAndrew Mayne11:18

      ... and how at first it was sort of the wispy, sort of weird, sort of the tendril sort of thing, and talking to one of the researchers like, "Is, is that gonna go away?" He's like, "I think two, probably two runs away from that." And then [fingers snap] just like that. The ability to predict that was amazing to me, and all of a sudden everything got crisp and clear.

   10. KHKenji Hata11:33

       Yeah.

   11. AMAndrew Mayne11:33

       And then also, like, looking at, y- you know, years ago I'd played with, like, you know, GANs and, like, doing those things. You'd, you have to squint and say, "I think it's a pickup truck," or something like that.

   12. KHKenji Hata11:41

       Yeah.

   13. AMAndrew Mayne11:42

       So it's interesting what you see as you say, "Okay, this just all of a sudden got much better." And-

   14. KHKenji Hata11:46

       Yeah. I mean, it was just very obvious. You just, you just take the early checkpoint, you just sample an image from it, and then you just sample an image from, uh, you know, Images 1, and you just look at the two, and you're just... There's just, there's-

   15. AMAndrew Mayne11:59

       Yeah. Why do I like this garbage? This is-

   16. KHKenji Hata12:00

       I forgot what the image was. It might have just been, like, a picture of, like, a woman at a sea- on the seaside, like-

   17. AMAndrew Mayne12:06

       Yeah

   18. KHKenji Hata12:06

       ... you know, overlooking a seaside. We just looked at it and we're like, "All right."There's like no, no question

   19. AMAndrew Mayne12:10

       Yeah, that was the big, the big-

   20. KHKenji Hata12:12

       [laughs]

   21. AMAndrew Mayne12:12

       ... the big jump was the photorealism of going from something that looked, that was more of a, a glossy, idealized magazine cover to something that looked like a really good photograph. So help me understand, like besides just more compute, how did this happen? How did you get a model that's much better and also that doesn't take an hour to generate an image? The times are still... I, I remember in the DALL-E days-

   22. KHKenji Hata12:34

       Mm-hmm

   23. AMAndrew Mayne12:34

       ... like we would literally have to, you know, "Tell, tell us what you want," and then an hour later it'd be on Instagram, to now these things are in ChatGPT, and it's faster. How is it getting both more intelligent and you're maintaining the same speeds?

   24. KHKenji Hata12:47

       I think we learned a lot, uh, in each release, like between 1 and 1.5, now 2.

   25. AMAndrew Mayne12:53

       Mm-hmm.

   26. KHKenji Hata12:53

       And so we take each, each of the learnings that we've made, and we've, you know... Like for example, speed, right?

   27. AMAndrew Mayne12:59

       Mm-hmm.

   28. KHKenji Hata12:59

       Um, you know, one of the things is like, "Oh, can we make the model more token efficient?" Or-

   29. AMAndrew Mayne13:03

       Mm-hmm

   30. KHKenji Hata13:03

       ... or, or something like that. And, uh, you know, we did a lot of work to make it, to make it pu- produce very good images with less tokens.
7. 14:06 – 22:16

   Evals, prompting, and creative control

   1. AMAndrew Mayne14:06

      you have any personal favorite benchmark tests you like to do, things you say, "I wanna see it make an image of this"?

   2. ALAdele Li14:11

      I have a eval that I call the me, me, me eval.

   3. AMAndrew Mayne14:14

      Okay. [laughs]

   4. ALAdele Li14:14

      It's essentially 100 photos of myself-

   5. AMAndrew Mayne14:18

      [laughs]

   6. ALAdele Li14:18

      [laughs] ... and my friends and my family. Um, and I put everyone in goofy positions. I have about a card or a birthday, um, for every single person. Um, and I think it's a really great eval in the sense that, uh, you only know the people around your, you know, faces the best.

   7. KHKenji Hata14:37

      Mm-hmm.

   8. ALAdele Li14:37

      Um, you also want to create funny things with the model and thi- do things that are relevant.

   9. KHKenji Hata14:41

      Mm-hmm.

   10. ALAdele Li14:41

       And so one thing for me as the product manager, um, that I'm testing is not only is the raw capability of the model really great-

   11. KHKenji Hata14:49

       Mm-hmm

   12. ALAdele Li14:49

       ... but also does ChatGPT understand what I want in that context?

   13. KHKenji Hata14:53

       Mm-hmm.

   14. ALAdele Li14:53

       You know, ChatGPT remembers, you know, that I have a brother, that I have a mom and dad, um, and what they like to do. And so does the model accurately know how to insert pieces of personalization in the moments that matter in the images? These are things that I'm testing for.

   15. AMAndrew Mayne15:10

       How about you?

   16. KHKenji Hata15:11

       Besides the grid one I mentioned earlier, that's probably the one I've used the most. For a while, I think Divya and I were doing a lot about photorealism. [laughs]

   17. AMAndrew Mayne15:19

       Yeah.

   18. KHKenji Hata15:19

       We were trying real hard to push on that. Um, uh, just basically, I know Divya's favorite one was, like a woman holding an or- a jug of orange juice. I don't know if you've seen that.

   19. ALAdele Li15:30

       Yeah. [laughs]

   20. KHKenji Hata15:31

       [laughs] There's like so many images of a woman holding a jug of orange juice. Um-

   21. ALAdele Li15:35

       Well, actually feel like the researchers had a more standard set of images like, than they like to-

   22. KHKenji Hata15:39

       I think so too

   23. ALAdele Li15:40

       ... to lead on.

   24. AMAndrew Mayne15:40

       Yeah, and you get like the standard, can it do somebody writing with their left hand and a watch on their right hand and a clock-

   25. ALAdele Li15:45

       The clock

   26. AMAndrew Mayne15:45

       ... showing this. I think the big, the big leap of the image is like probably 1 or 1.5 was like a half-full glass of wine.

   27. ALAdele Li15:52

       Hmm.

   28. KHKenji Hata15:52

       Or the wine glass full to the rim?

   29. AMAndrew Mayne15:53

       Yeah.

   30. KHKenji Hata15:54

       Yeah, yeah, yeah.
8. 22:16 – 22:27

   Creative agents and what comes next

   1. AMAndrew Mayne22:16

      How do you see the progression of this? This is great, but typically any time I talk to somebody at OpenAI about what they're working on, they're like, "Yeah, this is good, but..."

   2. ALAdele Li22:23

      I think we're still super early in exploring all the different use

9. 22:27 – 28:08

   Images + Codex

   1. ALAdele Li22:27

      cases that people are really trying to push the model with. Um, and so one of the things that we're really excited about, um, is what is that next, um, stage for Image Gen, um, which is to create the creative agent.

   2. AMAndrew Mayne22:39

      Mm.

   3. ALAdele Li22:40

      Ultimately, the agent that can work alongside you, be your creative assistant, um, and really understand how you work, what your preferences are, what is the output that you wanna get to, um, and build the product and model ecosystem that helps users kind of have a personal interior designer, personal architect, um, personal, you know, wedding planner, et cetera, all in one Image Gen.

   4. AMAndrew Mayne23:05

      I'll tell you another thing that was kind of amazing was like, um, I write books, and so, like, every now and I have a book come out, I've gotta change my social media headers. And I just went, and I said, "Oh, find my book cover and write, you know, create a, a po- you know, create appropriate size social media header that I can put on X or Facebook or whatever." Like, let's see. First shot. First shot. Right aspect ratio, everything.

   5. KHKenji Hata23:24

      We basically did that from the start or trained the models-

   6. AMAndrew Mayne23:26

      Mm

   7. KHKenji Hata23:27

      ... to be good at that from the start. I remember, like, I worked on the initial de-risks of-

   8. AMAndrew Mayne23:32

      Mm

   9. KHKenji Hata23:32

      ... of ev- basically it could do any aspect ratio that you ask.

   10. AMAndrew Mayne23:37

       Yeah.

   11. ALAdele Li23:38

       Yeah, you can now, um, really just easily specify the outcome that you want.

   12. AMAndrew Mayne23:42

       Yeah.

   13. ALAdele Li23:42

       Like in the case of yourself, you're like, "I want promotional material."

   14. AMAndrew Mayne23:45

       Yeah.

   15. ALAdele Li23:45

       "I don't have an idea. I didn't specify exactly what I wanted." But the model was able to do the research and then give it to you in the style and aspect ratio that was relevant to you, and that's super powerful. We're already seeing this. Um, you know, you're, you're an author. I've talked to real estate agents who are using Image Gen to help them create listings for their apartments or stage their listings.

   16. AMAndrew Mayne24:08

       Mm-hmm.

   17. ALAdele Li24:09

       Um, YouTube creators have talked to me about using Image Gen for their thumbnails and-

   18. AMAndrew Mayne24:13

       Mm

   19. ALAdele Li24:13

       ... promotional content. I've talked to-Top artists who wanna use Image Gen to connect with their fans, and I think the ability for all different kinds of professions to start to use Image Gen to help them with visual creation is super powerful, especially if you're working in a visual and a creative industry.

   20. AMAndrew Mayne24:30

       Mm-hmm.

   21. ALAdele Li24:31

       Image Gen is such a hack in your professional toolkit. I think it has to be a part of everyone's everyday workflow in the future.

   22. AMAndrew Mayne24:37

       This does feel like the... I think it feels like the first time where anything I can reasonably come up with, it does a pretty good job of it.

   23. ALAdele Li24:44

       We think it's a new paradigm for-

   24. AMAndrew Mayne24:45

       Yeah

   25. ALAdele Li24:45

       ... image generation altogether. Like, if, you know, we said this in the launch video, if DALL-E was the Stone Ages, Image Gen 2.0 is the Renaissance.

   26. AMAndrew Mayne24:53

       Yeah.

   27. ALAdele Li24:53

       Um, and I think that is so true because the model, it's not only great artistically and aesthetically, but it also incorporates, you know, science, art-

   28. AMAndrew Mayne25:03

       Mm

   29. ALAdele Li25:03

       ... architecture, all in one image together, and I think that composition, um, and knowledge that the model has just means that the outputs are so much more trustworthy, um, are more powerful-

   30. AMAndrew Mayne25:14

       Mm-hmm
10. 28:08 – 29:21

    Prompt tips

    1. ALAdele Li28:08

       with Image Gen.

    2. AMAndrew Mayne28:09

       Any, any parting prompt tips for people?

    3. ALAdele Li28:13

       Well, one of the things I would suggest people try is Image Gen Thinking.

    4. AMAndrew Mayne28:17

       Okay.

    5. ALAdele Li28:17

       So if you navigate to the Thinking or Pro models-

    6. AMAndrew Mayne28:20

       Mm-hmm

    7. ALAdele Li28:20

       ... we have a more powerful version of Image Gen in that experience, and in that model, uh, you actually are able to search the web, analyze files, um, leverage tools under the hood, um, which then yields a better quality and higher composition photo. And the suggestion that I have for prompting that experience is be open-ended.

    8. AMAndrew Mayne28:42

       Mm-hmm.

    9. ALAdele Li28:42

       I think the model will go and do the exploration itself to understand and try to reason, um, and find information that matters. And I also think giving it a sense of an aesthetic is also super helpful. Um, using, grounding that in a style has been really, um, fruitful for a great result.

    10. AMAndrew Mayne29:01

        Good one. Good one.

    11. KHKenji Hata29:02

        I think just being very particular about the style or, like, what you like in general. Like, for me, I like minimalist infographics.

    12. AMAndrew Mayne29:10

        Mm-hmm.

    13. KHKenji Hata29:10

        Sometimes I think the model can be a little dense.

    14. AMAndrew Mayne29:12

        Mm.

    15. KHKenji Hata29:12

        And so I just... Maybe I'm just a simplistic kinda guy.

    16. AMAndrew Mayne29:16

        [laughs]

    17. KHKenji Hata29:16

        So I just like very th- very clean, a very clean look, so I like that.

    18. AMAndrew Mayne29:20

        Adele, Kenji, thank you very much.

Episode duration: 29:22