Unlocking Creativity with Prompt Engineering

1. 0:00 – 1:41

   Introduction

   1. GPGuy Parsons0:00

      It's still quite hard to describe things with words. Designers, yeah, when they go and work-- when they do work for clients, like it's one of their pet peeves because clients are like, they don't like it, but they can't explain why.

   2. SSSteph Smith0:10

      With every new technology, some jobs are lost while others are gained. And while people often focus on the former, in this episode we're highlighting the latter, a highly creative role that emerges alongside AI, the prompt engineer. Until AI can close the loop of its own, each tool still requires a set of prompts. And just like a composer feeds an instrument a set of notes to play, a prompt engineer feeds the AI a map of what to produce. And if we know anything from music, it's that composing great music takes great skill. So in this episode, we dive into the emerging importance of prompting, the early learnings and how to do it effectively, and also where this field might be heading. And we do so with Guy Parsons. Guy has been an early mover in the text-to-image AI space, having written the DALL-E 2 Prompt Book in July of last year. So will the prompt engineer be more like the highly sought-after DevOps engineer or a proficiency like Excel that you find on every resume? Listen in to hear Guy's take. By the way, we're thinking of running a prompt competition coming up, so if you think you have what it takes, email us at podpitches@a16z.com with the subject, prompt engineer. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures. [upbeat music] Guy, welcome to the show.

   3. GPGuy Parsons1:40

      Thank you for having me. I'm excited

2. 1:41 – 5:33

   DALL-E 2 Prompt Book

   1. GPGuy Parsons1:41

      to be here.

   2. SSSteph Smith1:41

      Let's take a step back. When we originally reached out to you, it was around six months ago, and you had just written something called your Prompt Book. Why don't you give everyone a little bit of an idea of what that Prompt Book was, what it is now, and also what prompted you to want to write it in the first place?

   3. GPGuy Parsons1:58

      This was in the in-initial heyday of DALL-E 2, which was like OpenAI's text-to-image model. They rolled it out to a few test people at a time. They were super cautious about how it might be misused, how it kind of could end up kind of having a backlash, all these kinds of things, which then only increased the sense of people wanting to get their hands on this thing, 'cause at the time this was pre-things that you might think of now as stable diffusion. Midjourney kind of predated those by some small margin and seemed way ahead of anything people had kind of tried using before. If you've used a text-to-image AI by now you know it's basically a text box, [chuckles] and it all comes down to what you type in. It doesn't have, you know, buttons and, you know, the kind of controls you might expect when you like lock into something like Photoshop. So the question then becomes like a lot of people, [chuckles] one's mind goes blank or you don't actually know the name or the words of like what you're trying to type in, right? So, uh, if you've actually been to art school or you're up on your art history or your design language, then you've probably got a head start on everyone else. But on places like Twitter and, and Reddit, there are people posting like these amazing images. But because of the nature of social media, it's all lost. So I started trying to like collect these cool examples and these cool kind of terms people were using to create these like amazing visual effects. So I, I started putting everything in a, essentially like a, a slide deck. By the time I'd copied and pasted all these cool things I'd seen, there was eighty, a hundred slides long, something like that. So then I rather grandly called it a book [chuckles] and shared it, shared it online, and it's just a jumping-off point for people to realize the kind of stuff at the time that these were just about becoming capable of. Obviously, now the capability is, uh, even more [chuckles] advanced.

   4. SSSteph Smith3:42

      Definitely. And we'll get into that because within six months it's crazy to see how these tools, the way people are using these tools, how that's all changed in a matter of, again, just six months. It feels like yesterday when we didn't even have access to this. But this idea that these are tools and just like any other tool, person A versus person B may not get the same result. They may not have the same understanding of how to leverage the tool. And so before we get into maybe the tips and tricks that you've learned, I just wanna give the audience a broad sense of how much time you've spent within the bowels of Midjourney, DALL-E, Stable Diffusion. Like if you could give an estimate, how much time do you think you've spent kind of mastering this idea of prompting?

   5. GPGuy Parsons4:27

      I wouldn't say I'm a master in it [laughs] in, in, in any sense. It's like so engaging and interesting to like experiment with these tools. So, you know, like in, in the last six months, sure, like a couple of hundred hours. What I really admire is people that are using these tools to kind of create this kind of like real body of work where they kind of, they really like pursue a direction to like discover what's possible. I think I saw a thread where in a, I think it's in Midjourney, you can get it to tell you how many p-prompts you've ever done, and there were people in the thousands, hundreds of thousands.

   6. SSSteph Smith5:00

      Yeah. And I appreciate how humble you are, but I think it's, it's one of those scenarios where, again, we're six months in, so it's kind of, y-you know, a parallel is when there's a new coding language and then you see people write job descriptions for developers looking for someone with five years experience when that particular language has only been around for six months or a year. And so, yes, I don't think anyone could definitively say they're an expert in prompt engineering, partially because it's only been around for so long. But I do think you've at least shared a lot more than, you know, the average person in this space. And given your experience with these

3. 5:33 – 6:58

   Parallel skills

   1. SSSteph Smith5:33

      tools, I'm curious if you see a parallel skill set, uh, where you can kind of compare prompt engineering to. Is it similar to learning to code? Is it similar to being able to story tell effectively? Is it similar to being able to process numbers in an Excel sheet? Like is there a, a parallel skill set where it reminds you of, you know, something you've done before?

   2. GPGuy Parsons5:56

      I think there was an era, I don't know if we're still in it, where there was a certain category of person who could con-consider themselves like good at googling stuff. [laughs] Do you know that there's this thing where you can t- type in like file type this and find this thing? There's this big debate over whether, like especially in text-to-image, you know, is it really that... Is there really, like any artistry to it? And I-- for me, like I'm not so sure 'cause I'm, I'm no artist, but there's definitely something. It's almost about f- It's like discovering an image that's already out there. You've just gotta find the, the words that summon it forth as if you're kind of navigating like an infinite Pinterest of things that haven't quite existed [laughs] until you manifested them.

   3. SSSteph Smith6:33

      Well, I mean, to that point, like we have so much information online. I feel like that is a skill set even s- before these AI tools. Like, I used to work on a product called Trends, and that really was about using the right tools like subreddit stats or Ahrefs or other data sets online and learning to parse them and learning to surface what other people find interesting. Um, but let's get into the nitty-gritty. Like, you wrote this prompt book, you've been playing around with these tools for quite some

4. 6:58 – 10:22

   80/20 prompting

   1. SSSteph Smith6:58

      time. Are there certain learnings, maybe the 80/20 approach of becoming a good prompt engineer in terms of things that you think are, are really valuable to understand? Maybe it's the prompt length, maybe it's using certain modifiers within your prompt. Maybe it's just like a, a framework for thinking about prompting. Is there anything that's surfaced that you think would be really valuable to someone who's just starting out with prompting?

   2. GPGuy Parsons7:22

      Oh, yeah. Like, I think if you've never used one before, like the best way to explain how they work is to always like describe something as if it already exists. Imagine that it's an image in some kind of downloadable clip art library or it's, it's on, it's on like a photography gallery and, you know, someone's written underneath, "Oh, this is a fine example of a modern, uh, photography shot." And those are the kind of descriptions that you're trying to kind of mimic to tell these kind of tools what you're looking for. And it also gives it like a natural sense of why these tools are bad at some things and, and the kind of prompts that don't really work because you never, you know, if there's like a, let's say some, uh, like an archive image of some women celebrating on the steps of a church in 1972, it will have that kind of caption where they never go, "The woman on the left [laughs] is wearing a yellow hat. The woman on the right is wearing a re-" You know, they just don't go into that kind 'cause you can see it. So ironically, they often describe very generally what the image is about, but not like how you would draw it step by step. And that's why these tools are, are less good at, at, at saying like, "I want this thing over here, and then that thing next to it, and then something on top, and that thing should be much bigger," because that's in the real life, that's not how images are described in language, so.

   3. SSSteph Smith8:45

      That's a really good point though, because that's how these AIs were trained, right? So I think DALL-E trained on 600-plus million images, and they're using that alt text from online of saying, again, that descriptor. And I've never thought about it that way, but actually training yourself to become a good prompter by reviewing the inputs to the tool, which I, I've never done this before, but I can imagine someone literally going online and reading the alt text on different images and going, "Ah, this is how this was described. This is how an AI might interpret my future prompt." And I think to your point also, something that I've learned from my very limited set of prompting is just the level of detail that you need with your prompt, where when I first started, I'm like, you know, monkey wearing a hat. You don't even realize until you start prompting the many iterations that could come from that. Like, you have one image in your head, but then you get back all of these different results, and then you, you end up looking on different prompt search engines or libraries and seeing what other people are doing, and you're like, "This prompt is like 200 words." [laughs] Like, I would've never thought to do that.

   4. GPGuy Parsons9:48

      For now, yeah. And I think there's something to be said, like I think the longer they are, the, the le- there's definitely like diminishing returns.

   5. SSSteph Smith9:56

      As I went through your prompt book, there were so many different ways that you could describe a shot. You could say a different camera angle, you could say a time period as you just spoke to. You could say a specific type of artistry or, or even a specific artist. I know there's some controversy around using specific artists' work to tr- to train your new images. But let's look forward to today. I feel like as we talked about, six months later, these tools have evolved

5. 10:22 – 13:00

   New ways of prompting

   1. SSSteph Smith10:22

      a lot. Are there any different ways that you can prompt today or kind of leverage these tools that didn't exist six months ago that are really important in maybe extending the way that you can use them?

   2. GPGuy Parsons10:35

      100%. So the main one, and these things are like changing all the time, right? So but now there's increasingly tools where you can prompt like with an image. Again, that's almost like an entire new field of exploration because it's not combining the image with the other w- with like your words in the way you would expect something like Photoshop to do it. Like, it's not collaging them together. So the results can be really surprising, really unexpected, probably quite difficult to control. But then you potentially have interesting opportunities like, okay, I can make a load of kind of abstract kind of stuff using like my brand colors or something that's important to me, photos of me, who knows. And then, yeah, and then I'm gonna use that and kind of multiply that baseline, that kind of visual base with custom other prompts, and then everything will have this kind of likeness. And then of course, like the big thing that happened since the days of The Prompt Book and so on was of course that huge spike in interest in, um, selfies, right? Like the lenses and the profilepicture.ai, and there were like a dozen of them, which was just prompting with your face basically. [laughs] Um, prompting with 20 selfies and being like, "Yeah, I wanna see more of this guy," 'cause it's me obviously. And then within the image-to-image Space, you've now got other startups that are doing interesting things where like, okay, give us 10 core images, and that will generate you, like, infinite versions of that based on, like, the modifiers that you wanna see. So there's all kinds of... That's l- a really interesting space that's gonna probably power, like, the next generation of how people, especially consumers, like, interact with these products.

   3. SSSteph Smith12:13

      Yeah. One way that maybe you could put it is that when we first got access to these tools, you were really starting from scratch. You didn't even have the prompt libraries available to you. You were just like, "Okay, I have this image in my head." But today, you not only have those libraries, you also have images that you can input, so you're not starting from scratch. You have a baseline of, as you said, maybe it's brand colors, maybe it's a certain style, and ins- instead of having to articulate that yourself, you can just say, "Hey, here's what I want." But to your point, sometimes it's hard to control, right? Because you're, you're trying to say something to the AI. You're trying to say, "I want this output." You don't always get it. And so something I wanna ask you about is how you've learned to rein that in, to really get a higher throughput of images that you want versus images you don't want

6. 13:00 – 18:13

   Pulling the AI slot machine

   1. SSSteph Smith13:00

      over time, because something that's interesting about these AIs is that they are a little bit of a black box. You can't understand every little piece that went from your input to your output, and so you can't, like, fine-tune it in the same ways as maybe some other skills that we've learned in the past. And so how have you learned to actually become a better prompt engineer given that black box nature?

   2. GPGuy Parsons13:22

      I mean, I think another aspect is you and I both type in the same thing, it's not gonna make the same, [laughs] the same picture because it kind of starts from this random cloud of noise, and your cloud of noise is different to mine, and then it's slowly turning these clouds more and more into something that looks like a orangutan in a tuxedo. So that's really f- frustrating when you're, like, testing things because was it good or did you just get lucky? Or alternatively, if you're not seeing what you expected, should you just hit it again and again? Like, and then when you see someone else has made something really cool, did they do something really clever or did they just... You know, is it, like, a persistence?

   3. SSSteph Smith13:59

      I have found myself in that exact spot where I, I have an idea for what I want. It's not something that is super important where I need to nail it. So I'm just-- I just need it close enough, and I'm, I'm getting these results, and they're getting a little closer and closer and closer. But I have found myself in that spot where I'm just like, "Let's just generate it again." Like, if I do this enough times, [laughs] I'll eventually get to something that's workable. So do you have any thoughts there in terms of, like, how you don't end up in that spot where you're just, like, hoping for a better image? You're kind of, like, pulling the AI slot machine, if you will.

   4. GPGuy Parsons14:33

      Uh, no. I mean, unle- I think unless you've seen, unless you kind of have evidence, and I think over time there will be more, and I think it's why some of these, like, other tools and communities are so important, you know, where you see lots of other people's work is, you know, if you can see someone else has done it, then you're like, "Okay, I, I know it's..." [laughs] May- I- ideally, you can also see the prompt they used and work out how they did it. But even if not, then you're like, "Okay, I can get there." Also, you run into these things where you would think it's, like, the most simple thing, and then you're like, it doesn't know what a hot dog is. Like, [laughs] it just doesn't understand the rules of how-- of, like, what j- j-- you know, physically what can and can't that look like. And you're, like, trying, and it's like now the sausage is a right angle, and the, the bun has ears 'cause it's starting to throw in some, like, dachshund like, you know, aesthetics. And then you're like, "And minus, minus. No, no dachshund." That's kind of the limitation of where the technology is at the moment, which is it's amazing until you're trying to do something very specific, and especially if you wanna do something very specific just also to, like, a very high, like, or professional standard.

   5. SSSteph Smith15:44

      Well, I'm glad you even mentioned the negative queries. That's something I think a lot of people don't know is that you can, you can say, "Hey, AI, I don't want this." It doesn't always manage to still generate what you're looking for. But there's also almost, like, these glitches. One of them that is kind of infamous now is hands, right? So you can generate these beautiful images of these Instagram-looking models, and you can put them in all these different backgrounds, and you're like, "Wow, this is amazing." And then [laughs] it's always like, "Well, look at the hands," which is kind of funny. I feel like it's a... It's, like, the perfect manifestation of how technology always is, like, much better in one direction when it's invented, but there's always, like, these things that need to be iterated on. And so are there other things worth knowing about, whether it's these negative prompts, whether it's [laughs] these glitches that are still in the matrix? What would you call out from your, again, many hours of being deep in, in these tools?

   6. GPGuy Parsons16:37

      I think it depends on the model. One example was when DALL-E came out. It's not very good at understanding that it's drawing things in a square. So if, uh, if, if you're drawing a person, it's often gonna have, like, its feet and its head cut off 'cause it's, it's seeing those in portrait photos. But one thing you could do with DALL-E is you can actually upload, like, an image to, like, do variations of, and if you upload an image that's just, like, a little white border, um, then it knows that nothing can go there, and that, that kind of encourages it, forces it to kind of think inside the box, if you will. But then, of course, you have now tools like Midjourney who've been, like, it- iterating on their text-to-image model, like, a lot more aggressively than OpenAI, who understandably, I think, maybe have some other things in the-

   7. SSSteph Smith17:19

      [laughs]

   8. GPGuy Parsons17:20

      ... in the s- in the cooker, you know, which have, have now grown that into the model itself. So when you type things in, it knows it's a square, and actually it will sometimes do quite clever things in order to fit it in that space. So if you ask for kind of like a group selfie of three people, you know, on something like DALL-E, that's gonna be cut off at the end 'cause you- it's used to seeing someone taking, like, a disposable camera photo. Whereas Midjourney's clever enough to know that one of them kind of needs to be standing behind the other or, like, leaning in from the side. So it's kind of clever how they've managed to, like, solve that composition problem, um, within the AI, which then I think, you know, the prompt engineering thing I think is just understanding the possibilities and the limitations of where you are at the moment. Meanwhile, there's these other people who are doing some, like, very technically serious work to kind of make those limitations kind of no longer, uh, relevant.

   9. SSSteph Smith18:10

      Yeah. Well, I'm glad you brought up the differences between

7. 18:13 – 21:03

   Comparing models

   1. SSSteph Smith18:13

      these different tools. So if we talk about just Stable Diffusion, Midjourney, and DALL-E, I feel like are three that a lot of people are familiar with. Would you liken the ability to prompt within each of these more like the difference between Excel and Google Sheets, where if you know how to use Excel, you really can drop right into Google Sheets and it's relatively straightforward? You might have to switch up your shortcuts a little bit or learn one little thing here and there. But for the most part, you can again drop from one to the other. Or would you liken them more to learning to speak different languages?

   2. GPGuy Parsons18:46

      It's not that different. I think the principles are, like, very similar, and then the nuances are slightly different. So I think now if you went from, like, like, DALL-E to Midjourney, it would be, like, amazing, and then if you went back the other direction, you'd be like, "It doesn't do what I want." But that's because Midjourney is doing so much of the heavy lifting to help you make something really good. If you are using the tools to create some, like, very specific effect, kind of l- like, imagine, like, I guess, yeah, like a very complicated Excel formula, that would not have the exact same output in the other tool because they're trained on, like, a different set of images. Stable Diffusion I think some 5 billion, and then, like, a smaller set of, like, 12 million for the what does nice look like. Fine-tuning that's happened on the top and how they've optimized it. You know, they, they, they have made different, like, creative decisions there. It's maybe like driving a different car. Like, if you [chuckles] if you, like, floor the accelerator in different... and then do this in various different cars, some are gonna take off, some are gonna trundle along.

   3. SSSteph Smith19:47

      It's a good analogy. Do you also find that... I mean, we've talked already about this idea where sometimes it's, it's pretty easy to get to that 80%, but then that final 20%, the r- real refinement to get to exactly what you pictured in your head or exactly what you want and didn't picture in your head sometimes requires another tool. And so have you found... I've heard some people are using FaceTune or different AIs to take it to the final level, or I guess you could also use inpainting and outpainting a little more discreetly. So how have you found the relationship of maybe one tool to the suite of other tools that exist out there?

   4. GPGuy Parsons20:22

      I think there's lots of exciting crossovers. But actually, I kind of think it's a big opportunity for, like, the Photoshops of this world, because those are tools that presuppose you have some kind of original image to be, to, like, manipulate. Whereas now there's that, that... there's a huge amount of, like, raw but maybe not perfect material there for people to work with. There's lots of things also that I've been trying to, like, do in prompting that are actually, like, more easily achieved in other tools. So you can, you know, spend ages trying to get this kind of vintage film look. But if you're, like, an Instagram influencer, you, you will know, which I'm sure you are, um, who isn't? Um, but [chuckles] there's loads of, there's loads of iPhone apps, right? That are out there just to, like, give all your photos that kind of, like, dreamy,

8. 21:03 – 27:44

   Requested features

   1. GPGuy Parsons21:03

      you know, vintage film look.

   2. SSSteph Smith21:04

      Yeah. I mean, I think back in July when you first wrote your prompt book, you had a requested feature list for DALL-E 2, and I feel like potentially some of those have been implemented. I haven't cross-referenced them. But are there things that are on your new list of, "Hey, these tools are great, but they're missing X, Y, Z, or they're lacking in these areas. This would be top of my list to see improved on"?

   3. GPGuy Parsons21:31

      You know, I think we're gonna see, like, more models come out. I, I mean, the fact that Stable Diffusion is kind of open source means that lots of other things are gonna be built on top of that, and I think it's gonna be really exciting to see some of the directions that people take that in, either kind of on a individual sort of prosumer level, people building their own models to create their own stuff, more likely some bigger organizations, you know, training it for, like, specific purposes. The whole challenge and the whole opportunity, I think, at the moment is, like, how do you go beyond the text box? How do you go beyond this, like, just blank rectangle to create something that is more user-friendly, that's more inspiring, that's more how people think? Because on the one hand, if you're, if you're not an artist, the ability to describe things with words is definitely a big step forward. But if you think about the next layer is it's still quite hard to describe things with words.

   4. SSSteph Smith22:28

      Yeah. It is.

   5. GPGuy Parsons22:28

      Designers, yeah, when they go and work, when they do work for clients, like, it's one of their pet peeves because clients are like, they don't like it, but they can't explain why or what they want different. They're like, "Oh, I want it to be more... Do you know what I mean? Like more..." And they're like, "I don't know. I don't know what that means." Which is basically the position these, you know, AI models are in. So I think you're gonna see could you see, like, a conversational interface? Can, can you do the generations fast enough that you're always showing people multiple options, possible new directions? Almost like in a sort of multi-dimensional space where it's like, do you wanna take it more this way or more this way? Part of the prompt book is I didn't know what, you know, metaphysical painting or Kodachrome, all these things were, but those at least have names. You know, once you know the name, you're like, "Oh, that, that thing. Okay, it's, it's cool. I didn't know it was called that." But there's probably other aesthetics, right? Other styles that we don't have wor- actually words for. It's like, you know that kind of gritty but, like, modern gritty, like, almost like shiny gritty. Like, the grit has a shine on it. And probably I can make you a mood board of that, and you'd be like, "Oh, yeah, like, that's a thing." But there's no word for it. If you can create ways of unleashing the inexplicable, the, uh, the undefinable, the, the... which is what in, at the end of the day, like visual art, that's the exciting thing about visual art, is to express things or moods or things that you can't quite put into words.

   6. SSSteph Smith23:50

      You totally have my mind spinning thinking of different ideas. A couple of them that came to mind, one of them is just a better onboarding experience, but one where you're guiding the new prompter to understand how all these things might fit together, to your point. Like, try this. Oh, look at what you got here. Oh, did you notice how when you use these two prompts together, this one kind of overshadows the other? Maybe there's a third word that's a synonym of this, and I think you've kind of done this on your own by just going through and prompting like crazy, going through these different prompt libraries and trying to sort through the signal from the noise. But I do think any one of these models or maybe something built on top of them could have just a much better onboarding experience. And then I also, in terms of these visual styles, I mean, it, it reminds me of, you mentioned Instagram. A lot of Instagram influencers for a period of time were selling these filters because they had figured out the, the precise tuning of every little variable, which sounds easy, but I had tried to do it myself. I, I never managed to create good Lightroom filters, but people had, and they, they would sell them. And so I wonder if you'll see the same thing where maybe someone creates a zip file [laughs] of a moo- mood board, and then they train the AI in some way that does make it, I guess, play nice with that particular concept that you can't distill necessarily into a single term.

   7. GPGuy Parsons25:12

      Someone did a paper on it, and I think it's almost kind of what led to that selfie craze, which was you don't need to put photos of Steph in that original 600 million training data or wait for the next time we do that again for it to teach it what you look like. There's this kind of, um, embedding trick where you can show it, like, a bunch of photos of you, and then you can refer to you as... And it knows how to kind of recreate that. And there was also an interesting thing in the same paper but hasn't really been used or, like, commercialized in the same way, which is to do that with style. So rather than show it, yeah, like, this is what this person looks like, it's like this is what the style of blah, blah, blah is called. Here it is, and then off you go. Which obviously has all kinds of potentially shady legal connotations. Um, but let's assume this is all lovely art we've made, um, ourselves.

   8. SSSteph Smith25:58

      To the idea of honing in a style, I do wish there was a version of the product where I could go and, like we've talked about, maybe upload certain brand images or certain brand colors, and then have it iterate with me where it shows me a bunch of images and I say, "Uh, it's okay, but I want a little more of this color. I want it X, Y, or Z." And then we keep doing that to the point where I get a bunch of images where I'm like, "Yes, this is the style." You can lock that in. You lock it into a variable that you can then plug into future, um, prompts.

   9. GPGuy Parsons26:28

      There's some people out there that have like-- Some people that have been, like, sharing it, just doing themselves. They've, like, managed to, like, lock in, like, a particular look, and now, like, every blog post they have on their lovely, you know, content marketing thing is, like, all... is always the same kind of thing, and that's, like, pretty cool. But we haven't seen that always built into the, like, foundation models yet as, like, a way of interacting with it. And then there are some... a coup- like a few startups, like, um, Scenario, which is doing it for, like, game assets, and then, like, Leonardo, which is, like, more multipurpose, I think, or is just positioning itself that way, which, yeah, is again all about can you, like, control things down to, like, a consistent look.

   10. SSSteph Smith27:06

       What we've talked about so far is this idea of controlling the AI, but I also like to think about the ways that when you work with these different models, you learn more about whether it's art, learn more about your own creativity or how you think about these, these images. So the example that it reminds me of is in chess when we finally built the bots that were better than humans in chess. Not only were we surprised by the fact that that could happen, but we were also surprised by all of the different openings or moves that humans in their thousands of years playing chess had never considered that were better than some of the

9. 27:44 – 28:44

   Learning with AI

   1. SSSteph Smith27:44

      moves that even the best chess players in the world had used. And so have you seen any of that even from a personal experience level, like, where you're in these tools and you're playing around and you-- again, you're, you're learning with the model, if that makes sense. It's, it's almost surfacing things that you had never considered before.

   2. GPGuy Parsons28:01

      I like that. I think, I think what you're touching on is this, whenever you're using these tools, you're-- you have these two modes, right? Where you're either, like, waiting to see what it shows you, or you kind of are visualizing it in your mind and you're like, "No, not that. Not that." But if you just let it take you where it wants to go, then you're suddenly like, "I have no idea what I'm looking at, but I'm-- apparently I'm here." You can do all kinds of... With, um, DALL-E, there's, like, this variations tool. So you just get it to, like, show an image, and it'll be like, "Here's four more that are kind of the same." But obviously over time it starts to, if you leap and leap and leap and leap, you end up on this, like, com- completely bizarre visual journey. It's very psych-- It's like a, yeah, like a psychedelic dream.

   3. SSSteph Smith28:43

      It's fun to

10. 28:44 – 32:08

    Practical use cases

    1. SSSteph Smith28:44

       play around in these tools, but ultimately there-- while there is a market for just interesting art in the world, a lot of this will need to ladder back into, you know, whether it's blog post sharing images, whether it's creating, you know, the next sneaker design that you end up selling. Are there areas that you've seen really emerge from this where people are using these tools today and applying them to, again, what someone might call a practical use case? And in addition to maybe what you've seen so far, are there other areas where you're excited to see this be applied?

    2. GPGuy Parsons29:18

       Sure. It's interesting, though, isn't it? Because I think especially given the tenor of the conversation, um, around these tools and the ethical and legal aspects therein, I suspect that when [laughs] uh, to an extent when you see these things used, especially in prominent context, they might not be advertised as such. Much as, like, green screen, right? Like, when green screen is used in films, you shouldn't be like, "That is an amazing use of green screen."

    3. SSSteph Smith29:47

       [laughs]

    4. GPGuy Parsons29:47

       You should just be like, "Oh my God," like, "he's dangling off a, a thing."

    5. SSSteph Smith29:51

       Uh-huh.

    6. GPGuy Parsons29:51

       "Oh, this must, this must have cost millions." So I think, you know, when we see AI- Tools used in lots of contexts. Not that it's just covered up, but, you know, they might also be just a narrow part of the creative process. I raised this point online, I think, that you were making, which is like, well, where's this all going? Like, will it ever make images good enough? And like, do- will other people, like, want to look at them? Because it's not like we have this huge history of, like, logging in to social media and looking at just, like, abstract pictures. Like, oh, a horse on a [laughs] um, on a surfboard. I mean, it's-- and things tend to have like a grounding in reality, right? Like, that's what makes them viral or interesting. It's like a real thing that happened. But then someone was like, "No," like maybe this-- it won't be that it's gonna make content so good that it's, like, better than Netflix or, like, better than Instagram or better than... It's like, it's the hobby of doing it. Like, that's the, that's the entertainment.

    7. SSSteph Smith30:41

       Well, I mean, there are skills out there, to your point, where writing, as an example, some people just like to write to write, and whether other people read it doesn't matter, they actually enjoy the process. And so I definitely could see an entertainment angle. But I do think that to an extent, you know, even with writing as an example, a lot of people really hate writing, and a lot of people find value in the money that they get paid to write, um, or the writing is used within a script, which then is published on Netflix. Um, and so it's like, how is this stuff used in the wider world, whether it's, you know, on an e-commerce website, whether it's actually, you know, one day integrating with three D printing and, like, the stuff that you generate in Midjourney then can actually be printed into, like, a real life product that you sell. This isn't just a gimmick. This isn't just a toy.

    8. GPGuy Parsons31:27

       There's this very high-level debate around kind of artistry, I suppose, and as if everything is either gonna be, like, in the Louvre or [laughs] saying that right. In the Tate, I'm from London, or, you know, or, or in the bin. But ultimately, if you look around just this-- any space that you're in and look at everything that has, like, a visual component or, like, a design component-

    9. SSSteph Smith31:53

       [laughs]

    10. GPGuy Parsons31:54

        ... there's so many different levels at which we engage with art. You know, like the pattern on a cushion, the, [laughs] the warning label on the coffee maker, the sausage dog on a,

11. 32:08 – 37:28

    A top 1% prompt engineer

    1. GPGuy Parsons32:08

       on a, on a card. They're all different things, and s- there's some things where human-- the human touch is, like, literally the point. For other things, it's like a soothing pattern to look at so, so that your wall isn't just gray. And so there's all kinds of layers in between, and I think we'll see them used more and more.

    2. SSSteph Smith32:25

       So the final thing I wanna ask you about is how this all fits into, you know, the wider skill set that people might have. So on one hand, I can see how there might be an argument that this idea of the prompt engineer is going to be one that only few can do really well, right? So people are really gonna master this skill set, and they're gonna be much more valuable than the people who don't know how to prompt well. But then I can also see an argument where, as you said, maybe this gets abstracted, and we have great UIs, where truly it becomes the type of thing where basically anyone can do it, and anyone can do it pretty reasonably well. And it just becomes, you know, similar to being able to write and read. These are just kind of fundamental, elemental skills that are in everyone's skill sets. They're taught in schools. Where do you sit with that in terms of how you see this progressing? Like, is it worthwhile, you could also position the question as, to become an excellent top one percent prompt engineer, or is it like, oh, everyone should kind of have this in their toolbox?

    3. GPGuy Parsons33:23

       Well, that depends. I think on the one hand, there's obviously every incentive for the people that make these foundational tools to make prompt engineering, for instance, not a thing because they want everyone to be able to do it, right? Again, if you look at, um, the most recent model of Midjourney, like version four, if you-- like stuff that would not have been even possible six months ago, you can literally do the thing where you type in-- I can't remember your example, but like, I remember 'cause I posted one, someone was arguing about it, and I was like, "Look at this space duck. I just typed in space duck, and it's like this amazing astronaut duck." And he's like, "There's no way you just typed that in." So I went back and checked, and I was like, "No, I, I lied. I actually typed in a really cool space duck." [laughs] But at the same time, like with any material, like artistic or otherwise, if you push things to the boundary, there's always gonna be someone that, like, explores everything that's possible or, like, just iterates and iterates, iterates on something. You know, they're obviously gonna explore further on the map of what's possible than someone that isn't. So I don't think it will become, like, this necessary skill that everyone needs to have. But I do think it will become, you know, like some people are expert wood whittlers or, you know, really good at animating hair or whatever, you know. The people that a-develop a real, like, passion for whether that's just, you know, creatively or out, out of curiosity to s- to see what's possible, you know, will, will be there, and will, like, do some of the most amazing things. And then there's also the kind of the secret prompting, I guess. Like a copywriting thing would be like the obvious example at the moment. You think you're typing something into UX, but really there's something else wrapping that in a prompt [laughs] and then sending it to like a foundational model. So there's probably gonna be some people whose job is to, like, come up with that layer of thing that the consumer or the, like, average person is never seeing, and they think they're just talking to the AI, but really they're talking to this thing that then adds a little bit of zhuzh to it and then tells the AI that.

    4. SSSteph Smith35:22

       This is gonna be a tangent, but it reminds me of-- I just listened to a Reply All episode where they were trying to-- someone had remembered this song from his childhood, and they were-

    5. GPGuy Parsons35:30

       Yes

    6. SSSteph Smith35:30

       ... trying to figure out what it was. You've heard this episode. If people haven't, it's one of the best episodes.

    7. GPGuy Parsons35:33

       That's the only one, but it was so famous.

    8. SSSteph Smith35:35

       [laughs] Yeah, of all time.

    9. GPGuy Parsons35:36

       Isn't it such a good listen?

    10. SSSteph Smith35:38

        Yes, but it reminds me-

    11. GPGuy Parsons35:38

        You're on the edge of your seat the whole time

    12. SSSteph Smith35:39

        ... it reminds me of, do you remember in the episode, they find this lady who is a music producer, but she is a music producer for specifically people who want to create music like The Barenaked Ladies?

    13. GPGuy Parsons35:51

        Yes. [laughs]

    14. SSSteph Smith35:53

        It's like, you know, people have jobs like this. When you grow up, and you're in school, and they tell you, you know, "You could be a doctor one day. You could be a teacher one day," they don't tell you you could be a music producer for musicians that want to sound like The Barenaked Ladies. And it makes me wonder or think about, you know, what- Specific niches are people gonna go into within this realm of prompt engineering, right? Like, maybe you specialize, as you said, in, in hair, maybe in hands, maybe in something for, like, enterprise SaaS companies. I don't know. Um, it's kind of hard to predict at this point since we're so early. But yeah, I think you're right that there's going to be, I guess, kind of a bimodal nature to it. It does seem like the kinda tool that's gonna be on everyone's desktop, but it does also seem like there is this opportunity to become, as someone might say, like, a 10X prompt engineer.

    15. GPGuy Parsons36:39

        I think that's interesting, isn't it? 'Cause that's such a tech world metaphor, like the notion of 10X and, like, like... 'Cause it even implies there's a scale where you can have one, and therefore you can have 10 of it, which like if-- in the, in the record industry, do people talk about being, like, a 10X rec- recording engineer? Like, obviously some recording engineers are, like, famous and, like, better than others, and there's all this kind of talent. But I don't know if people are like, "Yeah," like, "I'm a 10X." But yeah, just like producers and all the kind of people that go into making, I think, music or film, you know, the-- that huge list of people you see at the end of every movie where you discover a whole new world [laughs] of careers that you might have had. I'll unfortunately never be a best boy, but I'm still hoping to be a gaffer. Then, you know, there'll be all those kinds of jobs, I think, in the AI, the creative AI industry.

    16. SSSteph Smith37:26

        You know, your point on the

12. 37:28 – 39:31

    The most popular images

    1. SSSteph Smith37:28

       spectrum of, like, what is 1X and what is 10X, it reminds me of what is the most popular piece of, you could say, art or imagery that is shared online? Like, what comes to mind for you there?

    2. GPGuy Parsons37:41

       [sniffs] I don't know. I feel like I-- Do you-- I feel you said that with a qui- like, as if you're-- you know the answer.

    3. SSSteph Smith37:47

       Well, I have an answer. I wanna see if-- what comes to mind for you there?

    4. GPGuy Parsons37:52

       Photos of parties? [laughs]

    5. SSSteph Smith37:55

       [laughs] So I don't know if this is actually the most, but what comes to mind for me, at least as someone who spends a lot of time on Twitter, is memes. And memes are, like, the most basic kind of imagery ever. It's, like, literally an image with, like, some, some capitalized text tossed on it. And so your point just reminded me of this idea where art especially is subjective, and what people like and resonate with is not necessarily the most refined or extravagant, precise type of imagery, which you can generate in some of these text-to-image tools, but it doesn't necessarily mean that people are gonna resonate with it.

    6. GPGuy Parsons38:33

       Exactly. Could you be a 10-- I mean, u-until they invent an AI that can do 10X memes, which is the last thing we need. [laughs]

    7. SSSteph Smith38:41

       [laughs] All right. Well, this was really fun. Loved hearing about h- where you see this industry, this skill set moving. We will definitely share the prompt book link in the show notes because I think people can benefit from seeing the different types of modifiers that you can include in a prompt, um, and also a link to your social 'cause you're constantly sharing new hacks, new things that you're discovering. But yeah, any, any other places that people should look to find you or your work?

    8. GPGuy Parsons39:07

       Uh, you can find me on Twitter, @guyP, G-U-Y-P. Um, and you can find my [laughs] Substack when I finally post at promptresponse.substack.com.

    9. SSSteph Smith39:19

       Awesome. Well, thanks for doing this. [upbeat music] Thanks for listening to the a16z podcast. If you like this episode, don't forget to subscribe here on YouTube to get our exclusive video content. We'll see you next time.

Episode duration: 39:41