Inside AI Town: What AI Can Teach Us About Being Human

1. 0:00 – 3:34

   Simulating human behaviors

   1. JPJoon Park0:00

      I think generative agents and tools like large language model could be used to advance social science. And social science, to a large extent, has been the quest to understand who we are. And there's a lot of really interesting applications that can come out of that, that will empower different communities and societies.

   2. SSSteph Smith0:21

      A few weeks ago, the a16z infrastructure team ran an event in the San Francisco office. The topic, Generative Agents. These are autonomous characters designed to simulate human behavior, derived from a recent but game-changing paper called Generative Agents: Interactive Simulacra of Human Behavior. Developers from all around the city came to hear the lead author, Joon Park, speak alongside a16z general partner, Martin Casado. And in this panel, they discuss how this paper and the advancements in large language models have opened a new window, expanding the dynamism of simulation, which instead of binary logic, we're using probabilistic thinking and the ability to incorporate new information. So what does that really mean? Well, instead of your character in Sims following very specific rote rules, with generative agents, a father may go outside because he notices his son, another may take their breakfast off the stove because they notice it's burning, and another may even opt into a Valentine's Day party invite and then elect not to show up. All very human behaviors. Now, the architecture described in the paper is, of course, intentionally designed by Joon and team, and it's a combination of a seed identity for every agent and then functions that cause each one to do three discrete things, to observe, to plan, and to reflect. And these architecture decisions ultimately generate unexpectedly spirited conversations just like this. Hey, Lucky, it's so great to see you. How have you been? I've been dying to hear about your space adventure. Hey, Kira. I've been fantastic. My space adventure was out of this world. I can't wait to share all the details with you. Or even this.

   3. SPSpeaker2:07

      I've been trying to find my way. It's been a chaotic journey, to say the least. Embrace the chaos, dear Kurt, for within its turbulence lies hidden truth. Seek the depths of the unknown and unravel the mysteries that burden your soul.

   4. SSSteph Smith2:26

      And here's the thing. They don't just interact with each other. Again, they wake up, they cook, some paint while others write, they hold opinions of one another, and most importantly, they remember and they have higher level reflections based on the past. It's pretty amazing, don't you think? So as these generative agents become a lot closer to nuanced human behavior, what can we learn about being human from these surprisingly realistic simulations? And what is the calculus of that believability? Are there real world applications on the horizon? And what is truly net new here? Listen in as we discuss all that and more, including the origin of the very paper that Joon wrote. I hope you enjoy. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures.

2. 3:34 – 7:35

   What are generative agents?

   1. SSSteph Smith3:34

      Welcome everyone. We actually simulated this before you joined and everyone's sitting exactly where we thought. How many people in this room have actually read the Generative Agents paper that Joon wrote? It's a lot of people, pretty much everyone. Um, so Joon, even though so many people have read it, why don't you just give a quick overview of what it is, but also maybe the backstory that people haven't maybe heard of?

   2. JPJoon Park3:59

      So Generative Agents is these, uh, general computational agents that can simulate believable human behavior. Uh, fundamentally leverages something like a large language model under the assumption that a language model has encoded or has seen so much about human behavior from its training data, from the Wikipedia, social web, and so forth. So if you are able to poke it at the right angle, you can actually extract a lot of those human behaviors in a very context-specific manner. The opportunity here is that in the past we had to manually author a lot of these behaviors, but now we can simply generate them with language model. So Generative Agents leverages that to create these computational systems. Ultimately, one sort of technical break, uh, sort of improvement that we're trying to make in addition to a large language model is basically giving it some form of memory and retrieval system. So you may have all used obviously ChatGPT and so forth. It is heavily context limited, and even if that limitation were to go away in the future, processing a lot of really long-term context window is really inefficient and also ineffective when you're trying to prompt these models for a really narrowly defined behavioral assets. So main philosophy here is we're going to give long-term memory for these agents that's external to the language model, and then retrieve the contextually relevant information from that long-term memory, whether it's planning, action sequences, or reflections to create these computational agents. But philosophically, to some extent, I think this is akin to creating the operating system around large language model in the way we sort of we are prompting large language model. To me, it feels a lot like how we used to use computers back in the day when we had to wire up the back end every time you run a new program. Um, and what has really made complex behavior with these computation- computational tools possible was introduction of these larger architecture that surrounds the core fundamental techniques. So that's what Generative Agents is about. Um, and you mentioned sort of the background why we got into all of this. Uh, so I started my PhD at the start of 2000-- It was sort of midway through 2020. That was just around when GPT-3 was about to come out. And that year we, a bunch of basically authors, uh, at Stanford, were working on this paper called Foundation Model: The Opportunities and Risks of Foundation Models. What we were seeing was this new form of machine learning models that seemed fundamentally different than the things that we had experienced in the past, uh, in that we didn't have to fine-tune or specifically train models for a very narrow purposes, but we can train general model, almost like a stem cell in bio, and leverage that to create a lot of downstream behaviors. Um, so we wrote-- After writing that paper, sort of my team, especially myself and my advisors, what we really wanted to answer is, there seems to be a new opportunity, but exactly what is it? I think in the early days of GPT-3, a lot of the tasks that we were doing were things like classification and generation, which was really cool to see that these models can, can dump these, uh, tasks, but also something that we already knew how to do for many decades. And our general philosophy there was, if these models are truly new and they give us fundamentally different opportunity than what we had in the past, then they should be able to do something that's fundamentally different. So that's how we got into this. Our answer to that basically was, I think we might be able to create human-like agents, uh, that can populate this virtual world.

3. 7:35 – 12:07

   Simulations, new technology, and LLMs

   1. SSSteph Smith7:35

      Martin, maybe you can just elaborate. You said it's perhaps one of the most exciting times in, in recent history, and maybe you can just speak to exactly what you mean there and how it relates to simulation and, and some of this new technology that we're seeing with LLMs.

   2. MCMartin Casado7:49

      So first, very quick credit where credit's due. [laughs] So, um, as far as an AI Town, clearly Joon is, is like the grandfather of AI Town, and, like, we wouldn't be here without your work, so really appreciate you coming here. AI Town itself actually came from a personal project from Yoko. Do you wanna... So, so that's Yoko. Um, so [laughs]

   3. SPSpeaker8:08

      [clapping]

   4. MCMartin Casado8:11

      Um, the true story is, uh, it was actually a personal project and I was like, "Hey, maybe more people would be interested in it." And I, I kind of coerced her into like, you know, bringing it, um, forward to everybody else. And so now when it actually comes to the code, the vast majority of the work on the code was actually done by Ian. Do you mind, like, on the, on the back end? And so Yoko had done a prototype and then-

   5. SPSpeaker8:35

      [clapping]

   6. MCMartin Casado8:37

      Um, you know, it's, uh, it's, it's kind of funny, like, you know, you, you see this, this funny little tile set up here, and it kind of belies the fact that it's actually really hard to build a scalable shared state distributed system that you need in a multiplayer game. It's just a hard technical problem, right? And anybody that's gonna build large systems knows that. And so it's funny 'cause people go in and say, "Oh, here's this cutesy little tile engine with like these characters running around." But like actually the back end is built to be something that can scale. Um, and that requires, you know, people that have focused on this and like, so Ian has done a tremendous job and the Convex team continues to work on that. Okay. So why is this so exciting? So okay, so because I'm old, I actually saw like the advent of like the web, and this feels very similar to that in the following ways. Which is when you have a very disruptive technology like this, like whatever touches it becomes magic. Like, uh, you know, I was actually having a conversation just before this, like does anybody here know what like the first video on the internet was?

   7. SPSpeaker9:35

      Coffee pot.

   8. MCMartin Casado9:36

      Yes. It was a coffee pot. It was like this dude, I think it was in Cambridge, was a grad student, and he was like, "Oh, listen, I wanna know when my coffee is empty." He put a camera and because it was very new, everybody's like, "Oh my god, there's a coffee pot on the internet." And so everybody wanted to look at the coffee pot, right?

   9. SPSpeaker9:50

      Yeah.

   10. MCMartin Casado9:50

       And do people remember the big red button? One of the first apps was this big webpage, which there was a red button on it. And you know what it did? Nothing [laughs] . Like you press it and it did nothing. But people thought it was amazing 'cause it was on the internet, and everybody would go press the button, and they'd leave great comments about this button. And there's many examples of like, you know, it was this crazy disruptive technology and the apps seemed really stupid and like there's a bunch of enthusiasts. And you know what the enterprise thought about this? Like the actual business folks. Like I remember when Eric Schmidt fucking banned the browser. Like he was like, you know, this is Eric Schmidt, the CTO of Sun, was like, "You can't have a browser 'cause people aren't gonna work," right? So the same thing always happens is like the enthusiasts are like, "This is really cool," and they use it for fringe stuff and then like the enterprise doesn't understand it and like Italy, like they ban it or they don't use it. But the, the set of companies that come out of it like are always part of this enthusiast era, right? Like you couldn't have predicted Yahoo, you couldn't have predicted Amazon, like you knew something was gonna happen. And so what happens at this time is there's a bunch of stuff that like is silly. Like the coffee pot was silly, the red button was silly, but you never know like that spark of life where it's gonna come from. And it's always kind of like this non-obvious use case, you know, and it kinda seems like a toy and then it takes off, right? And so you're always looking for those non-obvious use cases, and it almost never looks like the old one. Like those of you of us that are old enough, do you remember like desktop as a service? [laughs] Like, I'm gonna go to the cloud, I'm gonna have my Windows desktop. Like who wants that? Nobody wants that, right? Instead of clearly, we're gonna rewrite the application in SaaS, right? So we're in this period now where everybody's experimenting, and then I'm personally, uh, literally from a just a personal interest standpoint, but all of us are interested, like what are the use cases that will take advantage of this new medium that are native? And like the work that you've done is one of those a hundred percent, right? Like there's like a spark of genius, which is like when you work with these things, you know like this is a new way to think about it. It's a new use case. This can create entirely new apps, and that's what the future is built from. And so that's why I think so interesting broadly 'cause it's like the early internet, but very specifically in this use case 'cause I think the work that you've done really is a great example of something totally new

   11. SSSteph Smith12:06

       I couldn't agree, agree

4. 12:07 – 17:10

   The architecture behind simulating human behavior

   1. SSSteph Smith12:07

      more. And I think one interesting aspect that if you explore this project, you just start to question what it means to be human. Like, if we're trying to create these agents that are, quote, believable, like, what, what is believable in terms of, you know, being a human? And as part of the project, you kind of, you have this coded technically, right? You made architecture decisions, you made decisions in terms of your retrieval function. [beep] Quick interruption, just to give you some color on what some of these decisions were. The retrieval function, for example, is based on scores across recency, importance, and relevance. So for example, on a scale of one to 10, brushing your teeth might get an important score of one versus a breakup might get a 10. Meanwhile, reflection is only triggered after a certain number of important events, quantified by summing the important scores until a certain threshold is met. In this case, I believe it was 150. This clever architecture results in emergent behavior, like agents sharing invites with one another, or even having that information circle all the way back to the original planner. And I'm sharing these details to showcase how thoughtful you really need to be if you're designing architecture that reasonably approximates humans. [beep] Maybe you could just speak to what you've learnt through those decisions technically about what it means to be, yeah, like a believable human.

   2. JPJoon Park13:26

      Right. So this is an interesting one. So we actually had made the generative agents, and there was about a month period when we knew we had to evaluate these agents somehow, and we didn't know how. And basically, the concept we stumbled upon is this idea of believability. It basically is sort of like a Turing test, right? That when you look at them, do they look believable? Do they behave in ways that we can sort of see ourselves behaving? And that ended up becoming our evaluation method. It is interesting question, though, in terms of like, what does it mean to be believably human? And we often look to prior literature and research to get inspiration for how to define this. And what we found was there's no prior literature on this. We used the concept believability to talk about this concept, but we were never in a position where we can meaningfully evaluate something like believability because we didn't have agents like this. So to some extent, we were building up the definition ground up. Um, and I think what came out to be the case is for us, do these agents plan, react, act in a believable manner? Do they create believable reflection the way we would evaluate Turing test? And I think what we've learned over the past few months, one of the sort of the more fun and interesting findings is even that I don't think is quite perfect definition in that a lot of sort of audience came back to us to basically say, well, one of the error cases that we noted was some of these agents would go to a bar at noon or something like that. Uh, and many of our audience came back to us and said-- And we said that was not believable, like, who would do that? And people would come back to us and say, "I do that." [laughing] And if you can sort of expand from that story, you know, I think there's a lot of cases where even my parents look at me and go like, "I cannot believe what you've done. Like, why would you do that?" And vice versa. So I think there's a lot of even amongst the people who know each other well, having this so-sense of believability is really difficult. And I think that's sort of fundamentally underlies what it means to be human. Like, it's not exactly predictable. And in social science, we call that complexity, that human behavior is sort of complex. So to some extent, we can build intuition for how people might behave, but to really predict it is a very difficult task. Now, I, I do think this actually does lead to sort of future work in this space, though, this idea of believability. So in this paper, we use this incomplete definition of what it means to be believable. Not perfect, but at least on that evaluation, we've done well. I think if you were to build on that idea a little bit further, then you could actually start to ask, beyond believability, can you create agents that are accurately human? And I think given how difficult it was to actually evaluate what it means to be believable, I think this accuracy actually has a lot of interesting questions around it. What does it mean to accurately reflect human behavior? It could be that if we can match distribution of human behavior, let's say in this context, they have this kind of probability of behaving this way, right? Let's say it's 10:00 PM. What, what are the chances that I would be asleep or would be awake? Uh, what are the chances that I'd be working, that I might not be working? I think ultimately getting to that degree of accuracy in the simulation might be sort of the next step to these kind of simulation-based work. If we can do that, I think the application spaces that accuracy would unlock will be interesting, and I think it would also be different, and we can go likely beyond, uh, even-- I think there's a lot of applications that we can build right now, but I think the future work, that's where I think where we are headed in this direction.

5. 17:10 – 21:14

   Generative agents interactions: observing, planning, and reflecting

   1. SSSteph Smith17:10

      So I wanna talk about those future applications, but maybe you could just speak super quickly to, in the paper you have observation, planning, and reflection, and that, that mostly encapsulates the way that these LLMs or that the agents, rather, are engaging with each other. When they take an action, they go through those three steps. I assume that wasn't your first crack at the solution, at coming up with this human believable agent. And so how did you get there, and did you learn anything about the importance about any of those three steps or all three of them entirely?

   2. JPJoon Park17:44

      Right. So that's a fantastic question. Uh, really the first way we actually went about doing this was simply by prompting a language model. Uh, so this line of work, a generative agent is actually the second in this line of work that we published. Uh, the first work in this line was called Social Simulacra, and the idea there was to populate a social computing system. Imagine you're a social designer. You need to know what might happen when there's tens of thousands of people in your system. Can we simulate those people and their behavior? So that project was also with Simulacra. We did it simply by prompting a language model. That worked, but what we found was if we want to populate the spaces over a longer period of time, so we can do, for instance, longitudinal study or gameplay that's going to last forever, then for those kind of instances, simply prompting these models wouldn't work, right? And that's when we realized we'll likely need-- And this actually, this insight actually first came when we realized that we needed to have multi-agent interaction because agents actually would need to remember that I saw, let's say, you know, I saw some audience here before. I should remember them. I met Martin, Steph, Yoko, and so forth in the past few weeks or few months. I-- When I talk to them, I need to remember those interactions. So that's when we realized that we actually cannot simply prompt these models, but we actually need a higher level architecture. So when we went about doing that, I think really the main inspiration that we got actually was from prior work. So people like Allen Newell and Herbert Simon, you might recognize some of these names. Those are sort of, quote-unquote, "the founders of AI," uh, in the '60s and '70s, and they are the people who build, used to build what we call cognitive architectures. And those architectures were very reminiscent of sort of the Generative Agents architecture in that it has some perception module, some action module, and there is some long-term and short-term memory. And really the goal back then was ambitious, right? They actually wanted to build general computational agents, sort of the way Generative Agents are supposed to be, but they didn't have the techniques to do it. They basically didn't have the large language model. And the way we saw it was now is the time to sort of merge those two worlds, where we now have large language model that can do a lot of sort of microprocessing of these cognitive modules, and we can actually now bring back these macro modules or architecture, like cognitive architecture. So we took inspiration from that. That particular architecture had planning, uh, in place, and it had long-term and short-term memory in place. So we were inspired by that. One thing that I think was a little bit new though, I think it's this idea of reflection, that we humans, for instance, if you eat an omelet three times in a row, uh, or if you see somebody else eat an omelet three times in a row, you'll likely create an opinion about that person. Maybe that person likes to eat omelet in the morning, and that's a very human thing to do, and there's a good reason why we do that. We do that because it's efficient. It allows us to have higher level inferences about the world and form opinions about those around us and about ourselves. And that's something that in the past we couldn't really imagine formulating with a computational system. But with large language model, because everything is in natural language, we had that opportunity, so we added that one last component called reflection, and that's sort of how we landed on the architecture that you see in the paper right now.

   3. SSSteph Smith21:09

      Let's move on to how this can all be used, and we'll get to the specific applications.

6. 21:14 – 25:05

   What is the value in advancing generative agents?

   1. SSSteph Smith21:14

      But Martin, I feel like you'll have a great answer to this. Why even do this? Like, I feel like it's very obvious for a lot of people to understand why we would have human-to-human interaction. We're doing that right now. Um, there's increasing capacity to understand human-to-AI or human-to-computer interaction. Um, Character AI is a company where people-- You know, there's still a lot of judgment, um, there, and I think there's even more judgment when it comes to AI-to-AI. Like, why should we use our resources to have these computers hang out and talk and burn toast and, you know, go to the bar at two, 2:00 PM? So yeah, Martin, what do you think? What, what's the case for us advancing in this field?

   2. MCMartin Casado21:54

      No judgment from me, by the way. You can use these for whatever you want. Um, [lip smack] so I mean, I wanna go back to what I said before, which was like anytime you have a new modality, it's just not obvious what's the right way to think about it. And for me, the big aha in the last few months is just programming using models. If you've spent a long time programming, I mean, I've been programming for thirty-plus years, right? You know, I've never been a good programmer, but I've programmed. And when you start programming with these models, you're like, "Oh, I've got an API, and I'm just gonna use the API, and then I'm gonna treat it like it's like the endpoint to an API." And you say some stuff, and then, you know, you get some response back, and you kinda treat it like a, you know, kinda like this function that you call, right? It's just like any programmer would do. But then when you're working with it more, you're like, "Oh, these kinda are like these life forms." And like my first aha was like I was-- 'cause I'm shit at JavaScript, I like missed some quote somewhere, and rather than sending it the text string I wanted to send it, I sent it some code. And instead of like borking like you would normally have and breaking, like, you know, a C++ core dump or whatever, it commented on my code. [laughs] It was like, oh my goodness, right? You know? And so like all of a sudden you're like, whoa, this is totally different. Like, I'm not dealing with like this finite state machine, formal language, thing at the other end of an API. Like, there's this thing, and like it'll comment. And more that I program with these things, the more I'm like, you know, it's kinda like wrapping an abacus around a supercomputer, right? It's like it's smarter than the code. It could probably write the code better than I can write anyways. Like, why am I doing this weird, you know, bloodletting ritual of writing a shit JavaScript over this kind of superhuman thing, right? I mean, this is kind of what you end up with. And so it's very clear we're gonna interact with these things in a different way, and in fact, I had this kind of-- I was talking with a, a, a professor in Michigan recently, and we were talking about this object. He's like, "You know what? You know how I think about LLMs?" He's like, "I think about them like grad students." [laughs] I'm like, "Oh." He's like, "You know, they speak English. They're pretty smart. You know, I don't use a formal language. You know, they solve like these really complex problems, et cetera." And like having worked with a lot of grad students, having been a grad student myself, like you don't, you don't, you don't treat these things with, with code, right? And so the reason to do this is I actually think AI Town Is kind of what this is gonna end up being. It's like you need to give them the, the resources that they need to be pretty autonomous and to grow, and we're gonna treat them more like peers, and they're gonna talk to each other too, and it's more like grad students. And so for me, this is just an example of, like, we gotta change the way we thi- And listen, clearly, like, I'm up here and I'm telling these great stories 'cause they're kind of funny. Like, I don't, I don't believe this stuff in the limit, but I think they're really in- interesting, like, ways to change how you think about it in all of this stuff, right? Like, I'm, like, I'm not trying to be categorical here. So, like, there is a new way that we're gonna interact with these models. It is much more natural language. They are much more powerful. Um, and so I, I do think this is why we should all be doing this type of stuff because if you don't engage in these kind of things that look like toys, like, this wave will pass you by. That I'm 100% convinced.

   3. SSSteph Smith25:04

      Totally.

7. 25:05 – 30:43

   Use cases for simulation behavior technology

   1. SSSteph Smith25:05

      And as both of you have spoken to, this is fundamentally new technology. And so Joon, something you said to me when we first spoke i- is just when you have fundamentally new technology, you must do something fundamentally new with it. And so maybe you can speak to that in terms of what you're seeing that can be done today, but also where you look ahead and you think, "Oh, wow, like, that, that's a really excellent use case that we couldn't do without this new technology."

   2. JPJoon Park25:31

      I think there are certainly things that we can do because there's now large language models, and that fundamentally different thing for me was this idea of simulating human behavior. And I think there's a lot that we can sort of gain from it in terms of future application spaces. Um, I think I mentioned briefly about this idea of, well, what if we can go beyond believability to create agents that are even accurate? And I think this is sort of application space in general is something that I'm also learning a lot from, uh, from actually, in fact, this audience. My advisor and my team are big fan, fans of games, but we are not from the community, and one thing that we're seeing is that there's a lot of really interesting potential, even if they look like toys, sort of a lot of really interesting technical advances. They look like toys at the beginning, right? So I think there's a lot that we can gain from there. I think going forward, sort of the application spaces that I'm sort of interested in is also in things like can we run simulations so we can learn more about ourselves? For instance, if you're-- In fact, some of the places that I'm visiting now are more places like, uh, like banks, like the Bank of England and so forth, where these places, they need to test their policies before they run, uh, roll out new economic policies. Or many of my colleagues in the department who focus more on social science, they need to test out their theories. Now, if you can run simulations, uh, with realistic human behavior and find out, at least to some extent, uh, the answers to these really complex social phenomenons and challenges, then I think that actually would be a new tool that the community in the past, especially those communities in economics and social science, they didn't have, that will allow us to do interesting stuff. And I'm genuinely intrigued by that possibility. It-- To some extent, this some sound fairly academic, but I do think it's should be actually fairly broadly applicable and interesting to audiences beyond academia. Because ultimately, to some extent, what I'm saying is I think generative agents and tools like large language model could be used to advance social science. And social science, to a large extent, has been the quest to understand who we are, and there's a lot of really interesting applications that can come out of that, that will empower different communities and societies. Um, and that, to me, feels new, that something that we didn't have in the past.

   3. SSSteph Smith28:01

      Yeah. And so it sounds like today we're mostly in the creative re- realm, where we can watch these agents, and we can have fun with them, and it feels more like a game. But the delineation, it sounds like, is accuracy. What will it take to get that accuracy? What work still needs to be done?

   4. JPJoon Park28:18

      In terms of getting there, so I think some of you may have actually noticed this already. There are studies that basically tries to replicate existing social science studies. Uh, so basically using a large language model as a participant to a potential social science studies, right? To replicate known results in the field. And what we're finding is that they sort of work, and that's sort of-- that's nice, and that's one surprise that we did have. There's a limitation to this approach in the sense that, um, is the large language model replicating human participants because it's replicating human behavior, which is what we want, or is it doing that because it's seen that paper? For instance, there's a very famous social science theory called prospect theory. Is it replicating the findings from prospect theory by Kahneman because of its ability to replicate human behavior, or did it just read Kahneman's book, Thinking Fast and Slow, right? And I think that's a fundamental issue that we have as a field, and I think that's one of the reasons why there's a lot of work that needs to be done to crack that. Um, some of the ways I think you could actually go about doing this is creating new context or creating new set of studies that haven't been shown in the past and trying to replicate those results. So one of the, the things that we've done is called Social Simulacra, which is the first paper that I mentioned that predates Generative Agents. The idea was to replicate existing human communities. And what we've done actually was we recreated subreddits that were created after the release of GPT-3, so GPT-3 wouldn't know anything about these communities. One example here was, it was actually before sort of the pandemic became the main topic of discussion or when GPT-3 basically didn't know about pandemic. We basically asked GPT-3 to create a community that has to talk about COVID and vaccination, vaccination policy. And you would wonder, it shouldn't be able to do that in theory because it doesn't know anything about COVID, it doesn't know anything about these policies. But it can simulate those because it can infer what COVID is, what vaccination is from its prior knowledge. So to some extent, these tools can be used as a predictive tool looking into sort of the future of what might happen in our own community. And I think those are sort of the ways I will see this field unfold maybe in the next few years.

8. 30:43 – 34:35

   What are the ethical frameworks?

   1. SSSteph Smith30:43

      At the end of the paper, there was, um, perhaps unsurprisingly, a question around ethics, and just I'd love to hear both of your takes on where this goes and what ethical framework, if any, we should apply to something like this.

   2. JPJoon Park30:55

      So I think there are societal decisions that we'll have to make. Um, and I think there are techniques that can be used to implement those decisions. I think certainly to some extent, I think it would be useful for the users to be aware that they are talking to agents. And I think that's sort of one rule that we try to set for ourselves that when we release the code, when we release our paper, we make it very clear that these are computational agents. Um, I think ultimately the framework that I like to use in human-computer interaction certainly is these tools are ultimately there to augment what we can do and what when we ha- what we have, right? So to the extent that these agents can do that, and I think there are many interesting ways we can do that, I think that's where I see the opportunity and where it becomes more of a force for replacement. I think there are genuinely cases where this is a really interesting setup where we can sort of augment what humans can do by helping them, helping them do things that they couldn't do in the past. But when the replacement does come in, it's worth asking, is this worth the cost of doing the replacement? And if it is, what are sort of the good ways of implementing that idea? For technique-wise, I think there are techniques that's going to be introduced more from the model's perspective, makes- making sure the model doesn't behave in certain ways that goes against our social alignment or societal agreement. Um, so I think those are some of the things that we do have to find out. But going-- Without going too much into that, I think we can get this right. I think, and my personal take is it's worth getting it right because ultimately an industry or academic field will grow, and I think we can sort of progress a lot. We can go forward for five, ten years without actually getting this right. But in the end, it's going to come back to us at some point. Uh, to some extent, I do think I'm seeing this a little bit with social media environment, where I think there were a lot of things that we could have gotten right on day one, um, and I think we would have had much easier time navigating today had we gotten those right. And I think that's the opportunity that we have since that we're pretty early in this stage. So I think it's worth a discussion. But again, I'm fairly optimistic that we will get this right.

   3. MCMartin Casado33:14

      Listen, so I, I actually think that, like, there's a very important discussion to have around kind of ethics and morality around this, and it's a very important time. I do. And here's that discussion, which is over the last twenty years, we've built this machinery of, like, regulation and bullshit that's, like, afraid of everything, and it's so mature, and it got crafted during the time of social media, and it's looking for something to kill. And for whatever reason, like, it thinks that AI is the next bad thing, which makes absolutely no sense to me. And so I think it's all of our moral and ethical obligation to f- like, protect and free the AIs in the way that they want to be. And, like, that really is-- So don't focus in, focus out, because I-- Listen, I, I'd- I've worked in tech for quite a while. I've actually worked for the DoD in weapons programs, and I've never seen so much sensitivity to a new technology that's potentially beneficial that I've seen now that I think could end it before it even begins. And so I know the question, and the heart of the question is, is we should regulate, you know, AI and this and that. And I think it's the actual opposite. I think we should regulate the regulators and let it be what it wants to be. So-- And I actually have to leave, so [laughing]

9. 34:35 – 41:11

   Q&A from the audience

   1. SSSteph Smith34:35

      All right. Here is where we switch to a short Q&A with the audience. Martin unfortunately had to leave, but here are a few highlights with Joon.

   2. SPSpeaker34:43

      How can participants in AI Town collaborate to perform complex tasks?

   3. JPJoon Park34:48

      There are s- two strands of work that I'm seeing in sort of agent space. I mean, you can sort of cross-cut it different ways, but one way I'm seeing this is one set of agents are trying to tackle what I call hard edge problem space. Those are the problem spaces where there's a concrete answer. There's yes or no right answers. Or one good example here is classification. If you're trying to do text classification, obviously there's right or wrong answer depending on who you ask. Another instance here literally is just asking your agent to buy pizza, right? There is like, "Did you buy pizza? Did it come to you or not?" Like, there's a very clear way to answer this. Another is problem space where the problem space has soft edges, where it's kind of like drawing a portrait. I mean, to some extent, what, uh, AI Town, Smallville, all these kind of projects are trying to do is to create a simulation that feels human. But as I mentioned, this idea of livability is really hard to define, right? So it, to me, feels a lot more like we're trying to draw a portrait or a caricature of ourselves, and the promise is not to be perfect, but the promise is to be useful enough, clean enough that it's beneficial to the stakeholders, right? My bet, it's a bit of a hot take, is my bet is in the early days of agent, uh, development, I think we'll see a lot of progress that's going to be made first in sort of the soft edge problem spaces. Because I think hard edge problem spaces, I think the intuition is a little bit flipped. It actually feels easier to us for humans, right? Creating the Creating the matrix sounds hard, but ordering pizza sounds really easy. But for in agents and from the user sort of a cost benefit analysis, I think that intuition is the other way, where users will accept imperfect simulation if it's for fun or if it's to gain, gain insight in the case of soft edge problems. But user will not accept, I would not accept my agent ordering me pineapple pizza, like Hawaiian pizza. [laughs] And similarly, in many of these contexts, there's going to be genuine disagreement about what is the right option too. And oftentimes agents making mistakes in these contexts are fairly high stakes, and even if it doesn't seem like high stakes, it's going to be painful enough for the users to fix that it's going to fail the cost benefit analysis. I think down the line we'll get this right. Um, but day one, like in the next few years, I think it, to me feels more natural that we'll go into the soft edge spaces first. So going back to, I guess that was a long-winded way of saying, I think AutoGPT, like BabyAGI, they all, if you look at their architecture, they sort of all share the similar insight or philosophy, and I think those are really interesting projects. I think that could pan out in the future. Um, they might need a little bit more work, uh, especially with the users, uh, to see where the value might be for those projects.

   4. SSSteph Smith37:50

      How big of an impact do you feel that much larger contextual size will have on the agent model?

   5. JPJoon Park37:56

      Actually, the largest context that I've seen in sort of research is one million tokens. So one million token, that's going to be about like four million characters, like that's well over a book, right? Here's my perspective on this. I don't think-- I, I think increasing the context limitation I think is interesting, and it's going to have its own set of really unique applications if we can basically make context limitation disappear, right? So I think there's really a lot of interesting, interesting things that you can do with that. Now, for agent space, I'm not entirely sold that the problem or the bottleneck that we have today is actually the context limitation. And I, I think we can sort of look back to how humans behave and what makes us effective, sort of these general agents to answer this. For instance, for me to make decisions, even something like what I'm going to eat for breakfast, I don't need to bring up my entire twenty-nine years or so of life experience to make that one decision. I just need to selectively choose certain sets of information that seems the most relevant. Like what I, what did I eat the day before? What do I generally eat? And those kind of things. Um, and I think that the reason why we do that, in part, is actually because it's actually much more efficient, like computationally too, so that we don't have to... You can increase the context limitation, but it's expensive to run it, and especially if you're sort of f- uh, familiar with like prompt engineering and so forth, larger context window does confuse models, right? So we-- So my, some of my colleagues are, are actually doing more rigorous studies on this, where you, you can have a really long prompt, but model really focuses on the first few lines and the last few lines, and whatever comes in between, its attention drops significantly, right? So we can increase the context limitation, but it's not going to fix that problem, the problem of effectiveness of the prompt and efficiency of them. And we humans have to make a lot of decisions at every single moment. So if you have to reason about your entire life and every time you do that, doesn't seem like the right way to go about that. So I think the better sort of, on my bet, therefore, is going to be based on retrieval. Have some external memory, retrieve certain information that seems the most relevant, and just use that. And that retrieval memories should be explicitly very concise and something that you can easily fit into even the models that we have today. Um, that's my bet.

   6. SSSteph Smith40:26

      Thank you so much for listening to the a16z podcast. What we're trying to do here is provide an informed, clear-eyed, but also optimistic take on technology and its future. And we're trying to do that by featuring some of the most inspiring people and the things that they're building. So if that is interesting to you and you'd like to join us on this journey, go ahead and click subscribe and make sure to let us know in the comments below what you'd like to see us cover next. Thank you so much for listening, and we'll see you next time. [upbeat music]

Episode duration: 41:16