Ethan Mollick: Why OpenAl Abandons Products, The Biggest Opportunities They Have Not Taken | E1184

1. 0:00 – 2:31

   Intro

   1. EMEthan Mollick0:00

      OpenAI abandons products like crazy. They wanna build a machine god. If you have any talented people, you're going to have them building the next technology for AGI, but if you have a computer, that's what you throw it at. I mean, they're incidentally making $3 billion run rate this year, I think, by like, just accident. There isn't really a product there right now. It's a chatbot and the API. But I think a lot of people in this space are just assuming scale solves issues. The real problem right now is every startup in the world is betting against AGI, which I find really funny because all the funders are like, "Yeah, AGI's coming in the next five years." If it is, why are you funding these startup companies? None of them will survive in an AGI world.

   2. HSHarry Stebbings0:34

      Ready to go? Ethan, I am so excited for this, dude. I told you just now, I am like your biggest fan from afar. So first, thank you so much for joining me today.

   3. EMEthan Mollick0:56

      I'm thrilled. It's, uh, it's great, and I've, I've been an entrepreneurship professor for a very long time before anyone knew about my AI work, so it's always great to be connecting to the VC and entrepreneurship world.

   4. HSHarry Stebbings1:06

      Now, for anyone that doesn't know your work, can you just give a 60-second intro on your work and how you've become much more well-known in the last few years?

   5. EMEthan Mollick1:14

      I'm a former entrepreneur myself, so the startup company I, I helped co-found, um, invented the paywall, so I still feel like I'm, I'm trying to make up for that, uh, in the late '90s. Um, so just p- trying to pay back after, after, after that. Um, but then I, I've been a professor of entrepreneurship. I got trained in MIT, and then I've been at Wharton ever since. You know, I do a lot of work on teaching and thinking about, you know, research on how entrepreneurs become successful, but I also have this side gig of thinking about AI and teaching for a long time. So, I worked at the Media Lab, uh, with a guy named Marvin Minsky, who was one of the founders of AI, and I was like, the non-technical person there who was like, trying to translate what the lab was doing for the world. And then I've been building tools for, how do we teach entrepreneurship at scale? 'Cause it turns out it really matters. Little bits of entrepreneur training make a huge difference in peoples' lives, and we've been playing with AI and other tools. So, when AI sort of came out, I was in the weird place of actually practically using these tools for a long time beforehand. It turns out everybody else who was taking this stuff seriously was computer scientists, so I sort of was there at the early days of like, oh, I know business stuff and entrepreneurship stuff and education, and these things are actually quite useful. And I already had a fairly large Twitter following, so I just sort of became the go-to person, and then there is this, there is a, um, Matthew effect of like, all the labs started talking to me and I get insider information on everything, and it becomes this sort of self-reinforcing prophecy,

2. 2:31 – 5:52

   Thoughts on the New Llama 3.1 Model

   1. EMEthan Mollick2:31

      uh, in that space.

   2. HSHarry Stebbings2:32

      By the way, your Twitter game is fantastic. So like, don't change that at all. I love it. Um, I wanna start though, and it's pretty perfect timing. I said we were pretty casual in how we did this. You know, we saw the new LLaMA 3.1 model come out yesterday. I'm just really intrigued to hear your thoughts, Ethan. What did you think? Is it what you expected?

   3. EMEthan Mollick2:50

      There's, uh, like four or five dimensions that the LLaMA model's super interesting in. Um, we can talk about open weights and open source being one model. I'm, I'm not surprised that they caught up with, uh, the leading-edge state of the art models. I think people are probably over, uh, underestimating how much ammunition the, uh, quote, closed source labs have and are gonna release in the near future. Um, but I think it's great. We now have an open source GPT-4 capable model, and it's going to be everywhere, and, you know, it's just interesting because how much of that gap gets closed by that model. So, you know, every national government had worse AI than any... than, you know, than every kid in Mozambique had access to through GPT-4O. So now, there's a openly available fine-tuned model. We're gonna see a lot of weird effects from AI that were delayed happen as a result. Actually using it, it's, it's pretty good. I mean, I, I don't think it stands out compared to a Claude at this point or something else, but it will soon because people will be working on it, and it is a downloadable open, open weights model, which is kind of a big deal.

   4. HSHarry Stebbings3:49

      How much of that chasm do you think will be closed by the cl- closed source providers with their next releases?

   5. EMEthan Mollick3:55

      I think we don't know a lot, and even the people training the models don't know a lot. I mean, part of the weird bit here, right, is the people training the models are all computer scientists, basically. I mean, doing computer science, and they don't have a huge idea of the implications of the systems. When OpenAI released GPT-3.5, they didn't expect to destroy higher educat- or you know, education, and then they would have to rebuild it because everyone's cheating all of a sudden, right? I mean, they were already cheating, but now they're just cheating really well. But we weren't expecting like, a large scale revision of like, how the world works, right? And so I think we don't know. I think the model... Everything I'm hearing from everybody is that the next generation of models is going to be smarter, right? The exponential continuum. Whether or not that translates to real world implications is a different kind of concern.

   6. HSHarry Stebbings4:35

      Do you know what I find challenging, Ethan? Eh, eh, is every week you go on Twitter, and there's this transience of dominance between the different providers. You know, OpenAI do something, it's like, "Wow, that's incredible." And then Claude do something, it's like, "Wow, that's incredible." LLaMA, what... And every week, it seems like this one's the winner and the rest are losing, and there's just such transience and speed. I almost don't know where to go. Is that understandable?

   7. EMEthan Mollick4:59

      It doesn't help that social media likes buzz. For normal people sitting back, like everyone's just gonna keep using ChatGPT because that's what they're using, right? They might gradually switch to Claude. Like, the enthusiast community is very different than when I talk to the outside world about this stuff, and I think that on the grand sweep of things what really matters is when these models top out, um, and how long that takes. And I think worrying about who's in the lead at one moment is probably less of an issue than the big labs are all gonna keep building. There's no tricks in LLaMA that they really told us that were unusual or indicated some sort of secret breakthrough. We still don't know if there's secret sauce in some of the other labs that are very different, like it's very early days in some ways. So I think trying to get... you know, if you're enthusiastic like me about this technology, great. Follow along and keep, keep track of the MML-, uh, you know, MMLA ratings, but otherwise, I do think there's a little bit of like, uh, you know, unnecessary to get into every detail

3. 5:52 – 8:24

   Four Potential Outcomes: A Framework for the Future

   1. EMEthan Mollick5:52

      at this stage.

   2. HSHarry Stebbings5:53

      I totally get you, and you mentioned there about topping out. Before we discuss kind of the potential topping out and what happens when that does, I do just wanna start on actually the four potential outcomes. You highlight this in your book, which I loved, and I just thought it'd be helpful to start there as a framing. What are those four potential outcomes first?

   3. EMEthan Mollick6:12

      The other four outcomes, Twitter, it only talks about like, and you know, uh, the...... press only talks really about one and four. So let me go through one and four first, then I'll give you the boring middle, right? So o- option one is, uh, this is it. Like, the, you know, w- the models don't get much better or, you know, and it's sort of this whole thing sort of fizzles out. I think this is unlikely because I think not only will models get better, but also we haven't even started integrating them into work yet, right? Like the way you work with these things is an insane process of actually using a chatbot and having a conversation with chatbot is how people are using it for work at this stage. So w- we're not even at the stage of integrating, but it's possible. The things, it's kind of, the, you know, in th- in which case we have 10 years or so of integrating the system slowly into human systems. I think everything sort of stabilizes out where it is. We're not gonna see w- we'll see an economic improvement, but we probably don't see kind of a massive large scale shift except indus- uh, some industries change more than others, right? I think, I think, you know, it's very likely that photography changes a lot and that there's other field ... And customer service changes a lot with our current systems.

   4. HSHarry Stebbings7:11

      Mm-hmm.

   5. EMEthan Mollick7:11

      But that's one option, right? Nothing much happens. Then option four is the machine god, right? Um, we sort of achieve this AGI plus superintelligence thing. Machines are smarter than humans. Uh, they p- we have a intelligence explosion and g- god knows what happens next. We've had a good, uh, couple hundred thousand year run as a species. Um, and, uh, you know, we'll figure out what our successes are and there's a lot of obsession with this because I think that's where everybody both boosters and, um, people who feel negative about this, this is where their minds go first, right, is like superintelligence. I think the more common scenarios when we see a technology, right, are either continued exponential growth or else linear growth in ability, right? And I think that's what we're under-preparing for. So if you look at the trends lines of this stuff, everyone's like trying to anticipate that every m- model is either going exponential from here or is, you know, everything's gonna top out. I think much more likely, you know, our study, AI was as good at the 80th percentile, uh, level of consultants. Next year is the 85th percentile, 90th percentile, 81st percentile, 180th percentile, we don't know. So I think a large part of this is that kind of world and a linear growth world where the models get a little better every year, um, I think that's much more adjustable to one where it could just grow exponentially, um, and we sort of get close to that

4. 8:24 – 9:56

   Will AI Achieve Escape Velocity or Plateau Like the iPhone?

   1. EMEthan Mollick8:24

      AGI world.

   2. HSHarry Stebbings8:25

      Maybe rightly or wrongly, I always think about actually iPhone releases and, you know, the first iPhones there were big differences between the early releases of like the three and then the four, and then slowly it just became kind of a little bit better camera and a little bit better battery and maybe, you know, the calculator, slightly bigger buttons (laughs) or whatever it is. And I'm like, what is it that AI has or people believe AI has that believes it will have escaped velocity of development and it will never achieve that plateauing?

   3. EMEthan Mollick8:53

      You're right. Now we're at the classic sort of top end of a technology where it's all about like who has the best ca- like the calculator was a major factor in the release of the new iOS and you're like, this is where we are right now is much better calculator, um, which I, is, I think hilarious. So I think there is this kind of, um, there is a topping out. Now, if you look at a process like Moore's Law, it's actually a ser- like it's bi- it's a sustained exponential curve for, you know, uh, for years. The difference is that it's a bunch of underlying technology that gets swapped out for each other. So the real question is, what is this sort of top line intelligence? What does that max out at, uh, for what an AI can do? Are its limitations, you know, exceedable? Like right now that w- uh, you know, as we talk about in our research, AI is jagged, so it's really good at some stuff, really bad at other things. So, you know, the, and, and as a result, it can't sub in for all of human work because on one hand it'll do a great job on some of the job stuff, some, and will do a bad job on other things, just like any machine does. The question is, can that jaggedness get overcome? We, we don't know the answers to any of these

5. 9:56 – 13:53

   Identifying the Core Bottleneck: Compute, Data, or Algorithms?

   1. EMEthan Mollick9:56

      questions yet.

   2. HSHarry Stebbings9:56

      Kevin Scott always says that, you know, compute will solve all problems and many have always believed that, that performance will be answered by compute and just more brute compute. I have other people on the show, your Alex Wangs at Scales who say that data is the core bottleneck. When we think about compute data or algorithms, what do we think is the core bottleneck to performance now and in the next 12 to 24 months?

   3. EMEthan Mollick10:21

      I'll try and answer that, but I wanna take the contrarian view first that I always wanna indicate first, which is for most people they just don't care, right? Like, let's say that LLMs top out and it turns out we have to switch to, you know, Mamba or some other like, you know, u- other art- like who cares? Nobody cares. They're using these systems. We don't know what tr- like mixture of experts. Like there's a lot of like in the weeds that you get when you're watching this like, uh, like a sports game of like who's winning and what situation that like y- uh, top line capabilities matter and there's a lot of room left there. Like to me, the thing that gets left out is computer science discussions are often the system, the human systems that these things have to interact with,-

   4. HSHarry Stebbings10:55

      Yeah.

   5. EMEthan Mollick10:56

      ... the organizational systems they have to interact with. And that's where we need to see kind of more growth, right? That being said, we don't know what the bottleneck is, right? There's, there's this idea in the history of science called the reverse salient, which is that technology sort of moves forward, but there's always something that's kind of lagging and all the effort goes into fixing the lag. So in the early days of electricity, we had generators where transmission was a problem. So there's a huge amount of work to make transition better. In our current electrical sort of new economy, it's been batteries. So there's huge amounts of work going into batteries 'cause solar panels are good, but batteries aren't good. I kind of feel like we're just gonna hit a whole bunch of reverse salients, right? So like, "Oh, the data pipeline isn't good enough." Great. Is there a way to, you know, is it gonna be real world data, synthetic data? Or maybe this is the end, but once every, all of science concentrates on one thing, we tend to find ways forward. So I think it's gonna be a bunch of debates over what the trailing indicator is, and then everyone forgets about that because s- it gets solved. It's not a bad approach. It also is just kind of how technology works, right? Because the money is all to be made in the reverse salient, right? So like if, if, if you can make a billion dollars as a data company and, but, you know, because that's the area everyone's stuck on, you become a data company, right? This is, I mean, it's kind of capitalism and science at work. The hard problems are the ones where all the money and prestige comes from.

   6. HSHarry Stebbings12:10

      You said where the money is. I loved an analogy that you said before, and it's you said a lot of people use the analogy of picks and shovels in the Gold Rush. You said that's maybe not such a good analogy, and that the steam train was more apt. Why do you believe that it's not a good analogy, and why is the steam train more apt, Ethan?

   7. EMEthan Mollick12:29

      Analogy are really powerful, and we have very bad ones in AI. VC people get taken in by this, right? So like, I hear, "You wanna sell picks and shovels," and first of all, I don't 100% know. Like, everyone defines that slightly differently. They're like, "Oh, no, no, you wanna sell compute. You wanna sell, you know, you wanna sell the tools that help people scale up and pick their..." You know, uh, that's actually not... You know, first of all, it's unclear what the analogy is, but the second, deeper problem of this is that that isn't actually how a new technology spreads across an organization. You don't wanna sell picks and shovels to the people trying to mine gold. What you wanna do is figure out how to get them to use this new technology, which doesn't have a Gold Rush analogy at all, right? Instead, the steam power, and the steam power, the secret was not James Watt's steam engine, which was important, right? Huge breakthrough. Um, two interesting things, by the way. The things didn't really take off until Watt's patents expired, uh, and it could be openly adapted, but the real value of the steam engine came from having skilled artisans in your factory who said, "I've got this thing that can make power go back and forth. How do I create the gearing to connect that to my, you know, my spinning jenny, my ammu- fa- ammunition manufacturing machine, my bottle shaping tool?" And it was the skilled artisans that made all of this work and made the manufacturers capture all the money. So, you wanna be a skilled artisan right now. You wanna figure out how to take the back and forth power of an LLM and convert that into usable work inside your organization.

6. 13:53 – 15:28

   Why Aren't AI Providers Offering User-Friendly Guides?

   1. EMEthan Mollick13:53

   2. HSHarry Stebbings13:53

      I agree. What flaws me is the lack of human descriptions around how to use these tools effectively. It's like, no one's written, uh, using LLMs for dummies, using AI for dummies, which everyone needs. Why are these providers not doing what is so obviously required?

   3. EMEthan Mollick14:13

      I think that if you talk to Silicon Valley people, they are very obsessed with the race for superintelligence, and I totally get it, right? If you could build a machine god, you win. So, that's kind of the secret story behind what's going on here, is there's a real belief that if scale solves everything, the only, the biggest thing you could do to waste your time is do anything that isn't scaling. Your smartest people have to be scaling. All of your compute has to be scaling. And the bigger models will solve all problems, as you were saying. You know, that's a sort of view in Silicon Valley. So, they're gonna come back and figure this out later, because why would you bother? You know, and there's some truth to that, right? I, I spoke to a very large financial institution, spent a huge amount of money building a GPT-3 powered sales assistant tool that as soon as ChatGPT came out was instantly obsolete, right? So like, you know, why, why bother with this? K, I mean, it was a smart idea at the time. They were way ahead of the curve, right? But I think that the real issue is, is that as a result, all use of this stuff has kind of been dropped, right? There is no manual out there for this stuff. There's not even a dis... Like, there's not even a set of points about what the AI is good and what it's bad at, and as a result, like, it's, uh, we call it documentation by rumor. Like, it's a bunch of people on Twitter, there's like 17 people posting about how they're figuring out how LLMs work, and then everybody else is just kinda using it like a chatbot. It is a very weird situation.

7. 15:28 – 18:49

   Should Powerful AI Models Be Open Source or Closed?

   1. EMEthan Mollick15:28

   2. HSHarry Stebbings15:28

      Can I ask you, you said there kind of about Silicon Valley and how they think about where the true value lies. You know, we have Vinod Khosla on the one hand that says we cannot have such powerful models open source. We have Marc Andreessen and others say that they have to be. What do you believe is best?

   3. EMEthan Mollick15:47

      I am generally in favor of technological progress, and I think that openness frees people to do lots of really interesting things. There's some very obvious low-hanging fruit with AI and healthcare and education that I think are, you know, going to be very helpful in large parts of the world that don't have access to good doctors or good tutors, right? Uh, you know, for places that do have access to that, there's a lot more nuanced discussion about when do you turn to AI for some of these things. So, I think open models will make a big difference. They'll spark entrepreneurship. We know that people who get advice from AI do better as founders in Kenya if they were already doing well. Like, there's a lot of really exciting stuff here about openness, but there's also downside risks, and it feels very weird for people to say like, "It's all one thing or another." And I, I do think that open models will immediately have their guardrails breached, and we already know three or four low-hanging threats. I think people are overly worried about science fiction threats, right? It's not good enough to help you build a virus at this point, you know, but it could be in the future. We just don't know. But what I am worried about is, you know, our entire computer security system depends on it being very expensive to spearphish somebody, and this does spearphishing at scale. What do we feel about that? Like, that you could do this? These systems are, will be, you know, uh, are gonna be harnessed for very, you know, for very good catfishing campaigns. How do we feel about that stuff? I just feel like there's not this conversation. So, I think the open models both carry risk and reward. I don't think there's a lot of thought going into this stuff. I think it's all corporate strategy at this point, right? So, Meta doesn't really wanna make money from models, so they're gonna spoil, you know, their rivals, right? Microsoft has a chance to go after Google, so it adds AI into Bing. Like, there's a lot of like, back and forth among a few firms, and I don't think we actually know the full meaning of open source AI, and it's a little weird to both say it's super powerful and can do everything and therefore it's high risk, and also it's not that big a deal.

   4. HSHarry Stebbings17:33

      You said there's a lot, not a lot of thought going into it. What thought would you like to see going into it? Like, what do you think would be a commensurate level of thought and analysis? (laughs)

   5. EMEthan Mollick17:42

      I think that we need to be built for fast reaction to these models. I, what I'm worry... So, there's, uh, Joshua Gans, who's a professor at, uh, University of Toronto, I think has a really nice model for AI regulation that I think is probably right, which is when you have a new technology and you don't know what the problems and issues are gonna be, you do fast follow-up regulation. So, you don't try and pre-regulate 'cause you don't know what it's good or bad at, but you do watch what's happening and have rules that you put into place and policies and fast reaction. Now, we can talk all about how government's not built to do that, how it's not cooperating well with industry, but I think that's the same way we think about open source right now. So, we've just released a very powerful model open source. Who is setting up to learn for what the implications of this are going to be, and do they have a pipeline back to the open source makers of these models? Is there something that would stop Meta? Is there an event that would stop Meta from outsourcing, uh, from open sourcing its models?... a pr- I don't know. Who's watching that stuff? Are we have- is there any kind of monitoring system out there to find out how this is disrupting the world one way or another? There doesn't seem to be. So, to me, a really responsible view would be, sure, let's release open source, but then let's be watching over the next six months to get a sense of what this is good or

8. 18:49 – 22:10

   Will Regulations Limit AI Growth?

   1. EMEthan Mollick18:49

      bad at, and, you know, react to it. And that's what's worrying me a little bit.

   2. HSHarry Stebbings18:53

      I'm sitting in Europe where we have the EU AI Act, which is incredibly stringent. Uh, EU is also particularly s- talented when it comes to regulation. Um, I'm very worried that after we all have such constraining regulation that it will actually cause the plateauing effect of AI, both in development and in adoption. Do you think intense regulatory scrutiny is a cause for concern in the path to much more developed AI systems?

   3. EMEthan Mollick19:19

      Yeah, I mean, I think that not being fast and reactive is a problem, right? You wanna have people develop this new technology, you want it developed in, you know, in, uh, you- you know, you want it developed by the societies that you wanna develop these technologies in, like, you want them to be used in democratic ways. All of that stuff indicates, like, we wanna see continued growth. That doesn't mean... It just feels like it's either/or for so many conversations, like, either there's no regulation and no scrutiny whatsoever and technology always benefits everybody, and I'm a technology optimist, like, it does benefit people, but, like, it's weird to have no downside risk. On the other hand, you have the, we must regulate in advance to stop a bunch of harms that haven't occurred yet, and that the current levels of models clearly will not cause, right? We're not gonna get a runaway superintelligence from a LLaMA, you know, a LLaMA 3.1. So, we have to have some sort of balance here. I think the EU has, you know, I mean, the EU has- has definitely put in a lot of stringent things in place. I don't know whether Europe would be leading in AI anyway. I mean, there's a weird ecosystem problem. Talk about, you know, this is 20 BC. I mean, VC is always been a US thing, um, it has bi- you know, London did okay for a while there, but aside from that, you know, more money went to graduates from Penn, from the school I teach at, in, uh, last time I checked in 2022, than everybody in France and Germany put together. We already have a whole bunch of e- in- innovation ecosystem problems in place, regulation is one of them, but I don't think, I think a lot of people are pointing at EU regulation being like, this is the cause. There's a multi-causal problem here in terms of Europe, um, versus the US on technology development. Everyone moves to K- Silicon Valley 'cause you kinda have to, and all the stats show that's actually a really good idea for almost every venture. Like, there's a machine here that keeps working, right? To go back to the bigger issue, I think a lot of putting a lot of tight regulation on AI at the beginning is definitely an issue, right? Because LLaMA breaches the high security risk level f- in terms of number of flops with the EU.

   4. HSHarry Stebbings21:12

      I have to, you said so many great things there. Would you tell your students today that they have to move to the Valley if they want to increase their chances of winning?

   5. EMEthan Mollick21:19

      That's an empirical result from a bunch of studies. Companies that mo- you know, there's been a study of Israeli companies in the Valley, New York companies. Like, it just, the issue is, is that that's where the connections are, and it turns out Zoom only gets you so far. The, I mean, the average distance, at least pre-pandemic, I would- I'm s- be surprised if it actually changed. Um, the average distance between a VC and a company it invests in is about 40 miles. Like, that's because when you look at wh- where VCs spend their time, it's networking and it's monitoring. It's networking with other, with, you know, and learning about companies, and then it's monitoring the portfolio of companies. And that's much easier when you're local. Zoom doesn't let you do monitoring the same way. In fact, when direct flight is added between SFO and another city, the, uh, VC investment in that city goes up, because it's just ge- it's easier to fly there and help do, uh, and do monitoring there. It's a local business, right? Everyone's like, "Oh, it's global, it's connected." It's a local

9. 22:10 – 26:00

   What Are AI Labs Missing About Business Needs?

   1. EMEthan Mollick22:10

      business.

   2. HSHarry Stebbings22:11

      Can I dive into a couple of different market participants? We've already touched on some of them, but I wanna start on AI labs. We mentioned LLaMA and model progression earlier. In terms of the AI labs, what do the big AI labs not understand about companies themselves, do you think?

   3. EMEthan Mollick22:26

      I think that's such an important question. I mean, there is just, the products being released are just super weird. I think there's very little consideration of use cases. I mean, you look at the number of people inside these organizations who've worked at large companies. I- I often joke, like, you know, when I go to, when I go to the West Coast, there's like, you know, it's all, like, cold plunges and how do you live forever and really, you know, and then in the East Coast, it's like, we dr- you know, we're drinking coffee till we die, and the goal is, like, get our work done, get home, like, you know, like, it's just a different, like, in a large company, it's very different, and there's a lot of, like, contempt, I think, for large companies. That's where most smart people are, right, are in large organizations doing, you know, other work that is not Silicon Valley work. For every coder, there are 16 managers, right? And, you know, I- I think that there's not a sense of what this stuff does for them, and as a result, there's a lot of half-built products that are brilliant and then get walked away from. Code Interpreter is- is a huge world-changing product for data analysts that got partially abandoned by OpenAI that haven't- they haven't moved the needle on that since. Chatbots and APIs remain the main area. Almost all the documentation is technical documentation, and almost all the interesting use cases are not being discovered by technologists, who are actually quite bad at using AI often, because it doesn't work like a normal technology. They're being discovered by end users, managers, and the system's not built for those things. So there's just this huge gap between technology and use.

   4. HSHarry Stebbings23:44

      I'm so sorry to be so naive. Why, if Code Interpreter is such a generational defining product for analysts, why would they walk away from it, or not walk away from it, but, you know, not progress in the same manner as they started?

   5. EMEthan Mollick23:56

      OpenAI abandons products like crazy. I think these products are passion projects from various people. Again, they wanna build the machine god. If you have any talented people, you're going to have them doing, you know, building the next technology for AGI, and if you have staff, that's what you throw it at, if you have compute, that's what you throw it at. I mean, they're incidentally making three billion dollar run rate this year, I think, by, like, just accident. But there isn't like a, there isn't really a product there right now. It's- it's the chatbot and the API, and the system gets smarter to solve more problems. I think a lot of people in this space are just assuming scale solves issues. So, why would I bo- or development solves issues. So, why would I bother spending some time thinking about, you know, how to productize this when the product's gonna be obsolete in a year anyway?

   6. HSHarry Stebbings24:40

      Can I ask you, on the flip side, we have the companies themselves. What are companies getting wrong about AI that they should know more about?

   7. EMEthan Mollick24:47

      I mean, I speak to organizations all the time, and first of all, just from a perspective, almost nobody uses these systems. I mean, they've all tried ChatGPT, right? Every, when I raise my hand, everybody's tried ChatGPT, almost always the 3.5 version or before. About 5 to 10% of people in any room, whether... And that, by the way, Silicon Valley, actual people, right, who aren't at a lab, whether that's at a large bank, whether that's at a conference of innovation professionals, uh, maybe 5 to 10% have used those models, and maybe 2 or 3% have used 10 hours, which has been my, you know, sort of guideline, you know, minimum number. And I think, again, there's no onboarding. You're faced with a chat bot, and when people are faced with the tyranny of the blank page, they panic. What do you talk to the system about, right? And like, there's no information, there's no instructions, and so people aren't really using it. So, the issue is that, partially it's them. They need to adopt, because when people start using it, they find uses, right? So, a, a new study just came out of Denmark of people who are using ChatGPT in, you know, in knowledge-intensive work environments, and, you know, they're estimating that in, uh, you know, over 30% of their tasks, they're saving 50% of their time. So, once people use it, they find productive uses, so then the question becomes, how are you harnessing those uses? What policies do you have? I mean, there's so much we could talk about, about what companies are getting wrong.

10. 26:00 – 28:22

    How Can We Better Harness AI to Drive Productivity?

    1. EMEthan Mollick26:00

    2. HSHarry Stebbings26:00

       In terms of how are we harnessing the uses, what would you like to see changed there? 'Cause that's where we can fundamentally drive productivity, which is arguably the most important thing.

    3. EMEthan Mollick26:09

       I mean, so first of all, it just starts with policies. When you look at companies, they have, um... So, first of all, a lot of them don't even allow access to GPT-4, because the regulatory environment's unclear. So, one thing that'd be great from a regulator perspective is not just, we've talked about the negative side of regulation. There's a reason why banks are regulated, or pharma companies are regulated. It would be useful for clear guidance about how to positively use AI, and I think there's been some movement towards that. That would be something I would want the EU to be doing a lot more of too, is like, "Okay, what are the ethical use cases that we should be pushing and opening regulation for?" So, that, but that extends to the company policy side. Company policies are often very vague. You know, "Don't use this, or use it, but don't get it in a way that doesn't get you fired." Um, and then, there's a whole bunch of, like, uncertainty over how you get rewarded. What happens if you figure out a solution to work? So, what I find is inside organizations, when I finish with the talk, all these people come up to me and reveal that they were secret cyborgs all along, so they've been using this for all of their work. But they're not telling anyone. They're not telling anyone 'cause they're worried they'd get fired. They're worried that people will stop respecting their work, 'cause they realize it's AI-written. Right now, Reddit's full of people saying, "I'm a, uh, people think I'm a wizard at work." Like, they don't wanna be, like, lose that. They're worried that people will realize you don't need as many staff members, so you fire them, or you fire their, their colleagues. They're worried that if they do this, they w- you'll just assign them more work, or you won't reward them for it. So, everyone's hiding AI use. I just spoke to a woman who banned ChatGPT at a major bank. She used ChatGPT on her phone to write the ban. Um, because of like, why do it by hand? So, once people start using it, they're all using it secretly. And, you know, and so, we need a clarity around, how do you get rewarded for this? Like, what happens if I automate my job? And to go back to our industrial revolution analogy, if you were a brewery in the early 1700s, and you were serving your local community, wh- 'cause they, uh, everything was kind of local, and you had steam power, you kind of have a choice. "Do I want to, um, fire a lot of people and make the same amount of beer for less money and have a higher margin? Or, do I want to be Guinness and expand my production around the world and hire another 100,000 people?" And we're used to IT solutions being a cost-saving measure, right? "If I get 30% productivity boost, I fire 30% of people." Your people are never gonna show you how they use AI at that rate, and you're never gonna win in a world if we really believe there's an industrial revolution happening. So, policies are really at the

11. 28:22 – 33:23

    Will AI Redistribute Talent or Eliminate Jobs?

    1. EMEthan Mollick28:22

       heart of the problem.

    2. HSHarry Stebbings28:23

       And I always feel that, in the majority of cases, we just redistribute talent, as you said, then more effectively for new projects, for new initiatives, for expansion. What I find worrying is here, especially in the early days, it is killing the lower classes, if you're being horrible and blunt. Which is like, Klarna had like, 70% improvements with AI in terms of customer service, and cut so many of their workforce. You're seeing especially customer service be the core kind of Trojan horse. Which is replacing, in most cases that I see, 90% now of customer service teams. To what extent do you think I'm over-worrying, and actually we'll see the continuing redistribution of talent, not the removal of talent?

    3. EMEthan Mollick29:04

       So, this is a case where I think we're being a little sanguine about this. I mean, I, I think, you know, again, every technological revolution, um, people lose jobs and then new jobs are created, right? But there's, you know, and we talk about this all the time, there are two big caveats to that. Caveat number one is not always. When the telephone switchboards went from sort of manual to digital in the starting, or not digital at that point, but mechanical, in the 19, starting in the 1930s. At that point, I think one out of every 16 women had spent time as a telephone operator. Um, it was like a job, and then if you got fired from that, if you were young, you found other jobs. If you were older, you'd never find another job as good, right? Because you were really good at telephone, as a telephone operator. So, not every job ends up with a new category replacing it. And the other thing is, living through the industrial revolution kinda sucks, right? Like, you can look backwards and say, you know, like, "Oh, great, everyone got better jobs, they're much richer now," but there were also people, you know, smashing machines because they didn't want their jobs replaced, and there was a lot of unrest. There's a reason why there was, that's when the great debates between capitalism and, and Marxism arose, because there was u- unrest during this period that was serious. So, I think part of what I worry about is a little bit of like, even if you have the sanguine view that everything's gonna be fine, that doesn't just happen automatically. Like, you can't say, "The market makes everything great, so let's not worry about it, or else everyone's gonna lose their job and we need UBI." There has to be something much more specific about, yes, there's going to be waves of disruption heading through the economy. How do we do things we were very bad at, like retraining? And, you know, it's, it's a problem that actually needs to be solved. It's not something that has to be made as science fiction.

    4. HSHarry Stebbings30:36

       Well, what I worry more about is actually the distribution of knowledge and productivity versus the distribution of wealth. And what I mean by that is, there is this like, 1% of like, Silicon Valley and tech elite, I think, who are using AI and the surrounding products incredibly well to do 10X the work that we used to do, and to be way more efficient and way more cost-effective. And then, there's the rest of the world. You know, we joked about Europe, but I live in the UK. You go to places in the UK, they've got no idea what ChatGPT is, let alone how to use it to create mark- marketing campaigns that are 99% cheaper in 10% of the time.I think it's just creating even more knowledge and productivity to 1%, and the world could get left behind. Am I right to fear that, too?

    5. EMEthan Mollick31:21

       I'll say yes and. Okay, so a Denmark study I told, I, I talked about did find that the people using this skewed mostly male and mostly wealthy, right? They were people finding use cases because that tends to be a fairly common tech adoption curve. The thing that is unusual about AI, though, is first, it's ubiquity, right? So normally, getting your new tech installed means I've gotta have, you know, know how to use a computer really well and be really in with, like, you know, "How do I get a, you know, a distro from GitHub," and, like, you know, there's work involved that is, that is a narrow set of work that requires time, effort, money. That isn't the case here, right? The chatbot is accessible from a phone in, you know, 169 countries around the world, have access to the world's best AI systems. So that, that's one thing, right, and chat is a fairly normal interface, especially when you have voice. The second is early evidence is that coders are not particularly good at working with AI, right, because it doesn't do the things you expect it to do. My favorite example is, uh, Simon Williamson, who if you don't follow, is, is terrific, um, and, uh, really, you know, great at this stuff, but he's been building a, uh, he works on data journalism, and he built, he was using Claude for OCR, um, on political campaign donations. And when he checked back, uh, he just found that Claude refused to do the work because there were names and addresses, and even though they were public, he was, Claude was like, "I don't wanna violate anyone's privacy." Like, we're not used to systems that object to the task that they're given, or, like, sometimes argue with you or give you a different answer every time, so coders are often not the best users. Often, the best users are people who are actually really good at working with humans. I mean, my, my wife is probably one of the best prompt engineers on the planet. Uh, she's got a doctorate. We work together. We're co-directors of the AI lab. Um, and, um, she's never coded a day in her life, but regularly does stuff that OpenAI and Anthropic are like, "Wow, that's a really amazing prompt. We didn't know." Like, Google used her prompt as the gold standard to measure their fine-tuned models against, right? But what she has is a, you know, doctorate in education and we've been building educational teaching games for a long time, and she has good theory of mind for other people. If you can write instructions, if you can manage, you can use this. So that's what I'm hopeful for. It looks like a tech adoption curve, but tech people shouldn't have the advantage they had in other spaces, and word just has to

12. 33:23 – 36:09

    AI and Consumers: The Future Interface Experience

    1. EMEthan Mollick33:23

       get out.

    2. HSHarry Stebbings33:23

       I do have to ask, you mentioned that kind of, uh, the quality of prompts and, uh, uh, how amazing your wife is with her quality of prompts. You also mentioned kind of the white screen of death and when you have kind of that blank template, not knowing what to do with it. You've said before about, bluntly, the kind of challenges of the chatbot interface and what a weird interface it is. What do you think will be the interface of the consumer between the power of AI and consumers?

    3. EMEthan Mollick33:46

       I think multimodal is really the answer here. I mean, all the pieces are in play. So some of the most interesting people I, you know, people I talk to who are really using AI are just having conversations with it. Like, I, you know, um, I think about, um, Ali Miller, who's a really great sort of tauc- you know, uh, person who has been thinking about, a lot about AI, ex-Amazon person, and she has conversations with the AI every morning while she's doing her hair, right, just the limited chat interface. Once these things have full visual, which they do, right, they have a lot of latent capabilities that people haven't recognized yet in multimodal, and you can chat with them, then it starts being more like having a human on call. I think once you start adding agency into that, where they can take action in the world, I, I, I wonder if we just sort of skip the step of, you know, how do you use these things to, like, "Oh, yeah, you talk to your phone, and your assistant does the thing that you want it to do." So there is this narrow window, I think, where prompting style really matters, where being really up-to-date on these systems matter, but then they come to your phone and, you know, and also, by the way, if they save you time in work, if they really do do that, humans are exquisitely designed to figure out how to minimize the effort they put into things. There's a reason why adoption rates are over 70% in universities for cha- ChatGPT and, while they're, like, at a few percent elsewhere in the world. We figure stuff out like this, and I think that that's the other piece that's missing.

    4. HSHarry Stebbings35:03

       Why do you think we figure stuff out at university when we wanna cut the time to do a p- assignment, coursework, whatever, but we don't cut the time at work when we still have, technically, assignments and coursework in our jobs?

    5. EMEthan Mollick35:16

       First of all, there's a lot of communication in universities that there isn't elsewhere. When I talk to large companies, no one talks to other people at other organizations very much, right? So you have to hope you hear this from a friend. There's not... But in universities, everyone is, like, talking about their work, and they're all in classes together, and we know that, like, that barrier to spreading information, it's not technical, non-technical. It's just sort of like, "Oh, cool, my friend showed me that you could solve, you know, does all the math problems for me," right? Like, that, you know, that's one thing, right? I think also, for better or for worse, chatbots are just really good at homework, right? Like, so there's, like, marketing requires you to be an expert in marketing. Chatbots, right now, like, if you wanna see the future where it's better than human, it's actually in homework, where in most cases, it kinda solves the issue, writes an essay better than most people. But it's, like, for marketing, you do need to, you know, work with it a little to be in your house style, to have a conversation with it, have it understand your context. So there is a little bit more friction there than, um, you know, writing a, uh, five-paragraph essay about George Washington.

13. 36:09 – 41:35

    AI Ambition in Startups: What's Holding Them Back?

    1. EMEthan Mollick36:10

    2. HSHarry Stebbings36:10

       Can I ask, one element that we haven't discussed is, is startups themselves, actually, and so on that, like, you've said before that you don't think startups are being ambitious enough in the face of AI. What should they be doing, Ethan, and why are they not doing it?

    3. EMEthan Mollick36:24

       I think the problems of, um, uh, of the lean method are coming home to roost. What every VC wants to see is, you know, and, and especially in, like, app-facing stuff, is they wanna see product market fit. There's a method we have, right? You come up with, like, a, you know, rough business model canvas, and then you go out, and you do, talk to people, and then you test in the world. That is not a good model for breakthrough innovation. That's a really good model for incremental innovation, where you find market need. So part of this is that we're incentivizing startups to find solutions right now for a moving technology, and they're just gonna get lapped, and they're not trained to be imaginative. They're not like... It's, they're trained to think money first, and, you know, "How do I get a market, product market fit?" Which is fine in normal technological regimes.... not a great idea in radical regimes.

    4. HSHarry Stebbings37:11

       What is a good mode- model for a radical regime, then? 'Cause I've been brought up, quite rightly, it- as you mentioned there, in the incremental innovation kind of economy, where it's like test, iterate, find product-market fit, someone pays for it, good, well done. So what is the right model in this new age of kind of radical innovation shift?

    5. EMEthan Mollick37:28

       VCs have funded this model, right? And it's like deep tech, medical, like things where you're making larger bets in the future, where there, you know, where there's payoff as- where, where when it's revealed to the world

    6. NANarrator37:39

       Yeah.

    7. EMEthan Mollick37:39

       ... that it's gonna succeed or not, right? And where you're making a bet on technology itself. That's where VC got its start.

    8. HSHarry Stebbings37:45

       Mm-hmm.

    9. EMEthan Mollick37:45

       And it sort of became, uh, you know, perverted a little bit to this, like, "How do I get m- you know, make money fast" machine. I mean, not that fast, right? It's still years till exit. But there's the idea of like, you know, with, uh, you know, it- i- it's all about pro rata rights and the idea of like I make a lot of small bets initially, and then I can double down on the people doing well and not do double down on others, and it's about finding the diamond in the rough. Like, all of that stuff is like a great model for funding incremental innovation. If the market's changing, like, but- and we're used to market changing slowly enough that, like, that's not a problem. I think it's an issue here. I think you need to be imaginative. I think you need to be subject-specific. I need- think you need to assume model- I mean, it is very strange from one hand, for all these people at Silicon Valley to be like, "Yeah, you know, AGI is coming." And then the applications they're building are like these very narrow, like, "Hey, I slapped something on top of LLaMA," and, you know, it's- uh, like that's not gonna do it.

    10. HSHarry Stebbings38:34

        I- I- I totally agree and get you. What should I and what should my fellow venture investors change, then, about the way that we invest, do you think?

    11. EMEthan Mollick38:42

        What you should be thinking about is have a position on the future, and the- and the startups you talk to have to have a position on the future of AI. How good does it get, and how does your model work? The second thing I think people need to be thinking about is how actual adoption happens again, right? It used to be that if you have a large enough market to play with, we just go after all of it, and, you know, some part of it starts to respond, and we double down on that section. You've got to be much more opinionated about how you imagine your technology being spread or adopted. How does it spread throughout an organization? Is it fit- how does it fit with the organizational structure and approach? Um, you know, I- I think that there's just- it's- it's just requires people to have more plan and strategy than they did before, r- rather than just letting the market tell them the answer.

    12. HSHarry Stebbings39:24

        Can I ask does- do they not go in, uh, contradiction? You said there about, hey, you know, people work on small, kind of minute things on top of LLaMA, say, and then it's like, well, you need to be opinionated about who you're going after and who you're not going after, you need to be more targeted. Is that not kind of one and the same, which is like the verticalization of approach and the targeted approach being the core?

    13. EMEthan Mollick39:45

        Well, I think it's not about verticalization as much as opinionated, right? I think you need to have a strong opinion of what the future looks like and where the gaps are gonna remain. This is a jagged technology. Trying to work on the exact tech- like figure out where you think there's going to be jaggedness, and that can be organizational jaggedness, interface jaggedness. But I mean, you're also- basically, the real problem right now is every startup in the world is betting against, um, AGI, which I find really funny 'cause all the funders are like, "Yeah, AGI's coming in the next five years." If it is, why are you funding these startup companies? Like, none of them would survive in an AGI world.

    14. HSHarry Stebbings40:17

        (laughs) Uh, uh, j- for those that don't understand, why will none of them survive in an AGI world, Ethan?

    15. EMEthan Mollick40:23

        So AG- the- the common definition of AGI is a machine that's smarter th- than humans at every task. So, like, the machine will decide what to do. You're not go- like, who cares about your stupid product, right? Like, you've been making this for humans to get a product-market fit, and the- but the humans will say, you know, "Optimize my trading strategy," or the AI will just decide, "Optimize your trading strategy." I mean, no one knows what AGI looks like, so I'm not gonna try and paint a science fictional future. But I will say there's a huge contradiction between a Marc Andreessen saying, "AGI soon," and, like, we're funding a bunch of companies that are helping, like, you know, already, I don't know if you've played with them, like not that we're in any way near AGI with this but, you know, you could tell Claude, "Come up with 30 ideas for a product to serve, you know, market X, then rate them all on quality and feasibility level. Then create-" this is one prompt, by the way, "Then create a, uh, a playable prototype of the interface for the application, then interview me as a user about how to change it and adapt it as we go." And it does it. Like, I get a little playable interface for a game, and I can then odd- edit the game and, you know, say like, "Oh, I wish it was more..." You know, it's- it's- it's just not fun enough in some way. And it's like, "Okay, great, I'll make it more fun for you." Like, if that cycle's really there, then you're rolling for world- uh, what is your stance on what an AGI world looks like becomes very relevant

14. 41:35 – 43:33

    Founders' Diverging Views on AGI Timelines & Funding

    1. EMEthan Mollick41:35

       is all I'm saying.

    2. HSHarry Stebbings41:36

       Do you find it interesting to see the different people's opinions on A- especially on the founder side, different people's opinions on the time to AGI and their requirements for funding? And so what I mean by that is like Demis and Zuck are very, uh, long term minded in terms of how long it will actually take to achieve AGI, and they also don't need any money, bluntly. And then there are other founders, who I will remain nameless, who are pumping it as being much sooner, but they need to present that future because they need the money.

    3. EMEthan Mollick42:08

       Yeah, I mean, I don't trust anything people are s- I mean, I think every- like people are very self-motivated, right? I think the signal you should pay attention to is that, you know, uh, is that people are betting their careers to a large extent, right, on this being possible. And I think there are people who care about their reputations. So that is a signal to me. They don't have to be right. I mean, look, I work with Marvin Minsky, like I said. Like, this is- that was, you know, th- he was there in the- in the '57 conference where they out- in Dartmouth, where they outlined, you know, uh, I- I figure what, right? Um, and, you know, the- the concept of AI- AI or at least his, his, his mentor was. Like, I mean, th- we're in a world where like, um, A- you know, AGI is always soon. So I think you would take everything with a grain of salt, but I think you need some coherence about your own viewpoint on this set of stuff. Now, the large companies, I mean, I- I think, you know, uh, um, the- we've seen a lot of, you know, people warning that this is coming soon. That, I mean, in the-You know, in the Meta, um, paper, the a- the paper outlining the release of LLaMA 3.1 that came out yesterday, it says, "We see exponentials continuing for the n- we don't see any reason why exponentials are gonna stop." What does that mean for you as a startup feels like a relevant question, and you're betting for a future world. So what does that future world look like? And you can't both say, "Everything is changing, but also I'm doing this minor thing." I also think crypto did us dirty in this kind of front, which is, like, it made all technology feel like hype, and it emphasized, again, short buck return, and, like, if you just believe something will happen, and I don't think that that's

15. 43:33 – 49:49

    Will You Thrive or Get Steamrolled?

    1. EMEthan Mollick43:33

       really a great way to think.

    2. HSHarry Stebbings43:34

       I actually liked Sam Altman when he said on the show the simple kind of heuristic of, like, whether you're gonna get steamrolled by OpenAI is would you be excited or scared by 100X improvement in our model? If yes, then you're gonna get, you know, steamrolled. If no, then great, but I liked it as a heuristic.

    3. EMEthan Mollick43:50

       But I don't think it's useful as a heuristic. What does that mean? Wha- what is 100 times better GPT-4? Like, it, it is a- it's a baffling heuristic to say better, right? Like, okay, it's, like, what does that mean, right? It's an uneven system. It has gaps in the world. So does that mean 100 times better re- like, how are you supposed to ac- like, this is what I mean. When you start looking at these things, it's like, "What the heck am I supposed to do with that? It's 100 times better. It's a machine god." Like, what? And so I- I don't like it as a heuristic because, like, I don't have any way to operate within that, right? I- and this is what I- like 100 times better, does that mean it will be able to process an entire legal document and do a very good legal review of a document on its own? Great. That's disrupts a huge industry, but that is a actual question about hallucination rates, its ability to handle words, you know, to think about words instead of tokens, to understand precedent, to be okay across different languages, to hold a huge amount in its context window. That feels like a useful question to ask. Um, will it be a- could it write an academic paper on its own, right? Where you give it a dataset, it generates hypotheses, tests them, writes a really good paper, fo- formats in LaTeX, writes a letter to the editor, and handles reviewer responses. We're getting close, but there's a lot of gaps there. Like, give me a concrete example of what this thing does, and then we can talk about a heuristic. But, like, 100 times better is a really hard one.

    4. HSHarry Stebbings45:09

       No, those use cases are 28 times better. They're not 100 times better, specifically 28 times. (laughs) I'm joking.

    5. EMEthan Mollick45:16

       Well, but, well, but maybe, but ma- but, but I mean, I think that's a valid question. Some of those things are huge gaps, right? Some of those things are small gaps. If you asked me about that reviewing the legal document, not really a problem, right? We're close to that. But, like, if it does that out of the box, that also implies a lo- lowering of hallucination rates below a threshold that they're not currently at, and we're not seeing... There's no benchmarks in hallucination, so we have no idea how good we're getting on the hallucination rate side. It also, though, implies the ability to, you know, to, um, seamlessly move between different perspectives. We can do that with agents today. Is it an agent-based model that's taking action? Like, there's so many questions, and I just think there is a degree of, like, I would love some specificity, and that's why I'm saying field-specific is great. If you are a lawyer who knows the law field really well, then you probably might have some interesting things to think about and where the real gaps are or not, and I don't think a lot of the, the AI firms know that. I know this 'cause we're deep working with all of them on things like education, and, like, they don't really understand education. There's no educators there, so they don't really understand what teachers do, and they don't really understand what classrooms are for. And so it's all the AI will replace everyone, and I think we're a lo- we're not there.

    6. HSHarry Stebbings46:22

       You, you said about the im- I love that as a- that was a trigger, wasn't it? (laughs) Uh, just give y- give you a Sam Altman heuristic. (laughs) Even I was like, "Whoo!" Um, tell me, you said there about kinda the importance of being opinionated, uh, and for startups to have strong opinions about where, you know, AGI will be, how they fit into it, organizational design. If I were to ask you and flip that on you, where are you most opinionated in your views around it? Where would you suggest or point to first?

    7. EMEthan Mollick46:48

       So I think education is a good starting point. We could talk about entrepreneurship and other areas, but in education, tutoring is the gold standard for, for interventions, based on the research we have. And AI's an incredible one-on-one tutor. Like, it's transformative. So- but the- but when I find Silicon Valley people and, you know, and, um, and, and AI in education people often think is, like, well, once we have a really good tutor, we don't need teachers, or, like, I hated this, the subject in school, or people will be self-motivated to learn. Absolutely untrue. People are not self-motivated to learn. Like, and even all the computer scientists out there were like I was, like, "Yeah, you're autodidacte at some narrow area, but you would've learned nothing about eh- very important topics because you only cared about one topic," right? It's like, people need extrinsic motivation to learn. It turns out that there's value in having an instructor guiding the direction of a class, that there's value in putting things into practice. So even an incredible AI tutor that knows you and loves you really well doesn't sub in for teachers. And also forget all of that. Let's talk about systems. Schools are in a complex system of society and where they are for, you know, providing daycare services, to how they fit into educational networks, how we do credentialing, to th- teacher unions, to... Like, there's a billion things about schools that don't get replaced by AI by having a magical button you push to make stuff happen. So there's gaps and opportunities that are very different than a naive view of how education changes.

    8. HSHarry Stebbings48:10

       So one of the biggest problems in UK education today, I'm not sure if it's the same in the US, so you can tell me, but it's the, uh, exponential increase in class sizes that we've seen, particularly in public schools, which is, you know, the school's provided by the state, um, and e- the quality of education's gone way down. When we look at AI's ability to increase education standards, will we see the ability to maintain high education standards with increasing class sizes? Like, how do you think about that?

    9. EMEthan Mollick48:39

       I mean, I hope so, but let's just talk. The first r- uh, randomized controlled trial we have, I have some of my colleagues at Wharton, um, was, uh, giving GPT-4 people for math tutoring in Turkey. Now they didn't do a huge amount of, like, you know, it was an assigned class, and they used the system. But it turns out that everybody who used it, um, uh, you know, just used GPT-4 without any special prompting or anything else had much higher homework scores and then did much worse on the tests, because basically the AI just did the work for them.... right? And once you have better, once you have better prompts that, that effect disappeared, though we didn't see educational gains from it, but I think it's a early sign of, like, not being naive about how these systems operate, right? Like, we need to put the work into building scaffolding around them. I absolutely believe that we can't be naive about the- the work that needs to be done here to make this stuff operate, so you can't just drop these systems in. But a good tutor will make a difference. I think, in the long term, we'll have flipped classrooms where that 20 per- where that giant classroom is actually fine, because a lot of your learning is done outside of class with, you know, AI tutor help, and then inside of class will be activities, exercises, application, where large class size doesn't matter as much. But there is a road to get from here to there.

16. 49:49 – 57:33

    The Future of Education with AI

    1. EMEthan Mollick49:49

    2. HSHarry Stebbings49:49

       I think this show's done well because I'm not scared to admit my own flaws and stupidity. Everyone talks about, kind of, the incredible optimism that AI brings for education and talks about tutoring. Great, but I kind of don't really understand what that picture of the future of education looks like, then. Does that look like, when kids come home from school, they just have Perplexity or OpenAI up and they have another tutor with them, where, as you said, in most cases they end up doing the work for them and so they don't learn? Is it a crutch that then they... I- I don't understand, actually, intangible reality. What does the future of education look like with AI, and why is it optimistic?

    3. EMEthan Mollick50:29

       First of all, there's a couple things you need to know about learning that people don't tend to think about, which is learning is hard and sucks, and that what makes you feel like you're learning isn't what's learning. Like, you have to do grinding work. There's no solution to it. It's just like any other thing, like exercise or anything else. It- it- uh, you have to be pushed to desirable difficulties where you're having trouble. If you're not failing at a pro- at a thing, you're not working hard enough. Like, there- that's e- which is why you often need intrinsigmo- extrinsic motivation. And the second thing is, we actually have some research. We know things like active learning, where you're in a classroom doing activities, um, beats the idea of passive, just receiving a lecture. When we have those sets of pieces, there's been a move that kind of fizzled called flipped classrooms that had some early evidence in its favor, which suggests this idea of, like, classroom should be about doing stuff, and outside of class should be about getting the basics, right? Because we can get you to do stuff in- in the classroom setting. So, that would mean that outside of class th- and what that practically meant is you watch videos outside of class of your teacher talking, so the lecture stuff is all outside of class. That's your homework. Read the book, do that. Then your homework is in class, where you can mess up in front of people and work in teams and, you know, that you learn by kind of watching other people and how you're doing it. The teacher can help you solve problems. I think flipped classrooms are a very natural fit, uh, in active learning with AI-based approaches. So instead of having a passive video you watch, you'll have an AI tutor outside of class. You'll log into the school's website, and that tutor will be amazing. It'll be adapted to you, and then it'll pass that information on into the classroom setting where you actually, the teacher gets advanced stuff. And by the way, we've actually built a version of this already at the Journal of AI Lab at Wharton. We'll be open-sourcing all of that, like, that does this kind of stuff. It's not that hard to imagine. We just have a ways to go still.

    4. HSHarry Stebbings52:06

       Is that really an order of magnitude improvement if we compare that post-classroom? You could give me incredible high-quality videos of you talking, lecturing, giving examples that you give to your students now, very easy to do, versus that AI tutor. Is it, is it 20% better? Sure, maybe it's personalized, but is it really an order of magnitude better?

    5. EMEthan Mollick52:27

       Education is a complex system, so I think order of magnitude's a very weird thing to talk about, because every student has their own talents, abilities, interests, and gaps. The early work on- in one-on-one tutoring, we don't talk about order of magnitude improvement 'cause that doesn't really work in the education world. This is very hard to say what an order of magnitude is, but we can talk about grades a lot. And the classic study that is probably- would not be replicable but it sets up our model is that one-on-one tutoring, um, accordin- i- i- it creates a two sigma increase in, uh, in classroom outcomes. That's two standard deviations, which is, you know, a fairly huge improvement. You go from the 50th percentile to the 97th percentile in class. We have no idea if that's gonna hold up with, you know, AI tutoring. But if we could do that, that is as amazing an improvement as you could possibly ask for. I mean, a 10% improvement's amazing. I- I kind of feel like we- aiming for order of magnitude education, if we can get improvement in a system, we're in great shape.

    6. HSHarry Stebbings53:20

       I also think that doesn't include a lot of different elements, like you mentioned extrinsic motivation being a big part of it. I think a big part of, like, having a tutor means you actually have a bond with them. You want to impress them. You want them to feel proud of you. Does that extend to an AI tutor, where you don't have that human?

    7. EMEthan Mollick53:36

       Maybe. I mean, w- we actually... It's not clear that that is the key to tutoring is the bond with the human being. It- it- it seems to be that across a wi- wide variety of tutoring approaches, that forcing people to confront what they don't know turns out to be a lot of the value of tutoring. So tutoring is also o- uh, often reflective back. So it's like, how do you... So when we built a tutor, um, a tutor chatbot, right, what that tutor chatbot... Like, the way we test, by the way, education technology chatbots, our rule of thumb is that if it asks you if you understand a topic or you're ready to move on, it's a bad tutor, because humans don't know when they're ready to move on or not. What the AI should be doing is asking you questions, probing what you know, and making you expand on what you don't understand, then helping you fill those gaps. So, it's not the one-on-one bond. Uh, there is a, there is, uh, we, there are, like, methods to teaching that we actually know make a difference. Self-reflection makes a difference, right? Repeated practice makes a difference. Low-stakes testing makes a difference. Like, there, this i- and this is kind of, like, to zoom back out to what we were talking about before, subject matter expertise is gonna be absolutely critical in making AI work. It's a system that e- experts... I can look at a prompt in entrepreneurship and education and instantly tell you about whether that's gonna work or not, or whether it's a stupid idea or a good idea, or whether the subtleties that the system is missing are a problem or not, because I am an expert. And if you're not an expert, you're gonna be like, "That looks really good." So, like, expertise actually matters. I'm sure in the same way... You know, it's one of the things I actually, when I talk to my students in, um, you know, and teach them how to pitch, right? One of the things I talk about is, there's this really interesting research that shows that venture capitalists are not swayed at all by the quality of the- the speaker.... their ability to be a good speaker or not is absolutely irrelevant. Amateur e- and angel investors are swayed by that. Why? Because you're an expert. You've seen so many pitches that you instantly see through all of that stuff, and you're like, you know what the core issues are right away 'cause you've seen 10,000 pitches. You've seen how they play out in the world, and, you know, you have to be a really amazing speaker to pull off, "I'm persuasive," on top of that, right? Um, and so in the same way, I think expertise is gonna matter a lot here.

    8. HSHarry Stebbings55:39

       When you look at the pervasiveness of AI and specifically ChatGPT in homework and in coursework and in the answers that many students give today, is there any point in university or educational facilities doing homework or coursework when it's largely done by AI today?

    9. EMEthan Mollick55:55

       Of course there is. We, like, uh, everybody was already cheating. Like, if- there was this great study at a reputable university that found that homework im- im- proved, when you did the homework, it improved something like 80% of people's test scores in 2008, and by 2020, it only helped 20% of people. And that's not 'cause homework stopped helping. It's because everyone was cheating. And so we have ways around this. There, there's really two options in how to use AI in education. One of them is to ban it cautiously, right? People are still gonna use it as explainers and stuff like that, but you, you have in-class tests and blue-book writing. Like, we solve this problem in math, and like, you make people do exercises and do work. Nobody likes it, but there's no shortcut to learning. It sounds dumb. It's like what your teacher said. Oh, it turns out it's true. You need to do a grinding amount of work to understand something. You need to do interleaved practice. You need to... Like, there's a lot of stuff you need to do to learn something. And so we absolutely can make you do blue-book work in class. We absolutely can install, um, terrible monitoring systems. I don't like this approach, but like, a couple of companies already have this, that watch what you're typing and make sure you're not pasting stuff in from AI. Again, I don't necessarily recommend it, but like, it, these are possibilities. Like, I think people are underestimating how much you can do those kinda things. Homework is valuable. Cheating is bad. What is AI cheating? We have to define that. I'm a big- um, but the other option is transformation. My classes are 100% AI-based at this point. The students have AI mentors and tutors they talk to. They have AI-based assignments. When they learn how to do hiring, I built a, we built a simulator that actually makes them have to fake hire somebody, um, and the AI plays the person they're interviewing and gives them multiple-choice answers, and then they have to reflect on the assignment. There's, uh, one of the other assignments is they have to teach the AI to do something. You could do really exciting stuff. It's just not gonna happen right away.

17. 57:33 – 1:00:00

    Energy Demands & Compute as Currency

    1. EMEthan Mollick57:33

    2. HSHarry Stebbings57:33

       All of the different avenues, functionalities that we've spoken about require a lot of intense compute. Um, uh, considering there was such a trigger when I gave you the last (laughs) quote from Sam, thought I'd give you another one. Uh, which is, uh, "Compute is the currency of the future," what Sam Altman said, and, "Energy is a concern when looking at the energy requirements that this next generation of AI will bring." How do you think about the energy requirements required for this next generation of AI usage in society and whether Sam is right that, you know, com- compute is the currency of the future?

    3. EMEthan Mollick58:05

       Sam believes in AGI, and he believes that it's gonna be achievable in the near term, right? And when you a- talk to OpenAI insiders, they feel the same way. If that's the case, if, if intelligence on demand is the case and intelligence on demand is power hungry and there's infinite demand for intelligence on demand 'cause there will be, right? Like, if you have an AGI, I want that to be looking over all of my medical records and monitoring our airspace and, you know, finding scientific ideas and helping me with a project I have to do and also booking tickets for the ultimate trip. Like, there is infinite demand for intelligence, right? So then compute becomes the currency, and energy becomes the big deal, and we're gonna build a lot of nuclear power plants, I guess, in, you know, in relatively short order. Um, and, you know, it seems like that's a pro- or AGI figures out how to do fusion, and it doesn't matter, or we all get turned into batteries a la Matrix, although we don't produce enough wattage, um, uh, you know, so I, I don't think that's really the issue. Uh, training data, that's what the AI will use us for. Um, but anyway, the, um... mostly joking. Right now, I think the energy debate's an interesting one because it's, again, it's one where doomers and, you know, and, uh, and optimists sort of like to talk about 'cause on the downside risk, when I meet people who are skeptical about AI, the first thing they talk about is energy use. And the truth is that AI uses a lot more energy per query, we don't know exactly, probably two orders of magnitude, than a Google search, but a lot less orders of magnitude of energy than a human doing the same amount of work, right, with a laptop. You know, how do we balance those kinda things becomes an issue, right? Right now, 1% of US power goes to data centers, and maybe 10% of that goes to AI at most. So we have a lot of room left at the top before this becomes an issue. So again, we're assuming AGI is available, instantly useful, and in which case, absolutely compute becomes and energy becomes the, the issue. But then that becomes a reverse salient, right? And you know, there's a lot of money to be made that if, if, if the currency of the future is compute, and compute is energy, then there's a hell of a lot about money to be made in building your own nuclear power plants, and so people will be doing

18. 1:00:00 – 1:04:40

    The Role of AI in Future Electoral Systems & Politics

    1. EMEthan Mollick1:00:00

       that.

    2. HSHarry Stebbings1:00:00

       The final one before we do a quick fire, uh, a friend of mine who's also, uh, quite a well-known venture capitalist, Jeff Lewis, said that when it comes to democracy, in the future, we will vote for algorithms, not for people. To what extent do you think AI pervades into electoral systems, electoral voting, uh, the political fabric of our society?

    3. EMEthan Mollick1:00:19

       When something feels like a dystopia to most people, it probably is something that's not gonna happen very quickly. Um, uh, you know, uh, human systems are complicated. I, I just keep seeing this technological view, which is like, you know, in a rational world, the machines will rule us all. It's just people don't want that, right? So like, you know, we already have algorithms ruling lots of what we do. You know, your FICO score determines a huge amount of, of, you know, things that happen in your life. Um, and that's an algorithm. Like, we have these kind of systems in place, but the idea of an overall all seeing kind of approach, uh, it's, it's hard. Like, now, on the other hand, we do find that AI is hyper persuasive already, right? In a controlled experiment where you do, where you're asked to be- talk to a normal person versus the AI, you're 81.7% more likely to change your views to the AI's view than to a human's view. That is going to change marketing in very big ways, which is gonna change politics, right? Deep fakes are going to... are a big deal already. Although it's been funny how little a big, uh, big deal they are because it just turns out all you need to do is show a video of politician X talking and say, "I can't believe he said he's gonna eat babies," in minute three, and everybody shares it online who should know better and without actually watching the video at all. Like, when I post have a viral tweet, nobody clicks the link, right? So like, I feel like we way overestimate people and therefore how much this stuff was gonna matter. But in a world where AI is hyper persuasive, this does change things. In a world where AI gives really good advice on everything, people should have an AI second advisor happening in every role, including in politics, right? That would make things better. But people aren't gonna listen to it. Politics changes much more slowly and is much more human than people think.

    4. HSHarry Stebbings1:01:48

       I always say this with founders we work with when they have, like, funding announcements or press releases or whatever. I say, "Nothing matters in the piece, but we want a good title. We want a good title." Because the only thing that shows on social in that, like, little thumbnail is, you know, Ethan endorsing, you know, wha- whatever that big title is. Uh, no one clicks on the actual article and goes down to it. (laughs) Like-

    5. EMEthan Mollick1:02:08

       Yeah.

    6. HSHarry Stebbings1:02:09

       ... so I totally agree.

    7. EMEthan Mollick1:02:10

       I, and I think that, by the way, plugs into larger issues of like, when we can produce all this stuff on demand, what- what is actually valuable or not. I mean, everything is gonna change. It's very hard to make predictions how a general purpose technology rolls out. But I do think people overestimate how quickly the short-term change is gonna be, and as usual from Moore's Law, underestimate the long term.

    8. HSHarry Stebbings1:02:29

       I completely agree with you there. Well, my biggest concern actually, uh, you know, as a content creator in many respects is with the infinite supply of content that value goes down and discovery becomes much more challenging. That is a big concern.

    9. EMEthan Mollick1:02:42

       Uh, I mean, that problem has already happened, right? I mean, uh, you know, to me, the really interesting thing is like, I mean, Suno and Audio and com- like, they're getting pretty good. Like, at what point does having an AI-generated song playlist, you know, that has a couple real musicians but also makes up songs based, like, how, like, that doesn't feel as far off for, in terms of people enjoying it. Like, what happens to content creation is a very big deal, right? I mean, like, you know, uh, I'm- I'm an author. My book's a New York Times bestseller. That's amazing. I don't think people realize how few copies you need to be to be a New York Times bestseller. Like, you're, if you're, like, you're selling, like, 6,000 hardcover copies in a week, like, that's getting on the New York Times bestseller list. Like, these are, like, attention's already scattered across a huge amount of content. The one thing you'd hope for is maybe AI creates better connections, right, and, you know, in some ways.

    10. HSHarry Stebbings1:03:31

        I'm not being rude, but could you not just bu- I know you haven't, but like, could y- could I not just do it, do a book, and spend $75,000 and be a New York Times bestseller, then?

    11. EMEthan Mollick1:03:41

        So people do that all the time, um, and, um, the way the New Y- like, th- the New York Times has a small cabal of people who refuse to talk about how they do this. So they use the number ranking, but then they also try and exclude b- bulk buys. So they actually try and cut that out. So those, uh, you'll notice there's a little dagger next to the name of, uh, of, uh, companies on the bestseller list that they think that they're including but they still had potential bulk buys. I actually got the little dagger on mine because a company, um, bought 500 copies, which wasn't the main reason for the list, but they would've found that suspicious. Um, so they're trying to filter that out by hand. Um, so, but yes, you can often buy your way onto the list, and people do, do that all the time.

    12. HSHarry Stebbings1:04:20

        Yeah. No, I've seen many of my friends who have VCs who have books, and I'm like, "Really?" You know?

    13. EMEthan Mollick1:04:25

        Yeah. Th- there are ways of doing this. You scatter buyers across multiple locations, and they all do buy. Like, it is, uh, very much true that a lot of your, um, uh, unnamed VCs do seem to have asked a lot of friends to buy book copies of their book.

    14. HSHarry Stebbings1:04:38

        Amazing. I love that. Uh, listen, Ethan, I could

19. 1:04:40 – 1:06:14

    Quick-Fire Round

    1. HSHarry Stebbings1:04:40

       talk to you all day. I wanna do a quick fire round. So I say a short statement, and you give me your immediate thoughts. Does that sound okay?

    2. EMEthan Mollick1:04:45

       Sounds great.

    3. HSHarry Stebbings1:04:46

       What do you believe that most around you disbelieve?

    4. EMEthan Mollick1:04:49

       So I feel like the very simple idea that AI is very profoundly im- uh, is much better than people think and is gonna keep getting better is something that I think most people don't actually believe.

    5. HSHarry Stebbings1:04:58

       What's the most concerning future that AI could bring?

    6. EMEthan Mollick1:05:01

       The most concerning future, I think, is, uh, is one where we lose agency, and not necessarily to the AI systems, but to the systems that incorporate AI. What I mean by that is we have a chance to make AI be used for human thriving. That's not an automatic process, right? That means not firing people when you have AI in- in your company, but it means figuring out other uses for them that are valuable. It means building systems that help people feel like they're accomplishing more as a result of using these things. And I worry we're not seeing enough people modeling that kind of behavior, that it's all about just the technology itself and then how do we get cost savings.

    7. HSHarry Stebbings1:05:35

       What have you changed your mind on most in the last 12 months?

    8. EMEthan Mollick1:05:39

       I have gone back and forth on how much juice the technology has left, and now I'm back to the it has lots of juice left. Like, the exponential continues for a while. And I think I was, uh, not clear on that for a long time.

    9. HSHarry Stebbings1:05:49

       What caused that shift backwards?

    10. EMEthan Mollick1:05:51

        Accumulation of evidence, right? So we talked about Kevin Scott saying scale. Like, there's a bunch of people who weren't talking about scaling solving everything six months ago or eight months ago who are now more confident, which indicates to me another generation of models came out and everyone at all the labs are getting that haunted look in their eye again. Uh, I don't know when we'll see these models, but they're clearly, people are seeing things that indicate to me that there's more, more left in the curve, and they're all talking about it.

Episode duration: 1:09:06