Zico Kolter: OpenAI's Newest Board Member on The Biggest Questions and Concerns in AI Safety | E1197

1. 0:00 – 1:29

   Intro

   1. ZKZico Kolter0:00

      The real negative outcome is that people are not gonna believe anything that they see. It didn't even need AI to get there, but AI is absolutely an accelerant for this process. It is a relatively new phenomenon that we have sort of a record of objective fact in the world. Humans evolved at a time during a- an environment where all we could do was trust our close associates. That's how we believed things.

   2. HSHarry Stebbings0:22

      Ready to go? (upbeat music) Ziko, I am so excited for this, dude. I've been looking forward to his one for a while. So thank you so much for joining me today.

   3. ZKZico Kolter0:40

      Great. Thanks. Wonderful to be here.

   4. HSHarry Stebbings0:42

      Now, we're gonna discuss some pretty meaty topics. Before we do dive in, can you just give me the 60-second context on why you're so well-versed to discuss them and your roles today?

   5. ZKZico Kolter0:50

      So I- I seem to have- be collecting jobs here. I have a number of different roles. Um, but I am- I'm first and foremost a professor and the head of the Machine Learning Department at Carnegie Mellon. Uh, I've been here for about 12 years. And here, the- the Machine Learning Department is really kind of unique because it's a whole department just for machine learning. And I've been heading that up actually as of quite recently. Kinda get to immerse myself in the business and- and, uh, the thought of machine learning all day, every day. Also, I am, uh, recently on the board of OpenAI, uh, which w- I joined, uh, at this point a couple weeks ago, and it's been extremely

2. 1:29 – 4:17

   Understanding the Basics Behind Modern AI Technology

   1. ZKZico Kolter1:29

      exciting as well.

   2. HSHarry Stebbings1:29

      Now, I wanna start with some foundations and mechanics. When we look at kind of the basic techniques that underpin current AI systems, can you help me understand, what are the basic techniques today behind current AI systems?

   3. ZKZico Kolter1:41

      Let's talk about AI as LLMs, but with, of course, the context that AI is a much, much broader topic than this.

   4. HSHarry Stebbings1:48

      Mm-hmm.

   5. ZKZico Kolter1:48

      Um, LLMs are- are amazing. Um, the way they work, at the most basic level, is that you take a lot of data from the internet. You train a model, and I know that's a very sort of colloquial term that we use here. But basically what you do is you build a great big set of kind of mathematical equations that will learn to predict the words in the sequence that- that- that's given to them. So, you know, if you see "the quick brown fox" as your starting phrase of a sentence, it will predict the word "jumped." Uh, this is a common phrase we use to, I think, use every letter in the English language, uh, (laughs) in a single phrase. People often use that... A- and that's what it does. To be clear, we train a big model on predicting words on the internet. Um, and then, when it comes time to actually speak with an AI system, all we do is we use that model to predict what's the next word in a response. This is, to put it bluntly, a little bit absurd that this works. And I think people often- often sort of... So there's sort of two- two chains- two- two philosophies of thought here. People often use this sort of mechanism of how these models work as- as a way to dismiss them o- o- oftentimes. I know. People say, "Oh, well, AI is- it's just predicting words. That's all it's doing, therefore it can't be intelligent. It can't be..." And I- I think that's just demonstrably wrong. What I think is amazing though is the scientific fact that when you build a model like this, when you build a model that predicts words and then just turn this model loose, have it predict words one after the other and then chain them all together, what comes out of that process is intelligent. And I think it's demonstrably intelligent, right? I- I really believe these systems are intelligent, definitely. And I would say that this fact, you can train word predictors and they produce intelligent, coherent, long-form responses, is one of the most notable, if not the most notable, scientific discovery of the past 10, 20 years. Maybe much longer than that, right? May- maybe it was much deeper than that, in fact. And so, I really think that, um, this is not oftentimes given its due as a scientific discovery because it is a scientific discovery.

3. 4:17 – 9:08

   Data Availability & Synthetic Data

   1. ZKZico Kolter4:17

   2. HSHarry Stebbings4:17

      Can I just dive in and ask-

   3. ZKZico Kolter4:18

      Absolutely.

   4. HSHarry Stebbings4:18

      You mentioned there the element of kind of the data input being so necessary. Everyone or a lot of people think that we've plundered the kind of resources of data that we have already. We will need synthetic data to really supplement the data that we already have, or we need to create new forms, be it the transcription of YouTube videos, which is like 150 billion hours or whatever that is.

   5. ZKZico Kolter4:37

      Yeah.

   6. HSHarry Stebbings4:37

      To what extent do you think it's true that we've plundered the data resources that we have available and we are running into a data shortage crisis?

   7. ZKZico Kolter4:44

      I think there's sort of two- two kind of answers to this question, uh, which are d- diametrically opposed, so... As- as with many- as with many questions, right? Um, because you're exactly right. You know, the thought is because these models are built to basically predict text on- on the internet, um, if you- if you run out of text, then they kind of... That would imply that they're kind of plateauing, right? That would imply that they're sort of reaching a limit. Um, I don't think this is actually true for- for several reasons, which- which I can get into. But just from a raw standpoint of training these models, I think there's- there's two ways in which this is sort of maybe true, maybe false. Um, it is true that a lot of the easily available data, sort of the mo- the highest quality data that's out there on the internet has- has- has been consumed by these models, right? We have- we have used this data. There is not another Wikipedia and things like this, right? There's- there's only so much really high-quality, good text that's available out there. On the flip side, and this is the point I often make, um, first of all, we're only talking about text there. We're only talking about publicly available text. If you start talking about internally available text, stuff like this, from a very straightforward standpoint, we have not gotten close to using all the data that's available. Public models are trained on the order of, you know, 30 terabytes of data or something like this, right? So 30 terabytes of text data.It's compressed a little bit less. This sounds like a lot, but this is a tiny, tiny amount of data. If you, if you load up a few micro SD cards, this will literally fit in the palm of your hand. It is a tiny amount of data. And there is so much more data that's available that we are not using right now to build these models. And of course I'm thinking about things like multimodal data, stuff like this, video data, audio data, all these things. We have massive amounts available. I mean, just a few, you know, tens of terabytes is not the amount of data these large companies that index the internet are, are storing. There is so much more data than this, and we have not really come close to tapping that whole reserve. Now, whether or not we can use that data well, right, because text data in some sense is the most distilled form of a lot of this, and a lot of this is not textual data, that remains to be seen. But we are nowhere close to hitting the limits of available data in these models. Arguably we're unable to process it because we don't have enough compute and things like this, but we're nowhere close to data limits in other senses.

   8. HSHarry Stebbings7:11

      What are the challenges of using these new forms of multimodal data well?

   9. ZKZico Kolter7:15

      I think the biggest challenge is simply compute. If you have something like video data, just think about the size of a video file versus a text file. So if we transcribe this podcast, you know, it would be a few kilobytes. If you take the, the dump of video from it, it'll be on the order of, I, I don't even know, uh-

   10. HSHarry Stebbings7:32

       I, I, I do. It'd be about 6.5 megabytes.

   11. ZKZico Kolter7:35

       Okay.

   12. HSHarry Stebbings7:35

       Uh, 6.5 gigabytes.

   13. ZKZico Kolter7:36

       Gigabytes, exactly, right. So we- we're- we're- we're- we're many (laughs) tens of thousands of magnitudes of difference, uh, orders of magnitude of difference, right? Now arguably, you know, depending on people's opinion, maybe the entirety of the actual valuable information is not in, you know, the audio of my voice and the video and maybe, I mean, maybe the stuff behind me is real valuable, right? But (laughs) ignoring that for a moment, um, you could argue that the, there's not as much usable content there. But I think if we talk about kind of what we think of as, as humans and, and, and the, or (laughs) when we think about what we think about, about what kind of data humans use, I would argue that visual data, uh sort of spatiotemporal data, this is hugely important to our conception of intelligence, right? This is hugely important to the way that we interact with the world, the, the way that we sort of think about our own intelligence. And so I can't fathom that there is not a value to many, many more modalities of data, be it video, be it audio, other time series and things like this that we sort of don't quite, that, that, that are not audio but they're sort of other sensory signals, stuff like this. There are massive amounts of data available, and I think we have not yet figured out how to properly leverage those due to either limitations of compute. I mean, you have to process all that data, and it does take... W- we don't have current models to do this very well. Or just d- due to limitations in sort of how we transfer it and generalize across these modalities here. But I don't, I think there has to be

4. 9:08 – 16:14

   Why AI Performance Doesn't Plateau Despite Data Limits

   1. ZKZico Kolter9:08

      a use for it.

   2. HSHarry Stebbings9:09

      If we just took it to a logical extreme though and said we had plundered the reserves of data, and you mentioned that even if we had, we were not seeing a plateauing in performance of models. Why is that? Because one would alwa- assume so.

   3. ZKZico Kolter9:22

      There are a few different, different sort of notions here. One is just the fact that we are still seem to be in a world where you can increase model size and get better performance even with the same data. Um, so obviously the real value of bigger models is they can suck up more data, right? They're able to ingest more and more data. But it is also true that if you just take a fixed dataset and run over it multiple times, if you use a bigger model, it will often work better, right? So I think that we have not really, we have not really, uh, reached the plateau there. The other thing though, and I think this is, this is a, this is maybe related to your point on synthetic data, and these, these ideas are in the air. We don't, we don't really know the right way of doing this right now. But what I would say is I don't think anyone would argue or m- most people would, would not argue that the current models in some sense extract the maximum information possible out of the data that is presented to them. And a very simple example of this is if you train a classifier just to, you know, to, to, to classify images of cats versus dogs on a bunch of images, you get a certain level of performance. If you train a generative model on those exact same images, generate more synthetic data from that generative model, and then train on that more synthetic data, you don't do that much better but you do a little bit better. And that's just wild. What that means is our current algorithms, we are not yet maximally extracting the information from data we have, and there are way more deductions and inferences and other processes that we can apply to our current data to provide more value, and as models get bigger and, and better, they arguably can kind of do this themselves either through synthetic data or through different mechanisms by which we train these models.

   4. HSHarry Stebbings11:14

      When we think about optimizing the data that we have in terms of kind of value extraction, what could be done further to get further value from the data that we have?

   5. ZKZico Kolter11:24

      I don't really know, to be honest. I think this is a major open question right now in research. How do we extract the maximal information content from these models, uh, or sorry, from the data that we have? Um, but again, as I said, I, I don't think we're close to even extracting all the data that's available, so there's... When I look at this landscape, right, and we know that we aren't close to extracting the maximal information in sort of, you know, the closure set of all the data that we have available, and we aren't come, we have not come close to processing all the data that's available to us. The idea that somehow this is a, a recipe for models plateauing in performance just doesn't jive to me with the reality of what we see.

   6. HSHarry Stebbings12:03

      Okay. So we're in the classroom together. We've got a big cross on that, like data, not the bottleneck. Good. Uh, what models...Honestly, again, the, the choice of this show is I can kind of just-

   7. ZKZico Kolter12:13

      Yeah. (laughs)

   8. HSHarry Stebbings12:13

      ... regurgitate statements that other smart people have said and test them. Everyone talks to me about kind of moving to this world of many smaller models-

   9. ZKZico Kolter12:20

      Mm-hmm.

   10. HSHarry Stebbings12:20

       ... which are maybe more efficient. To what extent do we agree with that? Is that right? How should we interpret that?

   11. ZKZico Kolter12:29

       I sort of don't really know here, um, to be honest. I think we have not yet... We have not yet reached an equilibrium point where we have a good sense of sort of what the steady state of model, size, and for what application, and how it's being used, and, you know, is it being used as a general purpose system or for a very specific reason. Um, this is all still being figured out right now. What I will say is that I use these models very regularly for my daily work. I work almost exclusively with the largest models that are available to me because it just works better. And when I don't have a given task that I'm doing over and over, when I want to have that generality, I want to work with the larger models that are available. The notion of sort of small language models and this kind of stuff... And I- and again, I think this might be very much a possibility in the future. It kinda comes after the- after we reach this point of generality, right? So once we've done something enough and we realize, okay, there is still a small task we wanna do many, many times. Maybe before we would have used custom-trained machine learning model for this, but the idea is that once you have a task, a rote task that you're repeating again and again enough times, and you know a small model can do it, it probably does become valuable to specialize a small model for that task only. But to be honest, I think this whole dynamic is still to be played out. We just don't know what the equilibrium point's gonna be and what kind of models are being used.

   12. HSHarry Stebbings14:09

       I had Aidan on from Cohere the other day, and he said that it is harder and harder to see visible gains in models given now the incredible performance and knowledge of them. And so before, you used to be able to kind of take anyone off the street and they'd be smarter than the models. But now actually, the models have got so smart, it's kind of harder and harder to distinguish, and it's almost getting to that kind of 92% versus 94%.

   13. ZKZico Kolter14:33

       I think that actually has much more to do with our benchmarks and the way people typically are used to using these models than the models themselves. If you look at some of the hardest problems that models sort of face, we are still seeing gains to, to larger models of different techniques, things like this. I think part of the problem here actually is sort of, uh, the- these models are a victim of their own success. People have started to use these very regularly in their daily lives, and they probably have a suite of questions that they ask these models, you know? When they first interact with a model, they'll ask a quest-... They'll probably ask it to write a biography of, of, of, or a history of your school or a biography of yourself or something like this. But you have a suite of questions you sort of ask these models. And on a lot of these preformatted questions that you know models already kind of do well on, the newer models don't do notably better, right? So if I say, "Write a history of Carnegie Mellon University," um, LLaMA's 7 billion can do that just fine, right? Or 8 billion now. Can do that just fine, right? There's no need... I mean, maybe it'll be a little bit better with the, with, with the largest closed source models, but these aren't the kind of questions that are, that are relevant. Um, the domain I use models for most probably is coding and also doing things like transcribing lectures and stuff like this. On those tasks, I am absolutely not seeing plateauing gains. The latest models, they are notably better than the previous iteration and just make my life ease- um, let me move up to sort of higher and higher levels of abstraction when I give them instructions, when I interact with them, and when I work with them. So I think this perception has more to do with people's limited imagination of what they can do with these models and less to do with the models themselves. But that will, that will evolve over time. People will start figuring out you can use them for better and better

5. 16:14 – 19:09

   How Will AI Models Evolve Amid Rapid Commoditization

   1. ZKZico Kolter16:14

      things.

   2. HSHarry Stebbings16:14

      One thing that I, I struggle when I look at kind of model ecosystems is just the commoditization of models. I remember, you know, a year ago, 18 months ago, it was so expensive, so hard. There were so few place. Now that there's so many, the commoditization, the reduction in cost. How do you expect this model landscape to play out given the commoditization being really one of the fastest commoditizing technologies we seem to have seen in years?

   3. ZKZico Kolter16:37

      I think that it's been evolving so quickly between, uh, recent releases of open source models, uh, continued progress in, in, in a lot of the, the closed source models. There was also this sort of prolif- proli- proliferation early on of a lot of open source models that sort of w- none were better than the other, and they just involved a lot of training kind of for companies, for lack of a better word, to just demonstrate that they could do it too. It's not clear that's a valuable thing, right? I mean, why would you want to train your own language model from scratch if there are very good open source ones now? Will that continue? Maybe, maybe not. And I think there will be most likely consolidation, but I'm not quite sure how it will play out.

   4. HSHarry Stebbings17:15

      What do you think the model companies that do survive and win, what decisions do you think they'll make? When you said about kind of the proliferation of operating systems and-

   5. ZKZico Kolter17:23

      Yeah.

   6. HSHarry Stebbings17:23

      ... only a few survive. What decisions do you think the model companies that survive and thrive will make to survive and thrive?

   7. ZKZico Kolter17:29

      There are a lot of companies that are right now thinking about training their own models and things like this, and it's just sort of the default that, "Of course you would do this," that this was- won't be an economically viable thing to do in the future, and so it won't happen anymore.

   8. HSHarry Stebbings17:47

      Can I ask, we've mentioned data.

   9. ZKZico Kolter17:49

      Yeah.

   10. HSHarry Stebbings17:49

       We've mentioned models. The, the third kind of pillar is something you've mentioned quite a few times, which is compute.

   11. ZKZico Kolter17:54

       Right.

   12. HSHarry Stebbings17:55

       And people are saying now, you know, "Ah, we've got to the stage of diminishing returns. More compute isn't leading to an aligned level of performance in models." Um, "We've really reached this kind of diminishing returns bottleneck." To what extent is that true or do we have a lot more room to run and throw and compute?

   13. ZKZico Kolter18:13

       I'm not really sure what...... the rationale is for saying that we've, we've plateaued in the compute sense. Um, most scaling laws that I've seen certainly suggest they can keep going. It's more expensive. You could argue that it's by far just scaling, is may not be the most efficient way to achieve better results, and I actually think that's very likely true. There are other better ways, you could argue, to achieve the same level of improvement than compute. But compute still does seem to be both, A, a major factor, and B, still seems to improve things. So I'm, I'm, I'm not quite sure. It's, it's more a calculus about also, you know, the, the, the monetary trade-offs of how much models will, will cost at inference time and how much they cost to train and all this kind of stuff. These are much more, I would say, kind of becoming more practical concerns than a concern about the actual limits of scaling.

6. 19:09 – 27:55

   Are Corporations Pursuing AGI or Profitable AI Products?

   1. ZKZico Kolter19:09

   2. HSHarry Stebbings19:09

      To what extent do you think the corporations that we mentioned are chasing AGI superintelligence versus making amazing products and leveraging AI to make them and make more money?

   3. ZKZico Kolter19:21

      Right. So, so this question is, is I think super interesting, and tho- those are not actually mutually exclusive, to be, to be clear. One thing I'll also say is the, the term AGI is thrown around a h- a whole lot. I define AGI as a system that acts functionally equivalent to a close collaborator of yours over the course of about a year-long project. So this is something that, you know, you would value as much as a close collaborator, you know, a student of mine or a, or, or a colleague of mine over working on a project for a year. Um, it's fine if this is virtual, by the way. Uh, I, you know, I think embodied AI is gonna take a little bit longer to, to reach us. But let's think about that definition of AI, or maybe even more so since we always think less of... I mean, I, I get this comment a lot is, is to say, "Well, AGI could automate this other people's job but, but, but not mine." Let's think about me. AGI would be a system that could automate everything that I do for the most part, maybe, you know, besides the, the, the softer kind of emotional qua- qualities that I, that I bring maybe. But AGI could automate everything that I functionally do in my job, uh, over a year. So you know, everyone that comes to me, that comes to me right now to ask me things, uh, you on this podcast, you would just say, "You know, I don't really care about talking with, with Zico. I'm just gonna talk to the AGI instead 'cause that's gonna be as valuable as, as, as Zico is." That's a pretty high bar. Um, and I am massively uncertain as to when this will happen. But a massive shift that I've undergone is I think this will probably happen in my lifetime. I think the answer to AGI has always been in academia not in my lifetime. And the, the timeframe I give this right now is I think this is between, you know, four and 50 years or something like this, right, which really captures my massive uncertainty. I mean, I don't personally think it'll be on the low side, but I have a hard time dismissing it also given the rate of progress and the things that I sort of see evolving here. We have to take that possibility very seriously. This is not some-

   4. HSHarry Stebbings21:19

      Is that... Is that not the same with all new technology introductions to society? There is a gradual curve and there is, you know, employment displacement, there is societal upheaval, and that is a natural cycle with technology.

   5. ZKZico Kolter21:33

      I actually also, uh, uh, agree with you, uh, that we will adapt to it. I don't wanna downplay the extent of transformation that might be necessary here. But I also think that the winners in this new world will... the companies that, say, survive and, and thrive and become dominant in this new world. Not talking about the AI companies for now, talking about sort of the rest of the companies that... the ones that people worry that they're gonna fire all their workers because, you know, they can have an AI replace them. The ones that succeed best will not be the ones that fire all their workers to have an AI that does the exact same thing as their old workers. They'll be the ones that understand, okay, what's changing and what are the things that people can best do now in terms of steering these systems, in terms of sort of providing the overall guidance and framework about where we want to go with our... (laughs) with, with all this intelligence. The companies that survive I think will be the ones that best leverage their workforce to achieve... to make the best use of this new technology.

   6. HSHarry Stebbings22:41

      Do you think the current providers of models in particular give a particularly good on-ramp to consumers for how to leverage their technologies best?

   7. ZKZico Kolter22:50

      This is actually a very nuanced question. Do we have AI products that are able to be maximally used by workforces? And the answer to this is... right now is no. Clearly, there is a gap between what people could use these things for and what they're using them for right now.

   8. HSHarry Stebbings23:12

      For large enterprises, the big concern is actually just the, the kind of mobility or transferability of their data. Uh, they want everything on-prem. There's a big-

   9. ZKZico Kolter23:21

      Sure.

   10. HSHarry Stebbings23:21

       ... unwillingness to have anything trained on their data.

   11. ZKZico Kolter23:24

       Yeah.

   12. HSHarry Stebbings23:25

       Do you think we will see AI bring back a movement from large enterprises away from the cloud back to on-prem?

   13. ZKZico Kolter23:33

       I mean, I find this kind of, kind of interesting in, in a way because enterprises are all very happy to put their data in the cloud. You know, they, they, they all use cloud services to store their data. But then, "Oh, train on this there? No, no. Can't do that." I think a lot of it comes honestly from kind of a misunderstanding about how this process works. Also, frankly speaking, I think it has to do with the fact that if you think about the model of just taking all your internal data and dumping it into a large language model, this is not tenable, right? We, you can't do this for a number of reasons, the most obvious one being that data has access rights, right? Not everyone gets access to all the data. And the default mode of language models is that if you, um...If you put- train on some data, you can probably get it back out of the system, uh, if you once to enough. And so this doesn't work with- with- the- the sort of- the access controls people have in traditional- traditional data. I think these are the kind of- the concerns. Now, there are ve- to be clear, there are very easy ways around this, right? So- so at- this is pro- probably why RAG-based systems are so- are so common here and so- and probably will remain, even with the advent of fine-tuning availability. They're gonna remain a- a useful paradigm. Uh, RAG is f- for those that maybe haven't heard the term, it's Retrieval Augmented Generation, it basically means that you just- you s- you- you go out and fetch the data you can access, that you have access rights to, uh, that is relevant to your question, you inject it all into the context of the model and then you answer the- the question based upon this data here too. So these RAG-based techniques are- are gonna remain popular, precisely because they kind of respect normal data access pr- uh, procedures, but I sort of feel like a lot of this hesitancy actually comes from a fundamental misunderstanding of how these models (laughs) are- are working. People think that if you type a- if you- if- if you- if you have ChatGPT answer a question about any of your data, that data is somehow being trained upon and merged into the model, uh, you know, w- whether it's an API call or whether it's a RAG-based call or anything else, and it's just- it's just not true. That's not how these models actually work. Um, these models are trained once on a very large collection of data, um, and if you use something like API access and things like that, your model is not going- you know, your data's not gonna be trained, uh, the model will not be retrained on that. And even if it was, that is not the same thing. The fact that a model can answer your question does not mean the model is training on it. This is honestly just very simple levels of misunderstanding, I think, that a lot of people have a very hard time getting over, and I still see these misconceptions when I talk with companies. So it's not like... We've done a- and sometimes done a very bad job of marketing because we just don't really always- people don't really understand at some level, sort of, this is not, in certain use cases, any riskier than just having your data in the cloud to begin with, which all of them typically do. They've all moved that way. So I- I think this'll just happen naturally with- with progression of time.

   14. HSHarry Stebbings26:21

       What do you think are the other biggest misconceptions that people have towards AGI? There's so many. I mean, I don't think people actually know. But what are others which really frustrate you?

   15. ZKZico Kolter26:28

       The thing that frustrates me most, honestly speaking, is the degree of certainty that some people have about whether we will definitely get there very, very soon or, even more on the flip side, that there's absolutely no way that we will ever achieve AGI with these current models because of X, Y, Z. Right? This- this- this does actually kind of- kind of start to- start to irk- irk me a little bit because I don't- I personally, you know (laughs) , even as a- a product of the AI winter skepticism, I see what's happening in these models and I am amazed by it, and the people that have been sort of ringing this bell for a while are saying, "Look, this is coming. This is- this..." You know, they've, in many cases, in my view, been proven right, and I've updated my sort of posterior beliefs based upon the evidence I've seen. And so what irks me the most about a lot of people's sort of philosophy of AGI is that, to a certain extent, how little it seems like observable evidence has changed their beliefs one iota. You know, they had certain beliefs about what it would take to get to general AI or maybe that AI was- was impossible by definition, or AGI was impossible de- by definition, and they kind of maintained those beliefs, in my view, uh, in the face of overwhelming evidence at least pointing to, uh, contrary outcomes.

7. 27:55 – 37:14

   The Danger of Misinformation & Lack of Trust in Objective Reality

   1. ZKZico Kolter27:55

   2. HSHarry Stebbings27:55

      Can I ask you, you know, a big concern for me actually is misinformation.

   3. ZKZico Kolter28:00

      Yes.

   4. HSHarry Stebbings28:00

      It's deep fakes-

   5. ZKZico Kolter28:02

      Mm-hmm.

   6. HSHarry Stebbings28:02

      It's the creation of malicious cyberattacks. I don't think we spend enough time talking about this. When you think about reality underlying practical dangers, what most concerns you, if those are some that concern me?

   7. ZKZico Kolter28:14

      I work v- sort of very heavily in the field of- of AI safety kind of broadly, and so I ha- I have a- a huge number of- of- of concerns here and sort of different tiers of concerns, I would say. And I can get into kind of-

   8. HSHarry Stebbings28:30

      What's- what's the highest?

   9. ZKZico Kolter28:31

      What's the highest concern for me right now? Um, before I answer that, actually, I'm actually gonna- gonna- gonna- gonna delve into your points since it's-

   10. HSHarry Stebbings28:37

       Sure.

   11. ZKZico Kolter28:37

       ... since it's on topic, uh, right now. So th- these are not my biggest concern, um, but let's talk about misinformation, deep fakes, fake, uh, just in general, kind of using these tools to proliferate different kinds of misinformation. Um, this is a massive concern, of course, and I am- I am deeply, deeply worried about this. But the net result of this outcome is not going to be that people start to believe everything that they see in misinformation. The real negative outcome is that people are not gonna believe anything that they see anymore. Right? So arguably, we are already well along this way, uh, or well along this path already where people basically don't believe anything that they read or that they see or anything else that doesn't already conform to their current beliefs. Um, it didn't even need AI to get there, but AI is absolutely an accelerant for this process. What I will say, though, uh, and this is a point that I- I- I do like to make about this, is that this is not a new phenomenon. This is actually the human condition as we were evolved- uh, as we evolved. Um, it is a relatively new phenomenon that we have sort of a record of objective fact in the world. I mean, things like video didn't exist, uh, more than- more than a hundred years ago, uh, a little bit more than that now, but (laughs) about a hundred years ago. Humans evolved at a time, uh, during a- an environment where all we could do was trust our close associates. That's how we believed things, and it's, in some ways...We see it as tragic right now that we are no l- maybe no longer in a world where we have a record of objective truth, and I- I- I am also troubled by this. But in another sense, maybe we're just getting back to kind of the world that we used to live in where all we could do was trust our close associates about what we believe about the world.

   12. HSHarry Stebbings30:29

       Does that not lead to a reduction in the advancement of human knowledge, though, if we only trust the- the- the people around us who we've known for years when we see them in person, not even when they send us something?

   13. ZKZico Kolter30:39

       Yeah. I mean, so obviously there are, there are, uh, you know, (laughs) massive negative externalities. But we did evolve knowledge at a time... very well at a time before video, right? Before videos existed, we still made scientific progress. So there will be groups that decide that certain bodies of scientific knowledge are- are valuable and they will advance those. Um, kind of even in light of large other portions of the population which have existed throughout all of history too that kind of don't value those scientific advances or think differently about the nature of scientific advances. We are already in this world, right? With th- this is the world we already live in. I think it is definitely an accelerant and a sh- and a shame that this sort of puts us m- more toward the camp failing to have an objective reality, but humans, and it's arguably our natural state, is to not agree on the nature of objective reality. This is a very... I mean, this sounds very negative. I don't wanna come across as too negative about this because I- I- I think that we will still absolutely make, make progress.

   14. HSHarry Stebbings31:42

       Being quite pos- I think you're being quite positive.

   15. ZKZico Kolter31:43

       Yeah.

   16. HSHarry Stebbings31:43

       But I think, to me, this is why you see the increasing value of existing media brands, because people place validity and trust in the content that they produce.

   17. ZKZico Kolter31:53

       They- yes.

   18. HSHarry Stebbings31:53

       So you will trust a New York Times tweet where it shows something, but, you know, some random account which has a picture, mm, don't know.

   19. ZKZico Kolter32:01

       Right. And, you know, you could argue there's gonna be a whole group of people that don't believe anything the New York- New York Times says. There is already that group, by the way. There's plenty of countries where they would not believe anything that's published in the New York Times, right? So we're- we're already there to a certain extent. And I think, yes, it will... we will need to rely arguably more on groups, but also sort of their associated, uh, belief structures about this. But this is human... this is h- the human condition to a certain extent. Not to get too philosophical here, but this is how we've always kind of c- had to be. Video is a short blip where we sort of think that there's some objective evidence for 100 years of our history, and that's gonna be now (laughs) that's no longer true pretty soon.

   20. HSHarry Stebbings32:40

       When you think about AI safety though, then, should the platforms themselves be the arbiters of justice of what's right and what's not right? You know, Twitter, Facebook, Reddit. And are they the ones to say, "No, this is not allowed content"?

   21. ZKZico Kolter32:53

       There are some things I believe that should not be shared on social media. And by the way, everyone else agrees with this too, right? There's obviously content that is outright considered illegal that you cannot post to social media. Everyone agrees on this. Um, everyone also agrees that... well, not everyone, but a lot of people also agree that in general there should be a share... the- you know, there sh- there should not be a requirement to conform to certain ide- ideologies and opinions if you want to express yourself on social media. And so there's obviously a middle ground. You have to... and you have to toe the line here and you have to adapt to the reality of the situation on the ground and kind of go from there. I think to- to... in- in- in many ways, and this is maybe what I was pointing out before, is that AI, when it comes to things like misinformation, it is not... it did not invent misinformation, AI. It can argue... there was misinformation and propaganda and this stuff long before there was AI. You can argue it's an accelerant to it, like it's an accelerant for everything, um, that- that... for a lot of things that we have, right? Um, but it did not invent these things. And my hope at least is that a lot of our existing social and economic and governmental structures can continue to provide the same guidance they have provided for our current kind of take on moderation and things like this, even in an AI world.

   22. HSHarry Stebbings34:13

       How do you respond as a government organization today where you are supposed to set regulation, supposed to set policy, and you are dealing with RAG, FLOPs, architect... transformer architecture, all of these bluntly technical words and architectural information that they have no idea what it means?

   23. ZKZico Kolter34:35

       Yeah.

   24. HSHarry Stebbings34:35

       My question is, are governments structurally set up to regulate AI effectively?

   25. ZKZico Kolter34:41

       On this point, I will... I hold two beliefs at the same time, largely. And I- I- I- I'll- I'll sort of say this straightforward. I also, I also... to be very clear, um, regulation is something that is not sort of my direct wheelhouse here, so it's not what I directly work on. Um, and I do also wanna get back to some of the AI safety points. I never described what I think is my biggest fear. But, um, I- I think that like a lot of new technologies, there's absolutely a role for regulation and for governments to provide frameworks for ensuring that new technologies do benefit the world. Right? This is- this is why we form governments to a certain extent. Um... to your more... and- and in that umbrella, I believe there is absolutely the need to better understand how and where we can regulate AI as a technology. I also, though, think that maybe to your point of the examples you were given, a lot of the details about how those regulations sometimes evolve can be a bit misguided or miss the point or somehow just... when I read them, the- basically, they're gonna become dated in- in a matter of months because they're- they're dealing with things and they're- they're approaching the problem from a way that doesn't really match the- the nature of how these systems are really developed in practice. I think that it is much easier, we have a much better handle...... on regulating the downstream uses of AI. Like, when it comes to misinformation, we already have laws, uh, that, that, that deal with sort of libel and things like this. Um, in many cases, because AI is acting as an accelerator, there are situations in which I think that existing laws, maybe with a slight tweaking to deal with the velocity and the volume the AI's capable of producing, can suffice to regulate, um, many of what we consider the harmful use cases of AI. But at the same time, I don't think that's efficient either. Right? I, I, I think that, of course there are going to be ways in which technologies, especially technologies powerful as this one, we, we have to think about ways in which we can, we can regulate it. And, um, I don't know what that looks like. I think it's extremely hard because it changes incredibly rapidly.

8. 37:14 – 44:45

   The Concerns and Hierarchy of Safety in AI

   1. ZKZico Kolter37:14

   2. HSHarry Stebbings37:14

      Speaking of kind of the safekeeping models, I, I ... terrible MC'ing that I am, kind of jumping between so many different topics. But I, I do wanna discuss the, the hierarchy of safety concerns-

   3. ZKZico Kolter37:23

      Sure, sure. Yeah, yeah.

   4. HSHarry Stebbings37:24

      ... you have. Because I mentioned mine-

   5. ZKZico Kolter37:26

      Yes.

   6. HSHarry Stebbings37:27

      How would you categorize yours?

   7. ZKZico Kolter37:29

      Sure. So, the biggest concern I have right now in AI safety, which I think leads to a lot of negative downstream effects, is that right now the model- the AI models that we have, for lack of a better phrasing, are not able to reliably follow specifications. And what I mean by this is that these models are tuned to follow instructions. You can give them some instructions as a developer, but then if a user types something, they can follow those instructions instead. Right? Uh, we've all seen this at- this goes by a lot of names. Prompt injection, um, sometimes depending on what you're getting out, this is called things like jailbreaking and things like this. Um, but the, the, the core point is we have a very hard time enforcing rules about what these models can produce. So oftentimes we say, you know, models are trained right now just to, to not do things. I use a common example of things like hot wiring, hot wiring a car in a lot of demos I give, right? So models are trained. If you asked a model, you know, most commercial models, "How do I hotwire a car?" They'll say, "I can't do this." It's very easy through a number of means to basically manipulate these models and convince them that they really should tell you how to hotwire a car because, you know, you're, you're, you're in desperate need of your- you've, you've locked yourself out and it's an emergency and if you don't get in your car

   8. NANarrator38:47

      (laughs)

   9. ZKZico Kolter38:48

      This is very different from how we're used to programs acting, right? We are used to computer programs doing what they're told, nothing more and nothing less. And these models don't always do what they're told. Sometimes do too much of what they're told and do way more than what they're told also some other times. And so we are very unused to thinking about these models, uh, or thinking about computer software rather, uh, like these models. And what that means is- and to be honest, I'm not- I don't really care if c- models tell me how to hotwire a car. I, I, I just don't. It doesn't matter, right? There's instructions on the internet on how to hotwire a car. They're not really revealing anything that sensitive. Um, however, as we start to integrate these models into larger systems, as we start to have agents that go out and do things, that parse the internet and go out and do things, if all of a sudden they are running their model parsing untrusted third party data, that data can essentially gain control of those models to a certain extent. Right? And this is from a sort of cybersecurity standpoint, not the normal cybersecurity, but sort of from a concept of cybersecurity, this is sort of like these models have a buffer overflow in all of them, um, that we know about and most importantly that we don't know how to patch and fix. Uh, we don't know how to fix this yet with models. And we're making- to be clear, I think we can make a lot of progress. Uh, we are making progress, but this is a real concern about models right now. And I think the, the, the negative effects in a domain like a chatbot are maybe not that concerning, but as you start having much more complex LLM systems, this starts becoming much more concerning. What I will also say is that, and this is maybe the, uh, the, the reason why I place this concern first, is that I think this fact is something we need to figure out or kind of all the other downstream concerns that we have about these models get much, much worse. So let me just take an example. O- oftentimes, um, uh, I'm, I'm touching a lot of points here, I know, too, but I, I think I'll wrap it up soon. Oftentimes people talk about risks like biorisks or cyberattack risks and stuff like this. I'm actually, to your point, I'm s- very concerned about cyber risks in particular. I think this is essentially already solved in many cases by these models. They can already solve and analyze code to find vulnerabilities. This is extremely concerning. The way we think about fixing this normally is, you know, we would have certain models, models that we release. We would say, "You know, don't, don't, uh, don't use your model ability that you have in- inside of you to sort of, you know, create obvious cyberattacks against, against certain infrastructure and stuff like this." Right? "Don't, don't do that." But we can't make them follow that instruction, right? Someone, someone either with access to a model itself certainly, but even with access sometimes to a, to a closed source model just can have the ability to jailbreak these things and oftentimes get access to these things. And now we- to be very clear, we are making immense progress in, I think, in solving this problem of sort of preventing jailbreaks, kind of, um, avoiding- making sure models follow a spec. But until we can solve this problem, it's very hard to say, you know, you know, all the other dangerous downstream e- effects that AI could c- or dangerous capabilities that AI could, could sort of demonstrate become much, much more concerning. And so this is kind of a multiplier effect on everything else bad these new models can do, which is why I'm so concerned about it right now.

   10. HSHarry Stebbings42:26

       And the multiplier effect is the subsequent elements that become heightened because of this-

   11. ZKZico Kolter42:30

       Exactly.

   12. HSHarry Stebbings42:31

       ... are-... it's like terrorist attacks-

   13. ZKZico Kolter42:34

       Yeah. S-

   14. HSHarry Stebbings42:35

       ... or fraud cases, or...

   15. ZKZico Kolter42:36

       (laughs) So, so here, I think there's a lot... Yeah. So, so then, then the next... So, so right, so that's sort of the- the- a good lead in, right? Because if jail breaks and sort of manipulation of models is the attack vector, what is the payoff? What is what we- what- what are the things we can do? And here, what we're trying to do really is we're trying to assess the core harmful capabilities of models. And people have thought a lot about this, right? People think about things like chemical, uh, chemical- creating chemical weapons, creating biological weapons, creating cyberattacks. Personally, I think cyberattacks are a much more clear and present threat than, for example, biothreats and things like this. Uh, at the same time, I don't wanna dismiss any of these concerns, right? I think people have looked at this much, much more than myself and are very concerned about these things. So I, so I, so I want to sort of, um, I want to treat this with the- with the, uh, respect that honestly it deserves 'cause these are sort of massive problems. There are a lot of potential harms of AI models. Some are associated primarily with scale and things like this, like the misinformation you mentioned. But some are just there are capabilities that we think these models might enable where they would lower the bar so much for some bad things, like say creating a zero-day exploit that takes down software over- over half the world.

   16. HSHarry Stebbings44:04

       Wow.

   17. ZKZico Kolter44:04

       The concern is that, not that they can do this sort of autonomously maybe initially, but that they can lower the bar so far in the skill level required to create these things that effectively it puts them in the hands of a huge number of bad actors. Uh, and the same is true for things like- like biological risk, or- or chemical risk, or- or other things like this. And these concerns have to be taken seriously, and they have to be things that we really do consider as genuine possibilities if we start putting into everyone's hand the ability to create really sort of, um, harmful artifacts.

9. 44:45 – 59:10

   The Considerations of Releasing Open-Source Models

   1. ZKZico Kolter44:46

   2. HSHarry Stebbings44:46

      Alex Wang at Scale-

   3. ZKZico Kolter44:47

      Mm-hmm.

   4. HSHarry Stebbings44:47

      ... said a brilliant line on the show. He said that, um, essentially we have a technology now that is more potentially dangerous and impactful than nuclear weapons. I mean, kind of psh, talk about a soundbite.

   5. ZKZico Kolter45:00

      Yeah.

   6. HSHarry Stebbings45:00

      My question to you is, if that is the case, or even partially the case, or even potentially the case-

   7. ZKZico Kolter45:04

      Mm-hmm.

   8. HSHarry Stebbings45:05

      ... is there any world in which it should be open?

   9. ZKZico Kolter45:08

      Two issues there. One, is AI as dangerous as nuclear weapons? And what does it imply about sort of the open release of certain models? Um, so I'll make two points on this. Um, I think the nuclear weapon analogy is actually not a great one because nuclear weapons have one purpose, which is to- to destroy things. Maybe a better analogy is sort of nuclear technology period, because it has the ability to create nuclear weapons but it also has the ability to do things like provide power, uh, non-non-, uh, CO2 emitting power to potentially a huge number of people, right? Um, and- and, you know, a lot of people are currently making a bet on- on nuclear as the- as the way we create carbon-free, carbon-free energy. But I think the analogy to nuclear weapons in particular is often overstated precisely because AI has many good uses. Nuclear weapons, arguably they- they do one thing, and it's- it's not considered a good use, right? This is- this is- so this is a very different kind of technology there. Um, but let me get to your second point now, which is the- the sort of the open model debate, which is also one that frequently is played out in kind of discussions on AI safety. I'm a fan of open source models, uh, in the- in- in- in the- in a general sense, so- so I wanna start by saying that, because honestly speaking open source release of models, and I- I really should just say open weight because, um, oftentimes these are not actually open source the traditional way. They're actually much more like closed source executables. They just- you can run them on your own, on your own computer. Open weight models have advanced my ability to study these systems. They've been the primary tool by which we- we conduct research in academia and beyond, and they're becoming, I would argue, a critical part of the overall ecosystem of AI right now, number one. Number two, if you look at the current best models there are right now, so things like GPT-4, Claude 3.5, um, Gemini, things like this, I would not currently be all that nervous about having an open source model that was as capable of ... these in terms of the catastrophic effects of it. Uh, 'cause these models actually aren't, by themselves, they're still not all that... They... We have a good handle on them, right? We- we sort of know what they're capable of. Arguably, we're already here because Llama-3-405 billion is pretty close. I don't think it's quite at that level yet, but it's- it's getting there. Um, and, you know, that this- this release has not yet caused some- some... I mean, I should probably not speak too soon here, but arguably has not yet caused some catastrophic event. Or it has not yet, and- and, you know, arguably it won't, because the reality is these- these models, they- they still have a ways to go. However, and this is the big... And so- and so right now to a certain extent I think things are- are okay with open source, uh, open weight release of the models. However, I think there will come a time when a certain capability, a certain ability of these models reaches the point that should give us pause when it comes to just turning these things over to whoever and whe- however they want to use them. And I do think this. I think there- there- there are certain levels or capabilities that you could s- that are within kind of eyesight of our current development, that if I was sort of to ask the question, you know, should- should we give this to everyone not just to use but to use and tune and specialize however they want...And I would just sort of say, "Ugh. This is, this... I, I, I don't..." I think there will be a point where I get uncomfortable with that, honest-

   10. HSHarry Stebbings48:44

       What is that point?

   11. ZKZico Kolter48:45

       ... to speak. So if you think about a model that really could anal- I mean, just as a simple example, a model that really could analyze any code base or any, even any binary executable or website or JavaScript, something like this, and immediately find a vulnerability that it could exploit to, to take down, you know, a large portion of the internet or a large portion of software. If this was demonstrated as a capability of a model, I would have a very hard time saying, "Of course, yeah, let's just release it." You know, "There's no problem 'cause we'll use it for good. There's dual use, so we'll use it for good purposes." Uh, I mean, we all know that patching software is much harder than finding exploits in sof- or patching all software is much harder than finding exploits in software. Yes, there's dual use, so you can use it to secure software better, but that takes time. It's hard. I don't think we should just immediately snap to release a model that could find a vulnerability in literally any code that's out there in the world. I wouldn't want that to be released open weight for anyone to use. Now I take some, uh, solace in the current situation we find ourselves in, which basically in the current situation we find ourselves in, um, at least for now, th- there is a constant stream of closed models that are released some time before an equivalently s- capable opens- open weight model. Right? And I think this is actually a very good thing because my hope would be that... And, and we've sort of found ourselves here by accident. It didn't have to be like this. I know some companies are pushing to open source more powerful models than we have ever had before right at the outset, um, and that, that makes me a little nervous. But, um, right now we're not in that world. We're in a world where the most capable models, the first releases of them of a certain capability typically comes from closed source models. I think this is a good thing. I think it gives us some time to essentially come to terms and understand the capabilities of these models in a more controlled environment such that we can reach a level of comfort to say, maybe not full comfort but at least a level of comfort to say, "Yes, it's probably okay if we release this, uh, similar model open source." And I hope, my, my sincere hope would be that if one of these models does really demonstrate the ability to create an exploit for any executable code or compiled code or anything else instantly, w- and we see that in the closed source model first, we would think a little bit about whether we really want to release this model, uh, uh, uh, an equivalent model, uh, open weight and just for anyone to use.

   12. HSHarry Stebbings51:13

       Is there anything I have not asked on AI safety that I should've asked?

   13. ZKZico Kolter51:17

       Uh, one of the biggest questions I am asked about AI safety, because wh- what I laid out to use so far was honestly a, a pretty practical set of recommendations and a pretty pragmatic view on the field. Right? Um, I mean, I'm talking about, you know, preventing jailbreaks. I'm talking about making progress when it comes to securing these models. I'm talking about the interplay between the current release of open source models and closed source models. I do think that while it is not my area, the more far-fetched scenarios about sort of agentic AGI systems that, that start sort of intentionally acting harmful against humans, this is often... These, these... So, so I'm thinking about, you know, the, the, the, the rogue AI that decides it wants to wipe out humanity and goes about planning on, on, on how to do this. What seemed to me, and I'll be honest here, far-flung kind of sci-fi-ish scenarios here, these are often the debates we have when it comes to AI safety. And I wanna say sort of two things about this. The first is that I think the vast majority of AI safety should not be about these topics. The vast majority should be about quite practical concerns we have on making systems safer like the kind that I've talked with you about so far. There are already massive safety considerations and risks that are present in current systems and would certainly be present even in slightly more capable systems, kind of irrespective and regardless of the, the, the, the time frames associated with AGI and certainly the time frames associated with, you know, uh, rogue intelligent AI systems. I also don't want to dismiss this entirely, and the way I would put it is I am glad people are thinking about these problems. I'm glad people are thinking about kind of the, um, the effects that, the, the capabilities and even the f- what I consider far-flung scenarios, um, they are good things to think about as, by the way, (coughs) are much more immediate harms of AI systems like misinformation, like misuse of these things.

   14. HSHarry Stebbings53:23

       What far-flung scenario do you think is most good to think about? Because most people just go, "Robots, killing jobs, killing humans ultimately post-killing our jobs."

   15. ZKZico Kolter53:33

       (laughs) I think killing jobs is much more, much more immediate of a concern than killing humans. An example I often use here to kind of try to bring a little bit of these two sides, the, the, the, the sort of AI taking over the world, uh, killing us all and kind of the, the more skeptical-minded academic folks, we'll say, I see a path right now to a world in which, you know, in a few years from now we start integrating AI models into more and more of our software. We start building it up more and more. We sort of make these things a little bit more autonomous in their actions. We start just naturally, because software does everything for us, we start naturally kind of infusing this into all software we have, including software that handles things like critical infrastructure, stuff that controls the power grid, things like this. Right? And now all of a sudden you have these agents that are sort of, you know, taking an active, playing an active role in doing things like controlling power grids. This leads to the possibility of, even in my view, sort of massive correlated failures that could do things like bring down power, electricity in a way that we can't restore it easily in, for a large portion of the country.Now, in this world, um, and I, I think it's, uh, honestly not, again, if we go down the wrong path, this is definitely not that impossible to imagine. Now, in this world where the power has been shut off, you know, we can debate and two sides can debate about whether this was, you know, a bug in the system and we should never have installed LLMs here in the first place. Or we could debate whether this was actually the rogue AI taking over and deciding to shut off the power so, so it could, so it could, could have killed all humanity. But who cares? The power is still off. This is still a catastrophic event for the country, and so we have to have a plan for how to (laughs) how to sort of think about events like this happening. This is an example I come to often. To a certain extent, it doesn't matter whether the AI is intentionally doing something in an evil fashion while deceiving humans or whether this is a bug and a flaw in the system. The end effects are the same in some cases, and so we need to desperately take kind of put in structures in place that prevent these things from, from being possible.

   16. HSHarry Stebbings55:50

       Or, we just appreciate that they had CrowdStrike. (laughs)

   17. ZKZico Kolter55:54

       Well, exactly. So yeah, we are all very familiar right now with the downsides of correlated failure, right? Um, and imagine if that was also true of all the SCADA systems that were operating power, the, the, the power grid, uh, right now, which, you know, not impossible to believe. And the problem is that these systems because we don't understand w- w- w- really, I mean, and, and we don't, we don't understand them, right? We, we do not understand how these things work internally, the possible correlated failures, the possible attack vectors, all those sorts of things. We don't understand it. And because of this, we need to think very carefully about how we deploy these systems, how we do consider safety concerns, especially when it comes to things like critical infrastructure that I think are extremely, um, just pressing concerns. And, and, and yes, things like bio-rust. Again, I, I, I, these I work much less on, but these are potentially pressing concerns and you don't have to believe in super intelligent evil robots in order to have these as pressing concerns. AI safety is a concern right now, and we all need to come to grips with the, the, the fact that it's a concern right now and start solving the problems right now.

   18. HSHarry Stebbings57:05

       The astonishing thing is, I don't know if you remember banking, but password tests were like, "My voice is my password."

   19. ZKZico Kolter57:11

       Yep.

   20. HSHarry Stebbings57:12

       Like, I really hope yours isn't right now-

   21. ZKZico Kolter57:14

       (laughs)

   22. HSHarry Stebbings57:14

       ... because ElevenLabs is doing pretty great things with my voice. (laughs)

   23. ZKZico Kolter57:17

       Yeah, it's, it's, it's really wild, the, and, and I think this does, our current sys- our, the current things that we have already upend a massive amount of sort of the systems we've built in place, and they will continue to be upended more and more from evolving AI technology, and the- these are real concerns that we have to, that we have to come to terms with.

   24. HSHarry Stebbings57:37

       Final one for you before we do a quick fire. Just are you, are you optimistic about this future we're moving into, and do you want your children to speak more to LLMs and models than they do to humans?

   25. ZKZico Kolter57:49

       I, I would classify myself as an optimist when it comes to, when it comes to AI. I already enjoy these tools, that I'm excited about the potential things we can do with these tools. Yes, even up to AGI, and I, I, I use the word tool here not pejoratively. Um, all, you, you know, they, they, they... Hopefully, AGI is a tool, right? Hopefully, AGI is a system that we still deploy to our ends to achieve our ends. And I am, and I, and I, I'm h- I can't help but be excited about these things. This is the culmination of a lot of the work that, that we in the field have been doing, and it's, it's kind of coming to fruition in a lot of ways, in a way that is directly sort of beneficial for a lot of the things that I do. So, I want to have these tools, and maybe this gets to the c- clear point here. I want to develop and improve safety of these tools because I want to use them. Um, it's not that we have some moral imperative that we have to develop these tools. I mean, maybe there are, or we have to develop AI and AGI. Maybe that's true, um, but I, that's not what motivates me to develop them, right? That I, I want to develop them because I want to use them and I want to be able to have them. To reach that point, they have to be safe, right? It's, it's a condition, it's a necessary condition, and that's why I work on building, uh, and improving the safety of AI systems.

10. 59:10 – 1:03:45

    Quick-Fire Round

    1. ZKZico Kolter59:10

    2. HSHarry Stebbings59:10

       Zeke, I could talk to you all day. I do wanna move into a quick fire.

    3. ZKZico Kolter59:13

       Sure, absolutely.

    4. HSHarry Stebbings59:13

       So I say a short statement, you give me your immediate thoughts. Does that sound okay?

    5. ZKZico Kolter59:16

       Okay. How long do I have for each one?

    6. HSHarry Stebbings59:19

       60 seconds.

    7. ZKZico Kolter59:19

       60 seconds, okay.

    8. HSHarry Stebbings59:20

       What did you believe about models that you later changed your mind on?

    9. ZKZico Kolter59:25

       I, for a lot of my career, was thinking that model architectures really mattered, and by having clever, complex architectures and sub-modules inside architectures, you would have... That was the route to sort of better AI systems. For the most part, I don't believe this as much anymore. I think basically models don't matter, architectures don't matter, and that, that applies to transformers too. I think, you know, anything could kind of work in their stead if we just spend enough time on it. Um, and so I think that to a large extent, we're kind of post-architecture in a lot of our AI work.

    10. HSHarry Stebbings59:55

        God, what a, what a breakthrough snap that is.

    11. ZKZico Kolter59:58

        (laughs)

    12. HSHarry Stebbings59:58

        Uh, what did you believe about data that you later changed your mind on?

    13. ZKZico Kolter1:00:02

        Um, kind of on the contrary. I thought that data was sort of, you know, data had to be highly curated to be valuable, and the value in data came essentially from very manual labeling of this data and, and human-intensive curation. The big amazing insight of current AI is that we can, to a large extent, just suck up data that exists out there on, on the internet, uh, train models based upon that, and get amazing things to come out of it. Not to say there's no value in curation. Of course there's elements of this, but to a very large extent, this is kind of on the old paradigm of unsupervised learning, and, and that's, that's absolutely incredible.

    14. HSHarry Stebbings1:00:42

        How does joining the OpenAI board look? Does Sam just call you up and go, "Hey..."Love the whiteboard.

    15. ZKZico Kolter1:00:47

        (laughs)

    16. HSHarry Stebbings1:00:47

        Fancy, fancy coming on our board?

    17. ZKZico Kolter1:00:50

        Um, I got, actually, the, the, the day before I started as department head, I got an email from, from, from Brett, the chair of the board, just saying, "Hey, do you wanna talk about maybe joining the OpenAI board?" Uh, so I figured, you know, I was already embarking on one massive career change, so why not, why not double down and just do, you know, do two at the same time? But basically, I, I, I, I started having some conversations with, with, with, with him and the rest of the board. It got really ex- you know, I, I got very excited about the potential, sort of provide my perspectives on AI and AI safety to the board and, and, and things went from there.

    18. HSHarry Stebbings1:01:20

        What are the, like, roles and responsibilities? Do they set them out, like four board meetings a year and, you know, a biscuit and a coffee in between?

    19. ZKZico Kolter1:01:27

        (laughs) There are four board meetings a year, yes. But I think, um, uh, I'm, I'm being brought on the board as, as an expert in AI and AI safety. And I am excited to provide my perspective and expertise specifically on AI to, to the rest of the board.

    20. HSHarry Stebbings1:01:43

        Do you believe the statement that China's two years behind the US in terms of AI progression?

    21. ZKZico Kolter1:01:47

        There is absolutely some element here of kind of a, a, a race, um, between different countries for AI dominance. Uh, but what I will s- I actually will take a different stance on this and say that I think there are certain things, like, for example, AI safety, where we very much need to work as a world to help set standards and help better the future of everyone here. Because yes, certain things can be done by countries. Capabilities can, can maybe advance more by countries. Safety is something that's inherently global, and so we need to work together to build safe AI systems.

    22. HSHarry Stebbings1:02:19

        What is the most common question you're asked that you don't think you should be asked?

    23. ZKZico Kolter1:02:23

        Um, the most common one has to do, I would say, with things like putting an overemphasis on the archite- Questions that put an overemphasis on the architectures involved in, in AI systems. So, you know, this notion that somehow the transformer was the thing that, that, uh, that, that made all AI possible. I'm often asked questions like, you know, "What comes after the transformer?" and things like that. And, and the reality is, as I said before and which is probably, I know makes for a good soundbite. Archite- We, we, we are arguably in a post-architecture phase. They don't, they don't really matter. We could do what we're currently doing with a whole lot of architectures right now. And I hope that I can steer the conversation to more of one where we consider these models not in terms of their particular structure, because it's somewhat irrelevant when it comes to capabilities, and we think about these models more in terms of the data that goes into them and the capabilities they produce downstream.

    24. HSHarry Stebbings1:03:13

        Zeke, a- as you can tell from my meandering conversation, I, I've so enjoyed this.

    25. ZKZico Kolter1:03:17

        (laughs)

    26. HSHarry Stebbings1:03:17

        Uh, I, I'm so glad you didn't have too much time with the schedule. Otherwise, I would have been screwed.

    27. ZKZico Kolter1:03:22

        (laughs)

    28. HSHarry Stebbings1:03:22

        Uh, but thank you so much for being so brilliant.

    29. ZKZico Kolter1:03:24

        Great. Well, thank you very much for inviting me. And yeah, hope- I, I, I don't look forward to, to you having to edit all this together to sort of form my co- my, my rambling thoughts into something that sounds coherent. But, you know, I'm sure you'll do a great job with this.

Episode duration: 1:03:45