Why Scale Will Not Solve AGI | Vishal Misra - The a16z Show

Vishal Misra 0:00 Anthropic makes great products. Claude Code is fantastic, CoWork is fantastic, but they are grains of silicon doing matrix multiplication. They don't have consciousness. They don't have an inner monologue. You take an LLM and train it on pre-1916 or 1911 physics and see if it can come up with the theory of relativity. If it does, then we have AGI.

Erik Torenberg 0:21 Just today, by the way-

Vishal Misra 0:22 Yeah

Erik Torenberg 0:22 ... Dario allegedly said that you can't rule out that they're conscious.

Vishal Misra 0:27 You can rule out they're conscious. [both laughing] Come on. To get to what is called AGI, I think there are two things that need to happen. One is...

Erik Torenberg 0:39 Vishal, it's great to have you in again.

Vishal Misra 0:41 Great to be back.

Erik Torenberg 0:42 This is one of my favorite topics, which is, um, how do LLMs actually work?

Vishal Misra 0:47 Mm-hmm.

Erik Torenberg 0:47 And I think that, uh, you, in, in my opinion, you've done kind of the best work on this, modeling it out.

Vishal Misra 0:52 Thank you.

Erik Torenberg 0:53 For those that did not see the original, um, one, maybe it's probably worth doing just a quick background on kind of what led you to this point, and then we'll just go into the current work that you've been doing.

Vishal Misra 1:04 Five years ago, when GPT-3 was first released-

Erik Torenberg 1:07 Yeah

Vishal Misra 1:08 ... uh, I got early access to it, and I started playing with it, and I was trying to solve a problem related to querying a cricket database.

Erik Torenberg 1:19 Yeah.

Vishal Misra 1:20 And I got GPT-3 to do in-context learning, few-shot learning, and, you know, it was kind of the first, at least, uh, to, to me, it was the first known, uh, implementation of RAG, Retrieval-Augmented Generation, which I used to solve this problem of, uh, querying, getting GPT-3 to translate natural language into something that could be used to query a database that GPT-3 had no idea about. I had no access to GPT-3's internals, but I was still able to use it to solve that problem. So it, it, it worked beautifully. Uh, we, we deployed, uh, this, uh, in production at ESPN in September '21, but-

Erik Torenberg 2:01 Wow. Wow, you, you did the first implementation of RAG in 2021?

Vishal Misra 2:07 No, no, no. In 2020.

Erik Torenberg 2:08 2020.

Vishal Misra 2:09 2020, I got it working, and by the time you talk to all the lawyers at ESPN and, you know, productionize it, it took, it took a while.

Erik Torenberg 2:16 Wow.

Vishal Misra 2:16 But October 2020, we had... Well, I had this-

Erik Torenberg 2:18 Yeah

Vishal Misra 2:19 ... architecture working. But after I got it to work, I was amazed that it worked. I wanted to understand how it worked.

Erik Torenberg 2:26 Yeah.

Vishal Misra 2:27 And I looked at, you know, the Attention Is All You Need paper and all the other sort of deep learning architecture papers, and I couldn't understand why it worked.

Erik Torenberg 2:37 Yeah.

Vishal Misra 2:38 So then I started getting sort of, uh, deep into building a mathematical model.

Erik Torenberg 2:43 Yeah. And now you've published a, a series of papers.

Vishal Misra 2:46 Yeah.

Erik Torenberg 2:46 The first one that I read was the one where you had kind of your matrix-

Vishal Misra 2:50 Yeah. Right

Erik Torenberg 2:50 ... kind of abstraction. So maybe we'll talk about that, and then we'll talk about the more recent-

Vishal Misra 2:54 Yeah

Erik Torenberg 2:54 ... work. So perhaps we'll just start with the first one, which is

Erik Torenberg 2:58 you were trying to, you, you were trying to describe... You were trying to come up with a mathematical model-

Vishal Misra 3:02 Mm-hmm

Erik Torenberg 3:03 ... of how LLM works.

Vishal Misra 3:04 Yeah.

Erik Torenberg 3:04 And you had, which was very helpful to me-

Vishal Misra 3:06 Mm-hmm

Erik Torenberg 3:06 ... which was, um... And at the time you were actually trying to, like, figure out how in-context learning was working.

Vishal Misra 3:11 Yes. Yeah.

Erik Torenberg 3:11 And you came up with an abstraction for LLMs, which is basically this very, very large matrix, and you used that to describe. So maybe you can kind of walk through that work very quickly.

Vishal Misra 3:18 Sure, yeah. So, so what you do is you, you imagine this huge, gigantic matrix where every row of the matrix corresponds to a prompt.

Erik Torenberg 3:29 Yeah.

Vishal Misra 3:30 And the way these LLMs work is, given a prompt, they construct a distribution of probabilities of the next token. Next token is next word. So every LLM has a vocabulary. You know, GPT and its variants have a vocabulary of about 50,000 tokens.

Erik Torenberg 3:46 Yeah.

Vishal Misra 3:47 So given a prompt, it'll come up with a distribution of what the next token should be, and then all these models sample from that distribution.

Erik Torenberg 3:55 Yeah. So that's the posterior distribution.

Vishal Misra 3:56 That's the posterior distribution.

Erik Torenberg 3:57 Right.

Vishal Misra 3:57 Right? That, that's how LLMs work. And so the idea of this matric is, matrix is for every possible combination of tokens, which is a prompt, there's a row.

Erik Torenberg 4:06 Yeah.

Vishal Misra 4:06 And the columns are a distribution over the vocabulary.

Erik Torenberg 4:10 Yeah.

Vishal Misra 4:10 So if you have, like, a vocabulary of 50,000 possible tokens, it's a distribution over those 50,000 tokens.

Erik Torenberg 4:16 And by distribution, it's just the probability-

Vishal Misra 4:17 Just the probability. Sorry, yeah.

Erik Torenberg 4:18 Yeah.

Vishal Misra 4:18 Just the probability that, uh, the next token should be this-

Erik Torenberg 4:22 Yeah

Vishal Misra 4:22 ... versus that.

Erik Torenberg 4:23 Yeah.

Vishal Misra 4:24 Uh, so that, that's sort of the idea. And, and, and when you start viewing it that way, it, it makes things at least, uh, clearer to, uh, people like me who want to model it, uh, what, what's happening. So c-concretely, let's say you have an example that, uh, let's say your prompt is just one word, protein.

Erik Torenberg 4:45 Yeah.

Vishal Misra 4:46 So if you look at the distribution of the next word, next token after that, uh, most of the, uh, probabilities would be zero, but you'd have non-zero, non-trivial probabilities on, let's say, two words. One is synthesis, the other is shake.

Erik Torenberg 5:03 Yeah.

Vishal Misra 5:03 Right? And now the LLM is going to sample synthesis, uh, or sample, uh, this next token, and might pick synthesis or shake.

Erik Torenberg 5:12 Yeah.

Vishal Misra 5:12 Or you as a human will give the prompt protein shake-

Erik Torenberg 5:17 Yeah

Vishal Misra 5:17 ... or protein synthesis. Now, depending on whether you pick synthesis or shake, the next, the, that row looks very different, right? If you pick protein synthesis, the terms that would have a high probability would be all concerned with biology, right? But if you pick pro-protein shake, it'll all be about gyms and exercise and all, you know, bodybuilding stuff. So that synthesis or shake completely changes what comes next.

Erik Torenberg 5:43 Yeah.

Vishal Misra 5:44 So this is an example of, uh, you can say Bayesian updating. You start with protein. You have a prior that after protein, this is going to happen. As soon as you get new evidence, then the next term is synthesis or shake. You completely updateThe distribution. So now you can imagine that the whole, the, the, the entirety of LLMs is this giant matrix-

Erik Torenberg 6:10 Yeah

Vishal Misra 6:11 ... where you have every row, protein, protein shake, protein synthesis, the cat sat on the-

Erik Torenberg 6:16 Yeah

Vishal Misra 6:16 ... you know, Humpty Dumpty, blah, blah, blah. Now, given, uh, the vocabulary of, uh, these LLM, let's say 50,000, and the context window. So GPD, for instance, uh, ChatGPT, the first version had a context window of 8,000 tokens.

Erik Torenberg 6:34 Yeah.

Vishal Misra 6:35 If you look at all possible combinations of 8,000 tokens and 50,000, uh, uh, vocabulary, the number of rows in this matrix is more than the number of electrons across all galaxies, right? [chuckles]

Erik Torenberg 6:50 That's a lot.

Vishal Misra 6:50 So, so there's no way that these LLMs can represent it exactly. Now, fortunately, this matrix is very sparse. Why? Because, you know, an arbitrary combination of these tokens is gibberish.

Erik Torenberg 7:02 Yeah.

Vishal Misra 7:02 We're not never gonna use that in natural, i-in real life.

Erik Torenberg 7:05 Yeah.

Vishal Misra 7:06 Also, the columns are also mainly zero.

Erik Torenberg 7:11 Yeah.

Vishal Misra 7:11 Right? If you have protein, then you won't have lots of, you know, you won't have arbitrary numbers or arbitrary words after that. It's very sparse, both in rows and in columns. So i-i-in kind of an abstract way, what all these LLMs are doing is coming, coming up with a compressed representation of this matrix.

Erik Torenberg 7:33 Right.

Vishal Misra 7:34 And when you give a prompt, they try to approximate what the true distribution should have been-

Erik Torenberg 7:40 Yeah

Vishal Misra 7:41 ... and try to generate it. That, that's what, uh, in my mind at least, it boils down to.

Erik Torenberg 7:45 And j-just from my understanding, so if you have a row of, uh, protein and then you have one with protein shake-

Vishal Misra 7:56 Uh-huh

Erik Torenberg 7:56 ... is protein shake a subset of protein or is it different?

Vishal Misra 8:00 It's different. It's a continuation from.

Erik Torenberg 8:02 I see.

Vishal Misra 8:03 Yeah.

Erik Torenberg 8:04 Right. No, but I'm just saying like the actual pr- the actual posterior distribution, is that a subset?

Vishal Misra 8:08 You, you can say it's a subset, right? Uh, if you have protein, then protein shake and protein synthesis are all continuations from protein.

Erik Torenberg 8:15 Yeah.

Vishal Misra 8:15 So both synthesis and shake have non-zero probabilities.

Erik Torenberg 8:19 Right.

Vishal Misra 8:19 So you can, yeah, you can think of it as somewhat a

Vishal Misra 8:24 subset, right.

Erik Torenberg 8:25 You, you know, you use this approach to describe how in-context learning works, and so maybe first describe what in-context learning is-

Vishal Misra 8:33 Yeah

Erik Torenberg 8:33 ... and then kind of the conclusion that you came from that.

Vishal Misra 8:36 So in-context learning is when you, uh, show the LLM something it has kind of never seen before. You give it a few examples of this is what it wants, uh, this is what you're trying to do. Then you give a new problem, which is related to the examples that you've shown.

Erik Torenberg 8:55 Yeah.

Vishal Misra 8:56 And the LLM lea-learns in real time what it's supposed to do and solves the problem.

Erik Torenberg 9:01 By the way, the first time I saw this, it absolutely blew my mind.

Vishal Misra 9:04 Yeah.

Erik Torenberg 9:04 I actually, I actually used your DSL-

Vishal Misra 9:07 Mm-hmm

Erik Torenberg 9:07 ... by when I was like first learning about it. So maybe like, kind of like a, like a-

Vishal Misra 9:11 Yeah. So, so I, I, like-

Erik Torenberg 9:13 The DSL thing was just, it's just-

Vishal Misra 9:15 Uh, it was-

Erik Torenberg 9:15 ... crazy this works at all.

Vishal Misra 9:17 I-it's absolutely, you know, mind-blowing that it works. And so going back to that cricket problem-

Erik Torenberg 9:23 Yeah

Vishal Misra 9:23 ... was, you know, i-i-in the mid-'90s, uh, I was part of a group that had created this, uh, cricket portal called Cricinfo.

Erik Torenberg 9:31 Yeah.

Vishal Misra 9:32 Uh, cricket, uh, i-is a very stat-rich sport. You know, you think baseball multiplied by a thousand, and it's got all kinds of stats. And we had created this, uh, online searchable database called StatsGuru, where you could search for anything, any stat related to cricket, and it's been available since 2000.

Erik Torenberg 9:52 Yeah.

Vishal Misra 9:53 But because you can query for anything, everything was be-made available. And how do you make something like that available to the general public?

Erik Torenberg 10:01 Yeah.

Vishal Misra 10:01 Well, they're not gonna write SQL queries. The next best thing at that time was to create a web form. Unfortunately, [chuckles] everything was crammed into that web form. So as a result, you had like 20 drop-downs, 15 check boxes, 18 different text fields. It looked like a very complicated, daunting interface. So as a result, even though it could solve or it could answer any query, almost no one used it. A vanishingly small percentage-

Erik Torenberg 10:29 Yeah. [chuckles]

Vishal Misra 10:29 ... of cricket fans use it because it, it just looked intimidating. And then ESPN bought that site, uh, in 2007. I still know people who, who run the site, and I've always told them, "You know, why don't you do something with StatsGuru?" And in January 2020, uh, the editor-in-chief of, uh, Cricinfo, Sambit Bal, he's, he's a friend, so he came to New York and we'd gone out for drinks. And again, I told him, "You know, why don't you do something with StatsGuru?" So he looks at me and says, "Why don't you do something about StatsGuru?" [chuckles] He was joking, but, uh, that idea kind of stayed with me. And when GPT-3 was released, I thought maybe I could use StatsGuru, use GPT-3 to create a front end for StatsGuru.

Erik Torenberg 11:10 Gotcha.

Vishal Misra 11:11 And so what I did was, uh, I designed a DSL, a domain-specific language, which, uh, converted queries about cricket stats in natural language into this DSL. Now-

Erik Torenberg 11:23 And to be clear, you created this. It wasn't, like, part of, like, any training data-

Vishal Misra 11:26 No training

Erik Torenberg 11:27 ... that was online that, like-

Vishal Misra 11:28 Nothing

Erik Torenberg 11:28 ... GPT could have seen. Yeah.

Vishal Misra 11:29 Nothing GPT could have seen. I created it. I thought, "Okay, this makes sense."

Erik Torenberg 11:32 Yeah.

Vishal Misra 11:33 So I designed that DSL, and then I did that few short learning thing. So I would, uh... So I created about a database of what I would say of 1,500 natural language queries and the DSL corresponding to that query. So when a new query came in, somebody's asking a stats question in English, what I would do is I would go through the natural language queries, do a semantic search, pick the most closely matching top few-

Erik Torenberg 12:04 Yeah

Vishal Misra 12:04 ... uh, and then use that natural language query and its DSL and send that as a prefix.Now, GPT-3, if you recall, had a, a context window of only 2,000 tokens.

Erik Torenberg 12:14 Yeah.

Vishal Misra 12:15 So you had to be very judicious about which examples had you picked. But you picked that, and then you send the new query, and GPT-3 would complete it in the DSL that I had designed.

Erik Torenberg 12:25 That's good.

Vishal Misra 12:25 Which until milliseconds ago, it had never seen.

Erik Torenberg 12:28 Yeah.

Vishal Misra 12:28 And I had no access to internals of GPT-3. I had no access to the weights.

Erik Torenberg 12:32 Yeah.

Vishal Misra 12:32 But still it worked. So that, that's how, so-

Erik Torenberg 12:36 So it's not obvious to me, given your matrix example-

Vishal Misra 12:39 Yeah

Erik Torenberg 12:39 ... of like a prompt and then a distribution, how something like in-context learning-

Vishal Misra 12:45 Works, yeah

Erik Torenberg 12:45 ... w-would work.

Vishal Misra 12:46 Yeah.

Erik Torenberg 12:46 And so, like, I think your first paper tackled this problem.

Vishal Misra 12:50 Right.

Erik Torenberg 12:51 Um, and so maybe you could walk through y-y-your understanding of how LLMs do in tox-- in-context

Erik Torenberg 13:00 learning.

Vishal Misra 13:00 Y-yeah. So, so w-when you think about what in-context learning is, is that a-as you see evidence, so, so, you know, uh, i-in the first paper, what I also did was I, I took this cricket DSL example.

Erik Torenberg 13:13 Okay.

Vishal Misra 13:14 And I, uh, uh, I depicted the next token probabilities-

Erik Torenberg 13:20 Mm-hmm

Vishal Misra 13:21 ... of the model as it was shown more and more examples. So the first time you show it this DSL, the natural language and the DSL, the probabilities of the DSL tokens were, were extremely low because GPT-3 had never seen this thing. When it saw the cricket question, in its mind, it was trying to continue it with an English answer. So the probabilities that were high were all English words.

Erik Torenberg 13:51 Yeah.

Vishal Misra 13:52 Once it saw my prompt where I had the question and the DSL, the next time I had the question in the next row, the probabilities of the DSL token started going up. With every example, it went up, and finally, when I gave the new query, it was like it had almost 100% probability of getting the right token.

Erik Torenberg 14:13 Yeah.

Vishal Misra 14:13 So this is an example of in real time, the model was updating its posterior probability. It was updating its knowledge that, okay, I've seen evidence, this is what I'm supposed to do. Now, this is a colloquial way of saying what Bayesian-

Erik Torenberg 14:30 Yeah

Vishal Misra 14:30 ... inference is. Bayesian updating basically is you start with a prior. When you see new evidence, you update your posterior. That's the mathematical definition. But, but-

Erik Torenberg 14:40 Yeah

Vishal Misra 14:41 ... uh, in, in English, it's basically you see something, you see new evidence, you update your belief about what's happening.

Erik Torenberg 14:47 Yeah.

Vishal Misra 14:47 Right? So it was clear to me that LLMs are doing something which resembles Bayesian updating. So in that first paper, I had this matrix formulation, and I showed that, you know, what it's doing, it looks like Bayesian updating.

Erik Torenberg 15:03 Yeah.

Vishal Misra 15:04 Then we can come to the sort of next series of papers.

Erik Torenberg 15:06 That's right. So, okay. So I mean, it, it, it seemed pretty conclusive to me at that time.

Vishal Misra 15:12 Yeah.

Erik Torenberg 15:12 And then you went quiet for a while, and then I, I still remember the WhatsApp text. You said-

Vishal Misra 15:16 Yeah

Erik Torenberg 15:16 ... "Martín, I know exactly how these things are working now." [both chuckling]

Vishal Misra 15:19 Yeah. Well-

Erik Torenberg 15:20 And then, and then, and then listen, you dropped a series of papers that kind of broke the internet. Like you went super viral on Twitter.

Vishal Misra 15:25 Yeah.

Erik Torenberg 15:25 Like, I mean, people really noticed. Um, uh, and so I, I want to get to that in just a second.

Vishal Misra 15:31 Yeah.

Erik Torenberg 15:31 But before that, um, I remember when your first paper came out, people would be like, "You know, these things are definitely not Bayesian." Like-

Vishal Misra 15:41 Mm

Erik Torenberg 15:41 ... you know, ev- you know, anything could be considered to be Bayesian, but they're not. Like, why do you think that there was this reaction to like, you know-

Vishal Misra 15:51 So-

Erik Torenberg 15:51 ... there's something new, they're not Bayesian. I mean, I felt like there's almost kind of a backlash just because of being characterized as Bayesian.

Vishal Misra 15:57 Yeah, yeah. I, I, I think this whole world of, uh, uh, probability and machine learning, that there have been camps of Bayesian and frequentist.

Erik Torenberg 16:07 Yes.

Vishal Misra 16:08 And I don't want to get in the middle of that sort of political battle, but Bayesian has become like almost like people had a reaction to that it's, it's part of that war.

Erik Torenberg 16:17 I see.

Vishal Misra 16:18 Uh, so-

Erik Torenberg 16:19 So it's, it's like the old-

Vishal Misra 16:21 Yeah

Erik Torenberg 16:21 ... Bayesian frequentist type battle.

Vishal Misra 16:22 Yeah.

Erik Torenberg 16:22 Right.

Vishal Misra 16:22 So the people just had, "Oh, no," and you can say anything is Bayesian, right?

Erik Torenberg 16:26 Yeah.

Vishal Misra 16:26 So I, I said, "Okay, maybe they have a point. Maybe what we are saying is not really Bayesian. How do we prove that it's Bayesian?"

Erik Torenberg 16:33 Right.

Vishal Misra 16:34 So then first, I have to thank you and, uh, Andreessen Horowitz for this. Uh, you know, when I sh- when I, when I said that I, uh, in my first paper, I showed these probabilities.

Erik Torenberg 16:47 Yeah.

Vishal Misra 16:48 Uh, it was because OpenAI had in its chat, uh, interface, uh, this option to display those probabilities, then they stopped.

Erik Torenberg 17:00 Oh, okay.

Vishal Misra 17:00 So we could not peer inside what's going, what's happening. For some reason, they stopped. OpenAI, uh, I'm not gonna get into- [chuckles]

Erik Torenberg 17:07 Yeah

Vishal Misra 17:08 ... the open and closed source, but, but, but they stopped.

Erik Torenberg 17:10 Yeah.

Vishal Misra 17:10 So then we developed our own interface, which could let you look not only at, uh, the probabilities, but also the entropy of the next token.

Erik Torenberg 17:18 Wa-was this on top of an open source model?

Vishal Misra 17:21 Yeah, yeah. So, so you can load any sort of open source model, but, you know, being in academia-

Erik Torenberg 17:26 Yeah

Vishal Misra 17:26 ... we didn't have access to compute. Thanks to-

Erik Torenberg 17:28 Oh, yeah

Vishal Misra 17:28 ... your generous, uh, uh, donation, we got, uh, the clusters-

Erik Torenberg 17:33 Yeah

Vishal Misra 17:33 ... to run, uh, w-what's called TokenProbe. So you can go to tokenprobe.cs.columbia.edu.

Erik Torenberg 17:39 Is it still running?

Vishal Misra 17:39 It's still running. It's still running, and people come to it. Uh, I use it in my classes, uh, to get students to do assignments. They write their own DSLs, and, you know, they say that, that it really helps them understand how these LLMs work.

Erik Torenberg 17:52 So I literally, my understanding of LLMs came from TokenProbe, 'cause-

Vishal Misra 17:55 Yeah

Erik Torenberg 17:55 ... you know, I would sit there and just look at the-

Vishal Misra 17:57 Yeah

Erik Torenberg 17:57 ... the distribution-

Vishal Misra 17:58 Right

Erik Torenberg 17:58 ... as, as you filled out, uh, a prompt. It's actually very, very enlightening.

Vishal Misra 18:01 It'd be-

Erik Torenberg 18:01 So for those of you that are listening, um, what's the URL again?

Vishal Misra 18:05 Tokenprobe-

Erik Torenberg 18:06 Tokenprobe

Vishal Misra 18:07 ... .cs.columbia.edu.

Erik Torenberg 18:09 Yeah. Check it out. It's actually-

Vishal Misra 18:10 Yeah

Erik Torenberg 18:11 ... a very, very useful way to like actually see how the probability distribution gets updated as, as you fill out a prompt.

Vishal Misra 18:18 Right.

Erik Torenberg 18:18 Yeah.

Vishal Misra 18:19 But then I cheated.

Erik Torenberg 18:21 Oh.

Vishal Misra 18:22 I, you know, it was running.

Erik Torenberg 18:24 Yeah.

Vishal Misra 18:24 But I also had access to the GPUs that were powering it.

Erik Torenberg 18:27 Mm-hmm.

Vishal Misra 18:28 And then a-along with colleagues at Columbia, and one of them now is, uh, is at DeepMind, we started to sort of think about how do you really prove that it's Bayesian? To prove-

Erik Torenberg 18:45 Can you just explain it? Actually, I, I actually don't know the answer to this.

Vishal Misra 18:48 Yeah.

Erik Torenberg 18:49 It seemed to me you proved it in the first paper. Like, what was missing?

Vishal Misra 18:53 Well, in the first paper we showed it, it was empirical.

Erik Torenberg 18:56 Oh.

Vishal Misra 18:56 And you could see-

Erik Torenberg 18:57 I see, I see.

Vishal Misra 18:57 You could see-

Erik Torenberg 18:58 Not a mathematical, because it was very obvious to me that, like-

Vishal Misra 19:00 That it's, yeah, it was even obvious to me. But to convince, uh-

Erik Torenberg 19:04 I see

Vishal Misra 19:04 ... you, you could say, you know, people who dismiss that, "Oh, anything can be Bayesian."

Erik Torenberg 19:09 I see, I see.

Vishal Misra 19:09 We had to show it precisely mathematically.

Erik Torenberg 19:12 Got

Erik Torenberg 19:13 it. Got it.

Vishal Misra 19:13 So then we came up with this idea, you know, my colleagues, uh, Naman Agarwal and Siddharth Dalal, we, the, the series of papers were, were written with them. We came up with this idea of a Bayesian wind tunnel.

Erik Torenberg 19:25 Okay.

Vishal Misra 19:25 So what's a wind tunnel? Well, wind tunnel in the aerospace industry is where you test an aircraft in an isolated environment. You don't fly it, and you test, test it against all sorts of, uh, you know, aerodynamic pressure. Then you see what, what it'll withstand, what kind of altitude, pressure, blah, blah, blah. Right? You don't want to do it up in the air testing.

Erik Torenberg 19:47 Yeah.

Vishal Misra 19:48 So we said, okay, why don't we create an environment where we take these architectures, and we tested transformers, Mamba, LSTMs, uh, MLPs, a-all architectures. We said, why don't we create, take a blank architecture, give it a task where it's impossible for the architecture to memorize what the solution to that task should be. The space is combinatorially-

Erik Torenberg 20:17 Yeah

Vishal Misra 20:18 ... impossible for, given the number of parameters, and we took very small models. So it's difficult enough that they cannot memorize it.

Erik Torenberg 20:26 Yeah.

Vishal Misra 20:27 But it's tractable enough that we know precisely what the, the Bayesian posterior should be. You can calculate it analytically. So we gave these models a bunch of tasks where, again, we show that it's impossible to memorize. We trained these models, and we found that the transformer got the precise Bayesian posterior down to ten to the power minus three bits accuracy. It was matching the distribution perfectly. So it is actually doing Bayesian in the mathematical sense, given a task-

Erik Torenberg 21:00 Wow

Vishal Misra 21:01 ... where it has to update its belief. Uh, Mamba also does it reasonably well. LSTMs can do one of the things. So the, the, in the papers, we have a taxonomy of Bayesian tasks. Transformer does everything, Mamba does most of it, LSTMs do only partially, and MLPs fail completely.

Erik Torenberg 21:20 So is this a reflection of the data that it's trained on, or is it more a reflection of the mechanism?

Vishal Misra 21:29 It's the mechanism, it's the architecture. The data decides what tasks it learns.

Erik Torenberg 21:35 Right.

Vishal Misra 21:35 So in the first paper, we had these Bayesian wind tunnels, and we show that, you know, it's doing the job at different tasks. In the second paper, we show why it does it. So we look at the transformers, we look at the gradients, and we show how the gradients actually shape this geometry-

Erik Torenberg 21:53 Ah

Vishal Misra 21:54 ... which enables this Bayesian updating to happen. Then in the third paper, what we did, we take, we took these frontier production LLMs, which have open weights so that we could look inside them, and we did our testing, and we saw that the geometries that we saw in the small, uh, models persisted in models which are, you know, hundreds of millions of parameters. The same signature existed. The only thing is that, uh, because they are trained on all sorts of data, it's a little bit dirty or messy.

Erik Torenberg 22:29 Yeah.

Vishal Misra 22:29 But you can see the same structure. So the, the whole idea behind the Bayesian wind tunnel was, unlike these, uh, production LLMs, where you don't know what they have been trained on-

Erik Torenberg 22:39 Right

Vishal Misra 22:39 ... so you cannot mathematically compute the posterior. So again, how do you prove it? I mean, it looks Bayesian, you know, from the first paper.

Erik Torenberg 22:47 From the first paper, yeah.

Vishal Misra 22:47 From the paper it looks Bayesian, but, you know.

Erik Torenberg 22:48 Looks Bayesian to me. Yeah.

Vishal Misra 22:49 So the wind tunnel sort of solved that problem for us.

Erik Torenberg 22:51 Mm-hmm.

Vishal Misra 22:51 We said, okay, let's start with a blank architecture, give it a task where we know what the answer is. It cannot memorize it. Let's see what it does. And yeah.

Erik Torenberg 23:02 So do you think this provides any sort of, like, indication of how humans think, or do you think that these things are totally independent?

Vishal Misra 23:08 No, no, it, it does provide, right? So, you know, human beings also, uh, update our beliefs as we see new evidence.

Erik Torenberg 23:16 Right.

Vishal Misra 23:16 Right? So we do in some sort of, uh, in some sense, uh, Bayesian updating, but we do something more than that. I-I'll come to that. But, uh, these transformers, uh, or even Mamba do this Bayesian updating.

Erik Torenberg 23:32 Yeah.

Vishal Misra 23:33 And, uh, but, but, but the difference with humans is, you know, we, we'll update our posterior when we see some new evidence, but the way our brains have evolved over hundreds of millions of years is our optimization e-objective has been, don't die and reproduce, right? That's been sort of the driving force, and our brains have learned to adjust. And so when we see some danger, there's some-something rustling in that bush, don't go near. We know how to react to that danger. We know how to s- uh, save ourselves. We internalize that learning, and our brain cells or our synapses remain plastic throughout our lifetime. What happens with LLMs is, once the training is done, those weights are frozen.When you're doing an inference or, for instance, in-context learning or anything, during that conversation, okay, you're doing Bayesian inference, but then you forget. The next time a new conversation starts with zero context, you don't retain any learning that happened in the previous instance. So, so for instance, with the, the cricket DSL that I was doing, every invocation of it was fresh. It did not remember the last time I sent a query what the DSL looked like. So that's one difference between, uh, how humans, uh, uh, use sort of Bayesian, uh, updating, which is we remain plastic all our lives-

Erik Torenberg 25:11 Yeah

Vishal Misra 25:12 ... whereas, uh, LLMs are frozen. And there's another, uh, sort of difference which, uh, if you want me to get-

Erik Torenberg 25:21 Tell me. Yeah, yeah, yeah. Yeah.

Vishal Misra 25:23 So, so the other difference is, uh, um, well, f-first, you know, our objective is don't die, reproduce. LLM's objective is predict the next token as accurately as possible, right? So a-a-all these, uh, scary stories that, uh, you, you read about that, oh, the LLM tried to deceive and it tried to prevent itself from being shut down, that's not a function of the architecture. That's a function of the training data.

Erik Torenberg 25:52 That's it, yeah.

Vishal Misra 25:52 It has been fed, you know, articles on Reddit or Asimov or whatever.

Erik Torenberg 25:58 I mean, just today, by the way-

Vishal Misra 25:59 Yeah

Erik Torenberg 25:59 ... Dario allegedly said that, uh, you can't rule out that they're conscious.

Vishal Misra 26:07 You can rule out they're conscious. [laughs]

Erik Torenberg 26:08 I think, I mean, come on.

Vishal Misra 26:10 As I said, you know, uh, Anthropic makes great products. Cloud code is fantastic. Cohort is fantastic. But they are grains of silicon doing matrix multiplication.

Erik Torenberg 26:22 Yeah.

Vishal Misra 26:22 They don't have consciousness. They don't have an inner monologue. They don't, uh, they're not driven by the same objective function, don't die, reproduce, right? They're driven by, don't make a mistake on the next token, and that's driven entirely by the training data, right? You train the LLM with stories of Asimov or Reddit where, you know, to survive it's going to do this or that, it'll reproduce that. So i-it's, it, it's a reflection, it's not a mind.

Erik Torenberg 26:50 And, and the results, just to say it for the tenth time-

Vishal Misra 26:54 Yeah

Erik Torenberg 26:54 ... are perfectly Bayesian-

Vishal Misra 26:56 Perfectly, yeah

Erik Torenberg 26:57 ... to the, to the, to the digit.

Vishal Misra 26:59 To the digit.

Erik Torenberg 27:00 Yeah.

Vishal Misra 27:00 Yeah. I mean, I, I trained it for 150,000 steps, and, uh, the accuracy was 10 to the power minus three bits.

Erik Torenberg 27:08 That's how-

Vishal Misra 27:08 I could have trained it for... You know, th-this happened in half an hour-

Erik Torenberg 27:12 Yeah

Vishal Misra 27:12 ... on the infrastructure that you provided for Token Prop-

Erik Torenberg 27:14 Right. Yeah.

Vishal Misra 27:14 In the background, I, I could use those APUs [chuckles] to train, but, uh-

Erik Torenberg 27:18 That's great

Vishal Misra 27:18 ... so thank you again for that. But, so now human beings,

Vishal Misra 27:22 coming back to it, we, we are Bayesian-

Erik Torenberg 27:24 Yeah

Vishal Misra 27:24 ... but we do something else. You know, when I, when I, when I throw this pen at you, what'll you do?

Erik Torenberg 27:30 Dodge it or-

Vishal Misra 27:30 Dodge it

Erik Torenberg 27:31 ... dodge it? Yeah.

Vishal Misra 27:31 Why will you dodge it?

Erik Torenberg 27:34 Um, to avoid being hit.

Vishal Misra 27:35 Avoid being hit.

Erik Torenberg 27:37 Yeah.

Vishal Misra 27:37 But your head is not doing a Bayesian calculation of, uh, okay, this pen is coming, the probability that it hits me, uh, it'll cause this much pain or all that.

Erik Torenberg 27:47 Correct.

Vishal Misra 27:47 What you're essentially doing in your head is you're doing a simulation.

Erik Torenberg 27:51 Ah, right.

Vishal Misra 27:52 You see the b-uh, the, the, the pen coming and you know that it'll come and hit me. Your mind simulates and you dodge it, right? So all of deep learning is, uh, doing correlations. It's not doing causation.

Erik Torenberg 28:11 Yeah.

Vishal Misra 28:13 Causal models are the ones that are able to do simulations and interventions. So, you know, Judea Pearl has this whole, uh, causal hierarchy-

Erik Torenberg 28:21 Yeah

Vishal Misra 28:21 ... where the first hierarchy, and the first hierarchy is association, which is you build these correlation models. Deep learning is beautiful. It, it's extremely powerful. I mean, you see every day, all these models are, like, amazingly good.

Erik Torenberg 28:34 Yeah.

Vishal Misra 28:35 They do association. The second is intervention in the hierarchy.

Erik Torenberg 28:39 Yeah.

Vishal Misra 28:41 Deep learning models do not do that. Third is counterfactual. So both intervention and counterfactual, you can imagine it, it, it's some sort of simulation. You, you build a model of, causal model of what's happening, and then you are able to simulate. So our brains do that. The current architectures don't do that. Another example I think which will make it clear is, uh, the difference between, uh, I'll use these technical term, Shannon entropy-

Erik Torenberg 29:11 Mm-hmm

Vishal Misra 29:11 ... and Kolmogorov complexity.

Erik Torenberg 29:13 Sure.

Vishal Misra 29:14 So if you look at the Shannon entropy of the digits of pi-

Erik Torenberg 29:19 Yeah

Vishal Misra 29:19 ... it's infinite.

Erik Torenberg 29:20 Sure.

Vishal Misra 29:21 It's impossible to predict-

Erik Torenberg 29:22 Of course

Vishal Misra 29:22 ... and learn what digit will come after.

Erik Torenberg 29:25 Yeah.

Vishal Misra 29:25 So th-that's the definition of Shannon entropy, and Shannon entropy sort of tries to build a correlation. It tries to learn the correlation. Deep learning does the Shannon entropy.

Erik Torenberg 29:36 Yeah.

Vishal Misra 29:37 Kolmogorov complexity, on the other hand, is the l- is the length of the shortest program-

Erik Torenberg 29:43 Yeah

Vishal Misra 29:44 ... which will reproduce, uh, the string that you, that is under question.

Erik Torenberg 29:49 Yeah.

Vishal Misra 29:49 Now, the program to get the digits of pi are very small.

Erik Torenberg 29:53 Yeah.

Vishal Misra 29:54 Thanks to Ramanujan and others, [chuckles] you know, there are all sorts of-

Erik Torenberg 29:57 Yeah, yeah, yeah

Vishal Misra 29:57 ... really small program that can reproduce it exactly. So the Kolmogorov complexity of pi is very small. Shannon entropy is infinite.

Erik Torenberg 30:06 Yeah.

Vishal Misra 30:07 I think deep learning is still in the Shannon entropy world. It has not crossed over to the Kolmogorov complexity and the causal world.

Erik Torenberg 30:16 Wow. Interesting.

Vishal Misra 30:17 Right? So, um-

Erik Torenberg 30:19 Do you-- So to what extent do you think this provides us research directions to kind of improve the state of the art? So let me just give you a-

Vishal Misra 30:27 Yeah

Erik Torenberg 30:27 ... specific example. You talked about human beings don't actually updateYou know, the matrix, they don't kind of update their weights. But right now there's a lot of research on continual learning.

Vishal Misra 30:39 Yeah. Yeah.

Erik Torenberg 30:39 Which is, you know... So does your work provide some guidance of how you might approach those problems?

Vishal Misra 30:46 I, I, I-

Erik Torenberg 30:46 And, and in particular, I've always had this question, which is, we use so much data and so much compute-

Vishal Misra 30:51 Yeah

Erik Torenberg 30:51 ... to create these models. Like, is it even reasonable to think that you could update the weights and actually have a meaningful impact, you know, with, in, in real time? I mean, it just seems like you would just need so much more data in order-

Vishal Misra 31:04 Yeah

Erik Torenberg 31:04 ... to do that. So can you start answering these questions?

Vishal Misra 31:06 You, you can start answering some of these questions, and, a-a-and o-one of the misconceptions that, uh, exists today is that scale will solve everything. Scale will not solve everything. You, you, you need a different kind of architecture, and this continual learning i-is a difficult problem. You have to balance the fact that you learn something new against the risk of catastrophic forgetting.

Erik Torenberg 31:28 Right.

Vishal Misra 31:28 Right?

Erik Torenberg 31:28 Right.

Vishal Misra 31:29 If you update the weights and you forget what, what was important and what you have already learned, then, then you are, you know, y-you're not making progress. Then it'll just be some sort of random chaotic model.

Erik Torenberg 31:40 Yeah.

Vishal Misra 31:41 So to solve that problem is difficult. That's one aspect of it. So, so, so, you know, to get to what is called AGI, I think there are two things that need to happen. One is this plasticity, which has to be implemented through continual learning.

Erik Torenberg 31:58 Yeah.

Vishal Misra 31:58 Secondly, we have to move from correlation to causation.

Erik Torenberg 32:01 Yeah.

Vishal Misra 32:02 That's, uh, uh-

Erik Torenberg 32:04 I, I, I mean, how, how, how much is this similar to what Yann LeCun talks about with the, like-

Vishal Misra 32:09 So Yann LeCun, uh-

Erik Torenberg 32:10 ... like the causality planning.

Vishal Misra 32:12 Yeah.

Erik Torenberg 32:12 You know, predicting how, like how your action would-

Vishal Misra 32:15 I-i-it is, it is related. You know, he's coming at it from a different angle than the GPAR model.

Erik Torenberg 32:20 Right.

Vishal Misra 32:20 But it is related. The, the, the other thing is, uh, you know, the first time I came on this podcast, I, I mentioned this test of AGI.

Erik Torenberg 32:28 Yeah.

Vishal Misra 32:28 The Einstein test.

Erik Torenberg 32:30 I don't remember.

Vishal Misra 32:31 So I said, you know, uh, uh, you take an LLM-

Erik Torenberg 32:34 Yeah

Vishal Misra 32:34 ... and train it on pre-1916 or 1911 physics.

Erik Torenberg 32:39 Oh, right.

Vishal Misra 32:39 And see if it can come up with the theory of relativity.

Erik Torenberg 32:44 Yeah.

Vishal Misra 32:45 If it does, then we have AGI. I mean, it's a high bar, but you know, we should have high bars. It won't, and this is the same test that I think Demis, uh, mentioned, uh, at, uh, the India AI Summit a couple of weeks ago. It's created a lot of news. But why, why is that, and how is that related to this idea of Shannon versus Kolmogorov? So at the time of Einstein, there were a lot of clues that Newtonian mechanics, there was something missing.

Erik Torenberg 33:18 Yeah.

Vishal Misra 33:19 Right? Uh, people knew that Mercury's orbit didn't make sense. There was something off about it. Then there were these experiments done, uh, the Michelson-Morley experiments, where they were trying to figure out, uh, uh, this, uh, uh, medium called, uh, the ether through which light travels. And they felt that if, you know, you bounce light in different directions, uh, the speed might change, and they, they could detect a change in the speed of light. They tried several experiments. They had really precise instruments which could measure the speed, and they found nothing. They found that the speed of light did not change at all. Then there were... There was the whole issue of black holes.

Erik Torenberg 34:07 Yeah.

Vishal Misra 34:07 Then gravitational lensing. So there were a lot of these signs that Newtonian mechanics is not really explaining everything.

Erik Torenberg 34:17 Yeah.

Vishal Misra 34:18 But until Einstein came up with a new representation of the space-time continuum-

Erik Torenberg 34:25 Right

Vishal Misra 34:25 ... we were stuck.

Erik Torenberg 34:26 Yeah.

Vishal Misra 34:26 So if you had a model that just looked at correlations and sort, uh, sees all of this, you know, the, all of these, uh, pieces of individual evidence and put together, it would not have come up with the beautiful equation that Einstein came up with. You know, uh, I'm forgetting exactly what it is. G mu v equals eight pi T mu v, some-something like that.

Erik Torenberg 34:52 Yeah, yeah.

Vishal Misra 34:52 Where, you know, uh, the, the, the equation of, uh, the rela-

Erik Torenberg 34:55 Sure

Vishal Misra 34:55 ... uh, the space-time continuum, the, the tensor. So he came up with a new formulation.

Erik Torenberg 35:01 Yeah.

Vishal Misra 35:02 So he kind of rejected the existing axioms. He came up with a very short Kolmogorov representation of-

Erik Torenberg 35:10 Oh, interesting

Vishal Misra 35:10 ... the world.

Erik Torenberg 35:11 Yeah.

Vishal Misra 35:12 One equation. From that equation, everything else follows.

Erik Torenberg 35:15 Yeah, yeah, yeah.

Vishal Misra 35:16 Right? Whether you're talking about gravitational waves or black holes or Mercury or how GPS works. You know, GPS, the GPS that we use every day in our phones, it uses the equation of relativity.

Erik Torenberg 35:29 So do you, does, does this end up becoming like, um, you, you, you almost have to ignore the majority of previous data in order to do it, which LLMs can't because they're trained on the majority-

Vishal Misra 35:44 Yeah

Erik Torenberg 35:44 ... of previous data.

Vishal Misra 35:45 Yeah.

Erik Torenberg 35:45 It's like you almost have, like, this kind of data gravity [chuckles] that's pulling you back. It's like, it's like everybody said it's X.

Vishal Misra 35:51 Mm-hmm.

Erik Torenberg 35:52 There's a little bit of evidence that it's Y, but because everybody said it's X, like, the LLM will always say it's X.

Vishal Misra 35:57 It, it'll always say X. It'll treat that Y as an anomaly.

Erik Torenberg 36:00 Actually, this Kolmogorov is actually a very nice way to say it.

Vishal Misra 36:02 Yeah.

Erik Torenberg 36:02 Which is like, it's like, ah, I, so now, okay, now I get your Shannon entropy versus Kolmogorov.

Vishal Misra 36:08 Yeah.

Erik Torenberg 36:08 Like, one of them is, like, the total amount of information there, that will always be bound to the total amount of information there, which is what happens right now.

Vishal Misra 36:17 Yeah.

Erik Torenberg 36:18 Where you can actually describe an- another, another motion, you can describe everything with a shorter description with the new data, which would be a totally different motion, which would be like-

Vishal Misra 36:31 Yeah. You, you need a new representation. Right.

Vishal Misra 36:34 Yeah.

Erik Torenberg 36:34 You know, another way that I've always thought about these, and I thought you articulated it well the last time we talked about it, which is the universe is this very, very complex space, and then-You know, somehow humans map it into a manifold-

Vishal Misra 36:49 Mm-hmm

Erik Torenberg 36:49 ... that's less complex.

Vishal Misra 36:51 Yeah.

Erik Torenberg 36:51 And then that gets kind of written down, and then the LLM... So that's kinda some, some distribution, some-

Vishal Misra 36:57 Mm.

Erik Torenberg 36:57 You know, it's still a very large space, but it's-

Vishal Misra 36:59 Yeah

Erik Torenberg 36:59 ... it's, it's a bounded space, and the LLM learned that manifold, and then they kind of use, you know, Bayesian inference to move up and down that manifold.

Vishal Misra 37:07 Right.

Erik Torenberg 37:07 But they're kind of bound to that manifold.

Vishal Misra 37:09 Yeah.

Erik Torenberg 37:10 And then, again, I don't wanna put words in your mouth, and then, but, like, what they can't do is, is generate a new manifold.

Vishal Misra 37:15 New manifold, yeah, yeah.

Erik Torenberg 37:16 Which requires understanding the way that the universe works, then coming up with a new representation of the universe.

Vishal Misra 37:21 Yeah. And th- this is what relativity is, right?

Erik Torenberg 37:23 Yeah, exactly.

Vishal Misra 37:24 Einstein had to create a new manifold.

Erik Torenberg 37:25 Yeah, yeah, yeah.

Vishal Misra 37:26 If you just stuck with the old manifold of the Newtonian physics-

Erik Torenberg 37:29 Right

Vishal Misra 37:30 ... then you would see these correlations, but you could not come up with a manifold that explain them. So you need to come up with a new representation.

Erik Torenberg 37:38 Yeah.

Vishal Misra 37:38 So to me, you know, there are lots of definitions of AGI, uh, you know, Turing test, we have already passed that. You know, performing economically useful work, every day you see, you know, uh, LLMs are doing that.

Erik Torenberg 37:51 Do we? I don't know.

Vishal Misra 37:52 No, I mean, they are.

Erik Torenberg 37:53 I mean, um, I mean, without human intervention?

Vishal Misra 37:55 No, no, no. So, so that, that's different.

Erik Torenberg 37:56 Okay.

Vishal Misra 37:56 But, but still, you know, it's like a car can run faster than humans, right?

Erik Torenberg 38:01 Yeah. I mean, that's a, that's the-

Vishal Misra 38:02 Yeah.

Erik Torenberg 38:03 That's, um-

Vishal Misra 38:03 Yeah. I mean-

Erik Torenberg 38:04 ... that's a very shallow definition.

Vishal Misra 38:05 Yeah. So all these definitions-

Erik Torenberg 38:06 Cars do useful work, but-

Vishal Misra 38:07 You know, maybe, you know, in six months you'll have, uh, Claude or what a Gemini do without intervention, the coding tasks which are well-defined, well-scoped.

Erik Torenberg 38:19 Yeah.

Vishal Misra 38:19 That's possible. But to me, AGI will happen when th- these two problems get solved. Plasticity-

Erik Torenberg 38:26 Right

Vishal Misra 38:27 ... continual learning properly, and building a causal model from, you know, uh, uh, i- in a more data efficient manner.

Erik Torenberg 38:36 W- we are hearing people now talking about, you know, seeing general, like Donald Knuth, for example.

Vishal Misra 38:43 Yes. [chuckles]

Erik Torenberg 38:43 In, in the last few days, right?

Vishal Misra 38:45 Yeah, yeah.

Erik Torenberg 38:45 You know, had this, you know, this, you know, aha moment apparently that-

Vishal Misra 38:50 Mm

Erik Torenberg 38:50 ... kind of made, went viral on X. So do you think that that suggests that we're seeing generality or?

Vishal Misra 38:54 No, no, no. So, so that actually, uh, I mean, to me it validates what I've been talking about for a while now.

Erik Torenberg 39:01 How so? How so?

Vishal Misra 39:02 So, uh, so if you, if you read what he did, uh, with the help of, uh, you know, a colleague, he got the LLMs to solve this particular problem of, uh, finding Hamiltonian cycles, uh, odd numbers. We won't get into that.

Erik Torenberg 39:17 Okay.

Vishal Misra 39:17 And he got the LLMs to keep solving for one odd number after the other, right?

Erik Torenberg 39:21 Right.

Vishal Misra 39:22 What he also got to do is, after it found a solution for a particular value of M, he made the LLM update its memory with exactly what it learned in solving that problem. So the LLMs tried many different things.

Erik Torenberg 39:39 Yeah.

Vishal Misra 39:39 You know, something worked, update the m- the memory. So that's kind of like hacking together plasticity.

Erik Torenberg 39:45 Yeah.

Vishal Misra 39:46 Right? It- it's learning what it has done as we went along. Again, it's, it's a hacked version of it. You're not changing the weights, you're just sort of, uh, improving the context.

Erik Torenberg 39:57 Right.

Vishal Misra 39:57 Right? But y- uh, as you learned, and even after that, so this whole space of Hamiltonian cycles and the, the, the associated math is well represented in the manifolds that these LLMs have been trained on.

Erik Torenberg 40:12 Right.

Vishal Misra 40:13 You just have, have to find the right connection, and LLMs, you know, compute, you throw enough compute, they will find the right connection. So Knuth was able to find the LLMs', uh, attempts, and eventually it needed him to put together what he saw into a solution. It definitely helped him get to the solution, but he had to create the new sort of manifold-

Erik Torenberg 40:42 Yeah, yeah

Vishal Misra 40:42 ... to come to the solution. The LLMs were, after a while, stuck, right? He, he, he, you, you read what he has written. I mean, it just hot off the press, uh, I think two days ago.

Erik Torenberg 40:52 It was two days ago, yeah, yeah.

Vishal Misra 40:53 Two days ago. But-

Erik Torenberg 40:54 Yeah

Vishal Misra 40:54 ... uh, eventually he used the solutions, and he came up with, uh, the proof.

Erik Torenberg 40:58 Yeah.

Vishal Misra 40:58 Right? So it's like, you know, it's like Einstein saw all these evidences, then he thought, "What will explain?" Uh, he came up with a causal model.

Erik Torenberg 41:11 Yeah.

Vishal Misra 41:12 So Knuth and his brain i- is sort of the-

Erik Torenberg 41:15 It's doing the, that's doing the Kolmogorov-

Vishal Misra 41:16 Kolmogorov, yeah, yeah, yeah

Erik Torenberg 41:17 ... is, is the human.

Vishal Misra 41:18 Right. And, uh, the LLMs are extremely efficient at doing the Shannon part of it. It, it, it found all the solutions by trying, you know, various things-

Erik Torenberg 41:26 That is such a-

Vishal Misra 41:26 ... and, and learning more and more

Erik Torenberg 41:28 ... clever way to decompose it. I'm wondering, like, do you think this, again, I'm gonna ask the, uh, the same question, uh, again, which is do you think this provides some sort of insight on, like, the next problem to tackle? Like-

Vishal Misra 41:37 Yeah

Erik Torenberg 41:37 ... like, like is there a mechanism that will get the Kolmogorov complexity or not? Like, is this...

Vishal Misra 41:45 It, it, it tells us, uh, which direction to pursue.

Erik Torenberg 41:49 But, uh, but clearly not how to do it, like-

Vishal Misra 41:50 Not how to do it, but even Kolmogorov complexity has largely remained a sort of a theoretical construct.

Erik Torenberg 41:56 Yeah, for sure.

Vishal Misra 41:57 It, it, it, it-

Erik Torenberg 41:58 There's no al- there's no algorithm.

Vishal Misra 41:59 There's no, uh, there haven't been practical implementations of finding-

Erik Torenberg 42:03 Yeah

Vishal Misra 42:03 ... the shortest program.

Erik Torenberg 42:04 Yeah.

Vishal Misra 42:05 We know it exists, you know. You can argue about it, it, but, uh, so, so that's where, uh, I think-

Erik Torenberg 42:12 Yeah

Vishal Misra 42:12 ... it's my bias, [chuckles] that's where our energy-

Erik Torenberg 42:14 Yeah

Vishal Misra 42:14 ... should be focused, not larger models with more tokens.

Erik Torenberg 42:17 Can you, and can you, can you tie the two things? Like, how does that pair with doing simulation, or is a simulation totally orthogonal?

Vishal Misra 42:26 No, s- simulation is, uh, is related, right?

Erik Torenberg 42:28 So you think it, like, basically you do simulation and somehow that is a step towards doing the Kolmogorov complexity?

Vishal Misra 42:37 It, it's, it's, the simulator is the, i- is the program that we create. It may not be the perfect program.

Erik Torenberg 42:43 Oh, I see. And you say-

Vishal Misra 42:44 But in our heads we create this, uh, simulator that when I'm throwing the pen, you know that it's coming at you, right?

Erik Torenberg 42:49 Yeah.

Vishal Misra 42:49 And you duck.So, so you're not computing the probabilities, uh, as it goes, but, but you have, you know, you, you build an approximate-

Erik Torenberg 42:57 That's a very physical thing versus we are talking more conceptually.

Vishal Misra 42:59 Conceptually, but, but it's a similar thing.

Erik Torenberg 43:01 And you think those are the same mechanism?

Vishal Misra 43:02 It's the same mechanism.

Erik Torenberg 43:03 Really?

Vishal Misra 43:03 Yeah. You, you have to build a causal model.

Erik Torenberg 43:06 Yeah.

Vishal Misra 43:06 Right?

Erik Torenberg 43:06 I see. I see. Yeah.

Vishal Misra 43:08 For most things, right?

Erik Torenberg 43:09 Yeah.

Vishal Misra 43:09 So you have to move from correlation to causation. I mean, we've heard this term-

Erik Torenberg 43:12 Yeah

Vishal Misra 43:13 ... you know, ad infinitum.

Erik Torenberg 43:16 Yeah.

Vishal Misra 43:16 But here it, it's making a difference in the way we view intelligence.

Erik Torenberg 43:22 Yeah. How, how, how has, how has the last three papers been received?

Vishal Misra 43:26 No, I don't know. They're, they're... Well, uh, uh-

Erik Torenberg 43:29 I mean, I mean-

Vishal Misra 43:29 The archive versions were like-

Erik Torenberg 43:30 Let me tell you.

Vishal Misra 43:31 Yeah.

Erik Torenberg 43:31 Uh, it, I mean, um, l- lot of great reception. A lot of people read it. I'm just wondering like what kind of feedback that you've got.

Vishal Misra 43:38 I'm getting good feedback, but I'm an outsider in this field, right? [chuckles]

Erik Torenberg 43:41 All right. That's right.

Vishal Misra 43:42 Like I'm a-

Erik Torenberg 43:43 This networking guy coming in.

Vishal Misra 43:43 I'm a networking guy. Why, why is he writing about, you know, uh, learning and machine learning and deep learning and Bayesian? So, but, but people who have actually taken the time to read those papers, I'm getting really good feedback. Uh, there was a recent paper by Google, uh, Research-

Erik Torenberg 44:00 Yeah

Vishal Misra 44:00 ... which tried to teach, uh, LLMs by some sort of RLHF to do Bayesian learning properly.

Erik Torenberg 44:06 Yeah.

Vishal Misra 44:06 And that's going in this direction. I think people are coming around to the view that, okay, LLMs are doing Bayesian learning.

Erik Torenberg 44:13 Okay.

Vishal Misra 44:13 Um, I know that some people also looked at the Bayesian Wind Tunnel paper, the archive version.

Erik Torenberg 44:18 Yeah.

Vishal Misra 44:19 And they reproduced the experiments.

Erik Torenberg 44:21 That's great.

Vishal Misra 44:21 They, they just saw what was written, and they, they did the training, and they saw, yeah, yeah, this is actually happening, so.

Erik Torenberg 44:26 That's great. So what's next?

Vishal Misra 44:29 Um, what's next is, uh, you know, these two parallel, uh, tracks, I hope to make progress there, plasticity and cause- causal, causal models.

Erik Torenberg 44:40 Because it, 'cause it, to date, you've taken an existing mechanism-

Vishal Misra 44:44 Yeah

Erik Torenberg 44:44 ... and you've created a formal model of how it works.

Vishal Misra 44:47 Yeah.

Erik Torenberg 44:47 And so now you're actually interested in, in creating-

Vishal Misra 44:49 In, in improving. Yeah

Erik Torenberg 44:50 ... in creating a new mechanism.

Vishal Misra 44:51 Yeah, yeah.

Erik Torenberg 44:52 And do you think it's an entirely different architecture?

Vishal Misra 44:55 I... Well-

Erik Torenberg 44:55 Or do you, or do you think, do you think LLMs are like part of the solution?

Vishal Misra 44:58 I, I think LLMs are definitely part of the solution.

Erik Torenberg 45:00 I see.

Vishal Misra 45:00 But, but there, there has to be something more.

Erik Torenberg 45:02 Another mechanism.

Vishal Misra 45:03 So, you know, I was not interested in sort of cataloging what all these-

Erik Torenberg 45:07 Yeah

Vishal Misra 45:07 ... LLMs can do.

Erik Torenberg 45:08 Yeah.

Vishal Misra 45:08 I was more interested in why are they and how are they doing it.

Erik Torenberg 45:12 Yeah.

Vishal Misra 45:12 I think now we have a good grip on the why and how.

Erik Torenberg 45:18 Yeah.

Vishal Misra 45:18 A- and the next step is to, you know, uh, move them to the next level. We, we... Now I, I think we have a fairly good understanding of what the limits are.

Erik Torenberg 45:26 Yeah.

Vishal Misra 45:27 Now how do you, uh, go to the next step?

Erik Torenberg 45:30 Is there an eq- is there an equivalent kind of theoretical framework for causality that applies here? Like similar to like Bayesian for inference?

Vishal Misra 45:40 Well, Judea Pearl's whole causal, causal hierarchy, I think, uh-

Erik Torenberg 45:44 I think that's the right one

Vishal Misra 45:44 ... that's, that's a very good one. You know, the whole do calculus, uh, approach. I, I, I think it's a good way to think about it. You know, the, the sort of association, intervention, counterfactuals.

Erik Torenberg 45:58 Yeah.

Vishal Misra 45:59 It takes you from correlation to actually simulation-

Erik Torenberg 46:02 Yeah

Vishal Misra 46:03 ... i- i- in a mathematical way.

Erik Torenberg 46:05 That's great. All right. Well, listen, really appreciate you coming. This is awesome. So we had you here for the first paper where you had the empirical results.

Vishal Misra 46:11 Mm-hmm.

Erik Torenberg 46:12 Then we had you back when you actually have like the formal proof.

Vishal Misra 46:15 Yeah.

Erik Torenberg 46:15 And hopefully, the next time you come back, you will have a proposal for the mechanism that, uh, that actually provides the next step.

Vishal Misra 46:23 Hopefully.

Erik Torenberg 46:24 That work?

Vishal Misra 46:24 Yeah.

Erik Torenberg 46:24 All right, cool.

Vishal Misra 46:25 I'm working on it.

Erik Torenberg 46:25 Thanks a lot for coming in.

Vishal Misra 46:26 Thank you for having me.

Erik Torenberg 46:35 This was a pleasure. [upbeat music]