No Priors Ep. 57 | With LangChain CEO and Co-Founder Harrison Chase

1. 0:00 – 1:45

   Introduction to LangChain

   1. SGSarah Guo0:05

      Hi, listeners, and welcome to another episode of No Priors. Today, we're talking to Harrison Chase, the CEO and co-founder of LangChain, a popular open-source framework and developer toolkit that helps people build LLM applications. We're excited to talk to Harrison about the state of AI application development, the open-source ecosystem, and its open questions. Welcome, Harrison.

   2. HCHarrison Chase0:25

      Thanks for having me. I'm excited to be here.

   3. SGSarah Guo0:28

      LangChain's a, a really unique story, and it started actually as a personal project for you. Can you talk a little bit about what, what LangChain is and what it was originally?

   4. HCHarrison Chase0:37

      Yeah. Absolutely. So how I'd - how I would answer the question what LangChain is has kind of evolved over time, as has the entire landscape. LangChain, the open-source, uh, package, started, yeah, as a side project. Um, so, so my background's in ML and MLOps. I was at, I was at my previous company. I, I knew I was gonna leave. I didn't know what I was gonna do. This was in September, October of 2022. Um, and so I went to a bunch of hackathons, went to a bunch of meetups, chatted with folks that were playing around with LLMs, um, and saw some common abstractions, put it in a Python project as a just fun side project. Turned out to strike a chord, be fantastic timing, you know, ChatGPT came out, like, a month later. Um, and it's kind of evolved from there. So right now, LangChain, the company, um, there's really two main products that we have. One is the LangChain open-source packages. I'm happy to dive into that more. And then the other is LangSmith, a platform for, for testing, evaluation, monitoring, and, and all of those types of things. And so, you know, what LangChain is has evolved over time as, uh, the company's grown.

   5. SGSarah Guo1:42

      One thing that we talked about the last time we say each other

2. 1:45 – 4:30

   Managing an open source environment

   1. SGSarah Guo1:45

      in person was just how quickly, like, the AI, um, ecosystem and research field is evolving and what it means to manage an open-source project through that. Can you talk a little bit about what you decide to keep stable and change when you both have, like, big ecosystem of users now and, like, very rapidly changing environment of applications and technology?

   2. HCHarrison Chase2:06

      That's been a fun exercise. So, I mean, if we go back to the original version of LangChain, what it was when it came out was essentially three kind of, like, high-level implementations. Two were based on research papers, and then one was based on Nat Friedman's, like, NatBot type of agent web crawler thing. And so there was some high-level kind of, like, uh, abstractions, and then there was a few, like, integrations. So we had integrations with, I think, like, OpenAI, Cohere, and Hugging Face to start or something like that. And those two layers have kind of, like, remained. So we have, you know, 700 different integrations. We have a bunch of kind of, like, higher level chains and agents for, for doing particular things. I think the thing that we've put a lot of emphasis in, um, to your point around kind of, like, what's remained constant and what's remained, uh, and what's changed, is, like, a lower level kind of, like, abstraction and runtime for, for joining these things together. One of the things that we pretty quickly saw was that as people wanted to improve the performance, go from prototype to production, they wanted to customize a lot of these bits. And so we've invested a lot in, uh, a lower level kind of, like, chaining protocol, so LangChain expression language, and then in, in, uh, different protocol LangGraph, which is one- something we're really excited about, and that's more aimed at, uh, basically graphs that are not DAGs. So, you know, all these agents are basically running an LLM in a loop. You need cycles, um, and, and so LangGraph helps with that. And so I think what we've kind of seen is the underlying bits of, um... There's all these different integrations, and, like, you know, there's, there's LLMs, vector stores, and sometimes they change, right? When chat models came out, like, that was a, that's a very big change in the API interface, and so we had to add a new abstraction for that. But those have, especially over the past few months, remained relatively stable. Um, we've invested a lot in this underlying runtime, which emphasizes a few things, uh, streaming, structured outputs, and, and the importance of those has remained relatively stable. But then the way that you put things together and the kind of, like, patterns, um, for building things has definitely evolved over time, from, like, simple chains, to complex chains, to then these kind of, like, autonomous agents, to now something, um, maybe in the middle of, like, complex state machines or graphs or something. And so it's really that upper layer, which is, like, the common ways to put things together, that I think we've seen the most rapid kind of, like, churn.

   3. EGElad Gil4:26

      What do you think is still missing from, uh, really getting to performant agents? There's a

3. 4:30 – 10:03

   Developing useful AI agents

   1. EGElad Gil4:30

      number of companies that have been started recently that are really focused on sort of the agentic world and pushing that whole thread and certain types of automation forward. What do you view as the big components that you all don't have, or that maybe the industry more generally doesn't have, that, that still needs to come into place to help drive those things ahead?

   2. HCHarrison Chase4:44

      Yeah. That's a, that's a really good question. I think there's a few things. One, I think, like, um, like, figuring out the right UX for a lot of these things is still an open question in my mind. Um, and, you know, that's not necessarily something we can help with. (laughs) Uh, I think there's a lot of exploration that applications need to do to figure out how to, you know, communicate what these agents are good at and bad at to end users and expose ways to, um, maybe let them course correct and see what's going on. And so, you know, I think we try to emphasize a lot of this, um, observability of intermediate steps and even correcting intermediate steps, but, but there's a lot of experimentation around UX that I think needs to happen. Um, another big part, I think, is, is basically the planning ability of the underlying LLMs. Um, I think that's probably the biggest... I, I think when we see people building agents that work right now, it's often breaking it down into a bunch of smaller components and, and kind of, like, imparting their domain knowledge about how information should flow through these components. Um, because I think the LMs by themselves still aren't able to, to reason fully about how that should happen, and I think we see a few kind of, like, uh-... in- in- a lot of research is actually around this, I would say, in the academic space. Specifically, I think there are two different types of research papers around agents that we see. We see some around, like, planning for agents, so there's a bunch of papers that do kind of, like, an explicit planning step upfront. Um, and then there are, uh, other research papers that do a bunch around reflection, so like after it- after, uh, an agent does something, is this actually right? How can I kind of like, um, y- you know, improve upon that? And I think both of those are basically trying to get around the shortcomings of LLMs in that, in theory, they should do that automatically, right? Like, you shouldn't have to ask an LLM to plan or to s- think about whether what it's done is correct. It should know to do that, and then it can kind of like run in a cycle. But we see a lot of shortcomings there. Um, and so I think planning the ability of LLMs is- is- is a big one, and- and that'll get better over time. It... The last one is- is maybe a little bit more vague, but I think even just as builders, we're still figuring out the right ways how to make all these things work. What's the right information flow between all the different nodes, um, in order to get those nodes which are typically an LLM call to work? Do you want to do few-shot prompting? Do you wanna fine-tune models? Do you wanna just work on improving the instructions and the prompt? Um, and so I think there's a lot of, uh, how- how do you test those nodes? Uh, that's a big thing as well. How do you get confidence in your LLM systems and LLM agents? Um, and so I think there's a lot of workflow around that to- to kind of like be discovered and- and figured out.

   3. EGElad Gil7:20

      One thing that's sort of come up repeatedly rela- relative to agents has just been, like, memory. And so I wasn't sure how you think about memory and implementing that and what that should look like, and... 'cause it seems like there's a few different notions that people have been putting forward and I think it's super interesting, so I was just curious about your thinking on that.

   4. HCHarrison Chase7:35

      I also think it's super interesting. Um, I have a few thoughts here. Um, so I think there's maybe two types of memory, and they're- and they're related, but I'll- I'll draw some distinction between kind of like system level procedural memory and then like personalization type memory. Um, so system level memory, I mean more like what's the right way to use a tool? What's the right way to accomplish this objective independent of- of who exactly the person is and how I'm different than Sarah or something like that. Um, and then for the personalization bit, I think it's like, okay, I... You know, Harrison likes soccer and he likes basketball, and I should remember that when he asks questions. Um, and so I think there's- there's, uh, maybe slightly different ways that we see teams thinking about both of these. So on the procedural side, I think the main thing that we see people doing, um, and that we think is pretty effective is, uh, few-shot prompting and maybe fine-tuning for how to use, uh, for how to use tools, 'cause that's basically what it comes down to. What's the right way to use tools? What's the right way to plan? And we see few-shot examples being really, really impactful for that. And so that's something where it... And so there... I think there's this really interesting data flywheel of, like, monitoring your application, gathering good examples, um, and then- and then plugging those back into your application in the form of few-shot examples that we're pushing really heavily with LangSmith right now. And then the other side of it is just, like, personalization level memory. Um, and I think there's a few different ways to do this. Like, I think OpenAI implemented it in their chat, uh, in their Chat, uh, GPT where it... In the way I think it does it under the hood is it basically has functions that it can call to say like, "Remember this fact," or, "Delete this fact." Um, and so that's an- a really interesting, like, active, um, active loop that the agent is engaging in where it explicitly decides what it wants to remember and what it doesn't want to remember. I also think one- one thing that I'm bullish on is a more kind of like, uh, passive background, uh, process that kind of looks at conversations and almost like extracts insights. Um, and then you can use those insights in kind of like future conversations. And I think there's pros and cons to each, and I think it speaks to the memory. In general, I feel is like a field that's just like super, super nascent. Like, I don't... I- I actually, uh, am- am underwhelmed at the amount of like really interesting stuff that's going on there. Um, and so I think it's... You know? Bunch of different approaches. No- no kind of like overwhelmingly best solution.

   5. SGSarah Guo10:00

      Has the, um, sophistication shape type

4. 10:03 – 14:17

   Sophistication and limitations of AI app development

   1. SGSarah Guo10:03

      of application that you see people building with LangChain or just generally in the ecosystem dramatically changed over the last few months? I do think that there are more examples, kind of as Elad mentioned, of, um, agentic applications that are much more productive and more sophisticated like multi-step RAG systems with much more useful ranking. Like it's... Does that match with the patterns you're seeing or like what are you- what are you seeing that excites you the most that you think is most useful?

   2. HCHarrison Chase10:30

      That does generally match. I think LangChain from the beginning has always been focused on those types of applications. Um, and- and, uh, not only the open source but also LangSmith, the platform. So I think, you know, a lot of the emphasis that we put into like the testing and the observability is really focused on these like multi-step things. We've always been focused on those. Probably it's generally true in the market that there's been more, uh, uh, of a trend towards those. But from our perspective, we've always been focused on those, and so I think, you know, that hasn't been as dramatic. I think there has been like interesting, um, things within that that have emerged. Just calling out like a few things. Um, within RAG, I think we've seen really interesting and advanced query analysis start to come into play. So, uh, you know, you're not just passing the user question directly to an embedding model. You're maybe doing some analysis on it to figure out which- which retriever should I send it to or like what is the bit that I should search? Is there kind of like a explicit metadata filter? So some... And then so now retrieval is like a multi-step, uh, process and more there, um, and explicitly around query analysis. Um, few-shot prompting and that whole data flywheel I think we're starting to see come into play more on the agent side. Um, I kind of alluded to this earlier but I think, you know, um, the way that we've kind of thought about things is there's kind of like chains which are sequential steps. You're gonna do this and then you're gonna do this and then you're gonna do this, and you're always gonna do those in this exact sequence. And then, you know, last...... March or April or whenever Auto-GPT came out, and it was like, "We're literally just gonna run this in a for-loop and it's gonna be, you know, this autonomous agent." And I think the things that we see making it into production and, and, and informed, um, a lot of the development of LangGraph, um, is, or is something in the middle, where it's like this controlled state machine type thing. Um, and so we've seen a lot of that come out recently. And so, I'd, I'd maybe call out that as, like, one thing that we've really updated a lot of our beliefs on over the past few months.

   3. SGSarah Guo12:22

      Yeah, I think a combination of that and tree search and just, like, trying to be efficient with, like, your sampling at every step has shown, like, a lot of really interesting, uh, effective applications recently. And I think the, like, cognition as one example of, like, a surprisingly amazing agent has, has come out. Like, where else do you think agen- agentic applications will begin to work? Or that you've already seen?

   4. HCHarrison Chase12:48

      I think on the customer support side, that's a pretty obvious use case. I think Sierra, um, you know, had, has emerged there and is doing, is doing quite well there. Um, I think, yeah, the cognition demo was very impressive. I think they did a lot of things right. I think they really nailed the really interesting UX, um, and that was maybe one of the things that, that I was most excited about. Um, and then obviously it seems to work very well. (laughs) And so I, I don't know exactly what they're doing under the hood. Um, uh, but, but those type, like coding, coding problems in general, we see a lot of people working on. I think you, there's a really nice feedback loop that you can get by just like executing the code and seeing if it works, um, and you know, uh, as well as the fact that the people building it are developers and so they can, they can, uh, test it. Um, coding, customer support. Th- there's some interesting stuff around, like, recommend, like recommendation, um, chatbots almost. Um, so I draw a distinction between that and customer support, where with customer support you're maybe trying to explicitly kind of like resolve a ticket or something like that, and the, um, and the recommendation bit is a bit more focused on like a user's preferences and, and what they like. Um, and I think we've seen a few, uh, I think we've seen a few things emerge there. Um, but I'd say customer support and coding are the two. Klarna as well, you know, they came out and, and had a, a pretty good release.

   5. SGSarah Guo14:07

      One, um, pattern that I think is very popular, and I can't tell if it is real or transient, is whether or not companies will be able to

5. 14:17 – 17:10

   Switching between model APIs

   1. SGSarah Guo14:17

      switch between different, um, LM models, right? Whether it's a, you know, self-hosted, like, dedicated, um, uh, inference, uh, you know, instance for, for them, or if it's an actual API provider. But for any given application, take your prompts and go from, you know, um, Anthropic to Mistral to OpenAI to something else. Um, in, in reality it feels like, you know, the way an application, uh, responds is probably going to be sensitive to the fact that these LMs are actually going to predict differently. Like, what do you think about this? Can you, can you switch? Is that a real pattern?

   2. HCHarrison Chase14:56

      It's not as easy as it seems like it should be, and I think the main thing is that the prompts still need to be different, um, for each model. I do think, um, the prompts will probably start to converge in the sense that if you think the models are getting more and more intelligent, then, like, hopefully these small idiosyncratic (heetsies?) don't matter as much. Um, and as more and more model providers start supporting the same things, um, then that will make it easier. And what I mean by that is, you know, so many prompts for OpenAI which is, you know, the leading and most used one, use function calling. Um, and you know, up until some period ago, like, no other models did. And so you just, like, couldn't use those prompts at all. Um, but now like Mistral has function calling, and, and, and Google has function calling, and so I think they're a little bit more transferr- transferable there.

   3. SGSarah Guo15:42

      What else is on that list? There's function calling. There's visual input. Like, what else is gonna differentiate these, um, model APIs?

   4. HCHarrison Chase15:51

      Context windows one as well. So I think this gets to like, yeah, what's the right context that you can be passing? If it's longer, uh, you know, if, if that changes, then that changes, that doesn't, like that changes the whole architecture of your application. Um, modality is one.

   5. EGElad Gil16:07

      Prompt injection for safety.

   6. HCHarrison Chase16:10

      Yeah, I, I think that's interesting. Um, I think that's a real enterprise concern. I think a lot of the agents are still just figuring out how to make agents work. (laughs) This is a different axis almost, but to the point around like switching models, I do think we see a desire for this, especially when you start going to scale. Um, so I think it's like make something work with GPT-4, but then okay, you're rolling it out. Are you really, is that, like, you know, are you really gonna eat that much cost with GPT-4? Can you use GPT-3.5? Do you wanna fine tune? A- and so I think that's, that like, that transition is where we really start to see people, um, thinking about switching models. Um, there's definitely some switching models at the beginning like if you just wanna play around with different models and see their capabilities, but I think the most like pressing need to switch models happens when you go from prototype to, to scale. Cost and latency would be differentiators there as well.

   7. EGElad Gil17:02

      One thing you mentioned that I thought was really interesting is this context windows. And obviously Gemini launched with, um, a million token context window, and I was

6. 17:10 – 21:37

   Context windows, fine tuning and functionality

   1. EGElad Gil17:10

      just curious, um, how you think about context window versus RAG versus other aspects of the model and how all those things tie together and, you know, once we get to very long context windows in the tens of millions of tokens, like, does that really shift things radically or how does that change functionality? And so I was just curious since you've thought about how all these things piece together, um, I was just curious how you think about tho- those different factors and what they mean.

   2. HCHarrison Chase17:33

      Very good question that a lot of people are thinking about who are a lot smarter than me. I think, um, I mean, a few thoughts. I think like longer context windows definitely make like single shot things much more realistic. Um, like extraction of elements in a long PDF. You can do that one shot. Um, RAG over a single long PDF or like five long PDFs. Okay, cool. You can do that, you can do that one shot. There are, I think, um...There are definitely things that scale that don't fit, um, you know, into a single, uh, uh, context window. There are also things where it requires iterations. You need to, like, decide what to do, interact with the environment, get that back. So this whole idea of chaining, um, and agents, I don't, like ... Th- that's n- that's less around context windows and more around interacting with the environment and getting feedback. And, and so I don't think that's going anywhere. Um, I think with respects to RAG in particular, 'cause I think that's where it often comes up, like, you know, did this kill RAG? Um, I think there's a few things. Actually, just today, one of our team members, Lance Martin ... There's that, like, ev- everyone's doing the needle in the haystack thing, and now all these models are, like, green across the board for whatever reason. They've all figured it out. Um, but I think, like, that, that actually really doesn't reflect a lot of RAG use cases in, in my opinion 'cause, like, that's ... The needle in the haystack is like, okay, given this long context, can I find a single information point? But oftentimes, RAG is about seeing multiple information points and then reasoning over them. And so I think with the benchmark he released is exactly that. Like, as you increase the number of, of needles, um, you know, performance goes down as you might expect, and then also when you ask it to reason rather than just retrieve, the performance drops as well. And so there ... I think there's more work to be done there. And then I think another thing is, uh, just around the ingestion for RAG in the in- in the indexing process. Like, a lot of attention has been paid to, like, um, s- text splitting and chunking and, and, and all of that, and I don't know exactly how that will change. Like, will you still do that but you now just retrieve the whole document? Like, we have a concept in LangChain of, like, a parent document retriever which basically creates multiple vectors for, for each document. So maybe you just do that. Maybe you still ... Maybe you chunk it up into larger chunks and just retrieve those larger chunks, maybe use a, a traditional search engine like Elasticsearch or something. I'm not ... I'm, I'm not sure. That's probably the place I have the least confidence in.

   3. EGElad Gil19:53

      The one other area that I see a lot of people talking about and I see a few people actually doing, uh, is fine-tuning. And, um, to some extent I think that's because with fine-tunes you lose generalizability and so people just start focusing on prompt engineering or other ways to effectively get the same performance without the actual fine-tune. But it's something that feels very, um, au courant and people talk about it a lot and people talk about doing it a lot. Um, you probably have a great perspective since you see so many different types of customers. Are, are you seeing a lot of fine-tuning happening in the wild, and if so are there specific common applications or use cases for it?

   4. HCHarrison Chase20:24

      We see people experimenting with it. I think the only real place where they're doing it is when they've reached, like, really critical scale, um, which I still don't think is that many applications to, to date. Um, I think there's a lot of difficulties with it. Um, one's, like, gathering the dataset for it. And so I, I think a lot of the things we have in LangSmith tackle a lot of these issues, but, like, gathering the dataset for it, um, so, like, having that data visibility and starting to curate that dataset. Um, e- evaluating the fine-tuned model, um, so, like, evaluation and testing is a, is a huge pain point there that we're trying to tackle in a few ways. The third is just like, yeah, back to this point of people are still just, like, experimenting so rapidly, it's much harder to change a fine-tuned model than it is to change a prompt or even change few-shot examples. And so I think we're seeing more and more people use few-shot examples, um, but not a ton graduating to the fine-tuning just because, yeah, I, I think, uh, much harder to just, like, iterate quickly on.

   5. SGSarah Guo21:21

      In terms of other major changes in the landscape, it's been a, it's been a big year. The first commit to LangChain I think was in October of '22 which is, like, when I launched, um, Conviction as a fund as well. Uh, at that time, we didn't have Llama 2, we didn't have Mistral.

7. 21:37 – 23:53

   Evolution of AI open source environment

   1. SGSarah Guo21:37

      There were not, um, nearly as many open source models with, um, what people would consider to be more useful reasoning ability. Has that changed in terms of, uh, like, what you see, um, application developers do with LangChain?

   2. EGElad Gil21:53

      Gemini too.

   3. SGSarah Guo21:54

      Oh, and Gemini, yeah.

   4. HCHarrison Chase21:55

      Fun story about that. The original models that we launched with OpenAI actually deprecated (laughs) like a month ago so the, like, actual original LangChain you can't run 'cause the models don't exist anymore. (laughs) Um, but yeah. (laughs) Um, like, there's ... I think we see increasingly interest in open source, but the reasoning abilities are still just, like, lagging behind Claude 3 or GPT-4. Um, and I think, like, for a lot of the applications that ... Aga- it kind ... It probably depends on the types of applications that you're building, but a lot of the applications that LangChain is focused on with this kind of, like, reasoning aspect, those are just so crucial. Um, and I don't think we see s- super compelling, um ... I don't, I still don't think we see super compelling reasoning abilities in the open source models. And maybe that's one of my hot takes, but I think for a lot of the LangChain apps, the open source models maybe don't live up to a lot of kind of, like, the, the Twitter hype or Twitter excitement, at least not yet.

   5. SGSarah Guo22:56

      Zooming out, like, you have a really broad view. Like, what do you feel like that no one is working on that's gonna enable better applications that should be?

   6. HCHarrison Chase23:05

      I think the most exciting stuff is at the application and UX layer right now. I think that's th- where the most exciting stuff is there. One of the, uh ... I don't know if this is, uh, this, this is maybe ... This is in more the capabilities side-ish, but, like, memory I think is super interesting, especially, like, personalized long-term memory. Um, I don't know if it ... I don't know if it's necessarily tooling so much that needs to be built there as it's just, like, a- an application and a UX that's really focused on that. Um, and, and, you know, i- if I, if I wasn't doing LangChain, if I was starting a company right now, I'd probably start something at the application layer, and it would probably be something that really takes advantage of, like, long-term memory.

   7. EGElad Gil23:45

      I guess at the, the high level similarly is there anything that you view as, like, a major prediction or things that'll change over the next year that nobody's really

8. 23:53 – 27:31

   The next big breakthroughs

   1. EGElad Gil23:53

      paying as much attention to?

   2. HCHarrison Chase23:54

      Memory is a big interest of, of, of ours, um, and so I hope that we'll have some kind of, like, breakthroughs there. I think a lot of the ... Specifically around, yeah, learning from interactions, incorporating that back in at a user level. Um, in a similar vein, uh-... also this, uh, type of more like system-level memory I think is really interesting and building up, building towards this idea of almost like continual learning. So there's, you know, like can you learn from your interactions? And you can do that in a variety of different ways. This may just be where we sit in the ecosystem, but one exciting, um, and probably under-talked about way is just the idea of, of building up few-shot example datasets and really using those. I think it's much faster and cheaper than fine-tuning models. Um, it's easier to do than trying to, like, programmatically change the prompt in some way. Like that's still kind of like a, a, a, a bit of a art. Um, and so yeah. Continual le- towards continual learning with few-shot examples is, is maybe one like really interesting area that, that we're excited about.

   3. SGSarah Guo24:54

      Can you help our listeners ex- uh, like just imagine like a little bit more viscerally like what, um, type of application experience that would enable? Like, you know, a consumer application or a business app- application of what that type of continuous learning would allow you to do?

   4. HCHarrison Chase25:10

      Yeah, absolutely. I think at a high level it would basically allow the application to automatically get better over time, and it could get better in the sense that it's just more accurate. So you know, it's, it maybe, you know, at first does a mistake. You then like tell it that it made a mistake, and it automatically kind of like incorporates that either as a few-shot example or update to a prompt, but it starts learning from its, its mistakes and its successes as well, right? There's a really cool project called D-S-P-Y, or D-S-Py. I don't know how to pronounce it. Um, but it's out of Stanford.

   5. SGSarah Guo25:39

      I say dis-pee.

   6. HCHarrison Chase25:40

      Oh, no. (laughs)

   7. SGSarah Guo25:41

      (laughs)

   8. HCHarrison Chase25:41

      There's, there's three ways now. (laughs)

   9. EGElad Gil25:43

      I say D-Spy. No, I'm just kidding.

   10. HCHarrison Chase25:45

       (laughs)

   11. SGSarah Guo25:47

       (laughs)

   12. HCHarrison Chase25:47

       So i- and I think that actually tackles v- like I actually see a lot of similarities between that and LangChain, LangSmith in some way. And I think it's all towards this idea of like, so, so, so D-S-P-Y or dis-pee or, or whatever, um, is basically this idea of like optimization. You have kind of like inputs/outputs. You then have your application which, uh, they similarly think as, as like multiple steps, and you basically, uh, you basically optimize your, your application through a variety of different ways. The main one of which I would say is probably few-shot examples, although O- um, we'll probably do a webinar with Omar and he can correct me if I'm wrong. Um, and I think the idea of like continual learning is basically doing that optimization but in an online manner, where you're feed- you don't have like ground truth necessarily, but you get feedback from the environment, thumbs up/thumbs down if, if things are good. And so I think, yeah, that kind of like optimization loop, whether offline or online, is really, really e- exciting. And I think a similar thing could maybe ... I think you can think of like personalization also as like what this would look like to end users in, in maybe like consumer-facing apps. So you start with like a generic application that does the same thing for everyone, but then it maybe learns to, to search the web differently for, for me and Elad or something like that. Um, and so I think that's like concretely how it could, it could manifest.

   13. EGElad Gil27:06

       Cool. Thanks so much for doing this. Um, it's obviously a pleasure to have you on.

   14. HCHarrison Chase27:10

       No, thank you guys.

   15. SGSarah Guo27:10

       Good to see you. (instrumental music) Find us on Twitter @nopriorspod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.

Episode duration: 27:31