No Priors Ep. 9 | With Perplexity AI’s Aravind Srinivas and Denis Yarats

1. 0:00 – 1:46

   Introduction

   1. SGSarah Guo0:00

      (music plays) Arvin and Denis, welcome to the podcast.

   2. ASAravind Srinivas0:08

      Thank you for having us here.

   3. DYDenis Yarats0:09

      Thanks.

   4. SGSarah Guo0:10

      Ah, thanks so much for joining. Um, so the two of you, alongside, um, Andy Chodakowski created Perplexity around August or so of 2022. Um, Arvin, do you wanna give us a little bit of a sense of why you started this company and what the core thesis of Perplexity is?

   5. ASAravind Srinivas0:26

      Yeah, sure. Actually, Elad, you're- you're our first ever investor who offered to invest in us so that was the founding days. Um, in fact, I remember the first ever idea we talked about in Noe Valley where we were sitting in the open space opposite Marta and I was telling you how it would be cool to have a visual search engine, the only way to disrupt Google was to not do text-based search but to actually do it from camera pixels, and you were like, "Uh, this is not gonna work."

   6. SGSarah Guo0:56

      (laughs)

   7. ASAravind Srinivas0:56

      "You need to think about distribution." And search was always the core motivation for Per- like, for me and Denis, and there's many others at the company. We were just bouncing around ideas then I was still at OpenAI around the time and then we lef- uh, left, and Denis also was still at Meta and he also left, and Andy came in to help us sort of, uh, inc-incorporate and get the company rolling. So that's sort of how the company started. The space of LLMs was exciting, generative- generative models were really exciting, and, uh, in general, we were motivated about search, whether it be gen- general search or vertical search, and we were bouncing around several different ideas. One of the ideas that you gave us was the working on text-to-SQL, uh, and like we were pretty excited about that and

2. 1:46 – 5:46

   How Perplexity AI iterates quickly and how the company has changed over time

   1. ASAravind Srinivas1:46

      started prototyping ideas around that. And I think Denis also like was hacking with us on building like a Jupyter Notebook extension with like, uh, Copilot for every cell, um, and then we were trying it with, uh, SQL around databases. But it's all like a bunch of nonlinear pathways to eventually get to where we are right now.

   2. DYDenis Yarats2:07

      Yeah, absolutely. And, uh, hopefully I caveated whatever feedback I gave with, I'm probably wrong, but since, uh, I think I'm often wrong on (laughs) -

   3. ASAravind Srinivas2:14

      No, I think, I think-

   4. DYDenis Yarats2:15

      ... a hundred percent. (laughs)

   5. ASAravind Srinivas2:15

      ... whatever you said, whatever you said still applies. Uh, search is tremendously a distribution game as much as a technology game.

   6. DYDenis Yarats2:22

      Yeah. I think one of the really impressive things about Perplexity is the rate of iteration. And to your point you've- you've gone through things like, um, text-to-SQL, Copilot for the next gen data stack, and I've always been impressed by how rapidly you've just been able to point in a direction, iterate really fast, prototype something, see if it's working, and then move on to the next thing. And, uh, to your- to your point, you always had search in the back of your mind. I remember even as you were prototyping these things you were talking about indexing aspects of Twitter or other sort of data feeds and then providing search on top of them. Um, how did you end up building a- a team that can iterate that rapidly as well as a culture of fast iteration? Like, are there specific things that- that you all do as a team to help- help reinforce that?

   7. ASAravind Srinivas3:04

      Yeah. I'll- I'll take the first part of this question and then also let Denis answer this because he's a big part of why this is happening. Uh, we both are basically from an academic background, so in general, the culture in academia is to you have hundreds of ideas and you just need to try them out pretty quickly, uh, run a lot of experiments really quickly and get the results and iterate. So we come from that background, both of us, so- so that's not really new to us. It's just that when it comes to trying out new products it- it- it's not just a result you get from running an experiment, you actually have to go to users and make them use it and, uh, talk to companies or customers or potential, you know, people in a company who will be using your product and get feedback. So there's that aspect of operational work that needs to be done to get results for your experiments, and there's this aspect of quickly doing the engineering to get it to a state where you can show it to people. So both of these things had to come together, and that's why the company exists. And Denis is incredibly good at engineering and recruiting, um, and so we found our other co-founder, Johnny Ho, who's als- like a big reason why Perplexity operates this fast, and full credit to Denis for helping recruit him. He was the world number one at competitive programming. Uh, it's like being Magnus Carlsen of programming. He got pretty excited about all the, uh, stuff that Denis was showing him, so h- help- it's- it's sort of like few people help you accelerate a lot and Joh- Johnny is, like, one big reason for that. And Denis sort of continued doing that by getting more and more people in the fold.

   8. DYDenis Yarats4:38

      Uh, are there specific things, Denis, that you, um, look for when you're hiring or when you're screening for great technical talent? Because I mean, one- the way one person put it to me is that you have a team that can build almost anything really fast. And so I'm curious, like, are there heuristics, are there places that you look for people, like how do you think about hiring? Yeah, that's a- that's a good question. I'm kind of like looking for the people, um, who maybe very- have like this general interest in, like, applying this emerging technology like LLM, right? So like a lot of people hear about it, maybe they don't have a specific background but they're like very curious to learn about it and they want to put like a lot of energy into it, so I feel like that's my number one indicator. So like personally I would rather like get somebody who has this burning desire to work on these things rather than somebody who ha- has- already has a lot of experience and not going to put a lot of- or like as much effort as other people. I think a- another concept that we still use until this day even though it's like takes a lot of time it's we have kind of like this trial periods, I would say, um, so we basically get to know a person a little bit better and once we see there is like good

3. 5:46 – 10:43

   Approach to hiring and building a fast-paced team

   1. DYDenis Yarats5:46

      chance we might hire this person we then ask person to sort of like work with us for like some time. We- we kind of like intend to keep our comp- at least like, uh, number of employees very small, I feel like it's much faster to operate. And because of that each hiring decision is very important, you know, and we want to like optimize for the best people, uh, and that's why we kind of like run this trial process where...... you know, we basically making very sure that we want to work with this person. So I think that- that's been helpful.

   2. EGElad Gil6:17

      Is there anywhere you've been, um, surprised as you've made hires where y- y- you feel like, um, you know, the signal was wrong or the trial surprised you?

   3. DYDenis Yarats6:26

      Um, yeah, there has been a few exceptions, uh, obviously. I think that's the part of it. But I feel like it's definitely, at least when comparing to my prior experience, it's definitely a smaller chance where you're gonna get surprised. Like, you know, normal, normal interviews that, you know, big companies run, you know, you have like four or five meetings, like for 45 minutes each and then you basically make decision after that. Um, you know, uh, sometimes it works, sometimes it doesn't, but I feel like the, the way we do things, it's gives us like much more confidence that we're going to make right decision. And I- and I think it's obviously it's very useful for us to get the signal, but I also feel like for candidates it also makes sense to, um... It's useful to make better decision as well, right? So they can understand do they want to work on things like we do? Do they want to work with the pace that we do? I think like one important thing for us and like many candidates sort of like don't wanna maybe do this is, uh, the energy we- we put in- into Perplexity. Um, it's kind of like work-life balance, maybe not the ideal, uh, but uh, you know, but that's the only way to sort of like beat competition and sort of like iterate very fast and do great things. So, um, so that's why we kind of like need to have this alignment at the beginning.

   4. ASAravind Srinivas7:49

      Yeah, if you don't have a clear idea of like, you know, which product you're going to build or which market you're going after, uh, and you still want to be giving- giving yourself a shot at success, uh, iteration speed is the only thing that you can hope for. And so we decided, okay, like until we actually figure out our business and product, we- we'll make sure that this is non-negotiable, like the- the people should iterate really quickly with us.

   5. SGSarah Guo8:15

      Yeah, I feel like the only real advantage that a startup has relative to incumbent is speed, right? Because incumbent has more people, they have more money, they have more distribution, they have more product. And so really the only thing you have is speed. Um, in the early days of mobile, you know, I was working at Google and I- I really helped get that team up and running and then I started a company that Twitter bought. And at Twitter there was this weird meme internally that the only people they should hire for mobile were people with prior mobile experience, which I thought was a really dumb idea, right? It was more you just find great engineers and they'll figure out how to write for iOS (laughs) and stuff like that. How do you think about that in the, in the ML world? Because there is this sort of dogma right now, I feel, that people believe that only somebody who's trained on LLM before can work on an LLM-centric company or things like that. So I'm just sort of curious, uh, you know, how you think about that?

   6. ASAravind Srinivas9:01

      Yeah, actually, uh, my thinking about this was already pretty clear, uh, because of having seen how OpenAI went about this. Uh, if you look at the people who did GPT-3, eventually they went on to start Anthropic. All of them are basically from physics background. The CEO, Dario, is a physics PhD and Jared Kaplan is a physics professor. He wrote the scaling loss paper. Uh, so they basically, OpenAI really succeeded in bringing these extremely talented physics people who wanted into machine learning and- and they came into this sort of language models and scaling and that paid- paid off well for them. And similarly, their whole engineering team, if you look at an engineering team, it's just Dropbox and Stripe people. All like software engineers, solid people who can build infrastructure, front end, and now they're doing AI work. So it's always been clear to me that, uh, in order to work in AI, both research and product, you do not need to already have been in AI. And that's being shown clearly with our, uh, with- with Johnny Ho, our co-founder. He was not in AI. He was a competitive programmer, a trader, he'd worked at Quora for a year, but uh, he's as good as anybody can get in picking up new things. Uh, so the other thing also is that LLMs are sort of in this weird territory where the people who use the LMS for building stuff understand it better than the people who actually did gradient descent and trained these models. Like, uh, you- you could find a PhD student at Stanford or Berkeley whom, who would know a lot about how to train the model, but, uh, they might not be the best person to build a product with it.

   7. DYDenis Yarats10:41

      Yeah, I want to like quickly add, uh, to this point.

4. 10:43 – 14:01

   Why you don’t need AI pedigree to transition to work or research AI

   1. DYDenis Yarats10:43

      So I was, um, at early days at, uh, Facebook AI Research in Menlo Park, the office, right? And at that time... And that- that was honestly one of my reason to do PhD, is there was like kind of this like very exclusive culture. So if you, like you, if you don't have a PhD you're not going to be a research scientist and, uh, you know, kind of like... I didn't like that too much so that's why I decided to do PhD later on but, uh, turns out from my experience, the best research scientists also very good engineers, like very good engineers. And, um, and we've noticed, like through D- DeepMind and OpenAI, it's just like the- the- the companies that made the m- th- the most progress over like last six years or five years were people... Were, were companies where there are like extremely good engineers. And it's kind of like I didn't like from the beginning this, uh, view from like academics that's just like if you're an engineer, you're probably not, you know, either like smart enough or you're not going to do, do like great things but turns out, it's actually the other way around. So, um, that was, uh, also I think like motivation and I feel like you don't need to be, you know, this like very impressive like academic with PhD to, to, to do great things.

   2. ASAravind Srinivas11:57

      Yeah, this also goes back to the thing Denis said earlier, that you want to find the people who really want to get into AI, uh, rather than who are already in AI. Uh, and every big company, ev- every company that's gotten big has done this in their early days, including Google, like they got a lot of systems people, compil- Jeff Dean was a compilers and systems person and then they-... they got all... And Urs Horzler is like a systems profes- uh, professor. They got all these amazing people and they told them, "Hey, you know, guys, like, we're, we're having the most interesting computer science problems to solve here, uh, and, and, and we can scale it up, so come work on search." So it's not like you have to go... There are a few people they hired from information retrieval or search background, but most of the celebrity researchers that they have right now are just, uh, people who wanted to get into that space for the first time.

   3. SGSarah Guo12:47

      It's funny, because now the, now the desired pedigree isn't the PhD from whatever, uh, you know, academic lab, because, um, the, uh, the industry labs fill- filled with physicists and engineers and people not necessarily from the domain made the most progress. And so, if you want somebody who's worked on... I, I hear the argument, you don't actually need it, like you want people who are really smart and motivated. But if, uh, but a lot of companies are looking for somebody who has, you know, experience with X billion parameter training runs, and those people come from now, OpenAI and DeepMind and such. So it's quite, quite funny very quickly how, uh, the pedigree has changed. What's been the area of steepest learning curve with both of you coming from these, uh, like research engineering backgrounds?

   4. ASAravind Srinivas13:33

      For me, it's like how to run company. That's mostly, uh, what I'm doing here. I'm not doing much, uh, with core engineering. Dennis does that. So that was not easy at all, uh, but I've had the opportunity to learn from many good people, including Elad here. So it's not... If, if at all there is an easy way to do it, it's like getting advice, rapid advice from people. Uh, the other thing is also like when you're making a mistake, like being super

5. 14:01 – 16:50

   Challenges when transitioning from AI research to running a company as CEO & CTO

   1. ASAravind Srinivas14:01

      brutally honest with yourself and, um, listening to feedback and quickly course correcting it. Uh, that was also like very new to me.

   2. DYDenis Yarats14:10

      Yeah. Um, I guess for me probably it wa- it was, you know, building a team, you know, kind of like organizing everything in terms of, like, who do, who does what, uh, you know, how to prioritize things, what to work on. I think like being a small team, it's even more important to identify this one thing that you want to work on and like put all, all, all your focus on it. Early on, we were like at this... sometimes made these mistakes that we're trying to go after like several things. That's why (laughs) , you know, uh, even though we're like six months or like seven months old company, we actually built so many things. We had like, you know, Twitter, we had like, uh, Salesforce, uh, integration, we have like HubSpot integration, like many other things, uh, that we never like released. One of the interesting thing is just like you, you have to have very precise, uh, focus and, and just like go after it.

   3. ASAravind Srinivas15:02

      Yeah, also as leader, uh, if you want to lead the company, uh, and if you cannot do everything, the people will lose faith in you, right? They think you, you don't really have any clarity and that's why you're making them do a... one new thing every week. And so you, you have more responsibility to sort of get it right and think more clearly. And even if you're wrong, like do one thing wrong and at a time and course correct rather than doing five things at once and, um, seeing like which one wins. It's, it's difficult for us coming from the academic background, uh, for this particular thing, because in academia, you're basically taught to hedge. You have like one first author project and like three or four co-author projects, and like one of them might become a big hit and might change your career. Whereas that's not how you should do startups. Like, you really have to focus and iterate multiple times on o- one thing or a few things. So that took us a while to quickly learn.

   4. SGSarah Guo15:55

      Yeah, makes a lot of sense. I think everybody goes through very similar, um, paths as they start a company. Um, you know, one thing that I think is really interesting right now is we're, we're at this point where consumers are really starting to wake up to how machine learning and AI is changing search. And to your point, you all were thinking about this actually before ChatGPT and before the Sydney News and before Bing integrated all these things, so you were quite early to realizing that, you know, this is gonna be a really core piece of search. Uh, Perplexity's mission, I, I believe, is to create the world's most trusted information service. Um, how do you think about important product points around factual accuracy, um, bias in presenting aggregated information for users and things like that?

   5. ASAravind Srinivas16:34

      Yeah, I guess you're, you're also from a PhD background, right? So when you write your first paper, the thing your advisors teach you is, uh, you only have to write things that you can actually cite. Anything else that you write in the paper is your opinion, not, not a scientific fact.

6. 16:50 – 19:33

   Why Perplexity only shows answers it can cite

   1. ASAravind Srinivas16:50

      And so that sort of stuck with us pretty closely, um, and that's sort of why we did the first version where it's citation powered search. So, uh, for factual accuracy, our first step towards that was making sure you can only say stuff that you can cite. Uh, this is a pretty subtle point here. It's not just that we want to retrofit citations into a chatbot. Uh, that's not what Perplexity is. In fact, it, it's more like a citation first service, that it'll never say anything that it cannot cite. So if you, if, if people have tried to play with it as if it's like ChatGPT where like, tell me who are you or like things like that. And even for those questions, it would still go to a search engine, pull up stuff and come back with an answer. It's not gonna say, "I'm a- I'm Perplexity, I'm like a bot design. How are you doing?" Or something like that. So, uh, this is because of our obsession about factual accuracy. Like even if it doesn't have a personality or a, or a character in it, uh, we don't, we don't care. We want... we only care about the other thing, which is obsession about truth and, uh, accuracy. Yet, the second point you mentioned about aggregation of things, wh- when you mash up multiple sources together, you might end up hallucinating. Like for example, if there's multiple people with the name Elad Gil, and like one of them is a venture capitalist and some oth- some other is like a doctor. Or let's say even if it goes to your own LinkedIn, uh, and things like, oh, the Elad Gil who did a PhD in biology is different Elad Gil from the venture capitalist, because someone might think that, right? It's pretty unusual background. Uh-... and then, so then that it might end up coming up with some entertaining or funny, uh, summaries that collate different sources together. We still haven't thought of a proper fix to this, uh, but we, uh, one thing is obviously as language models keep getting better, they're gonna understand these things, these subtleties even better, and we are already seeing that. Uh, and second thing is we are giving users the power to remove sources that they think are irrelevant, uh, just like how you can curate sources in Wikipedia. So we, uh, we- we- we've sort of working towards this accuracy and the bias issues, uh, once, st- step by step at a time, but I, I feel like it'll take more iterations to get this truly correct. And I also don't think one LLM will just magically solve this problem. Uh, you need to build an end-to-end platform where users can cor- uh, correct the mistakes of an LLM, and that also means you need to design the platform where the incentive is right for the user, because this could also be used in the other way where users can use it to hide information. So, we haven't really thought through all these issues thoroughly, but we are committed to sort of figuring these things out over time.

   2. SGSarah Guo19:25

      That makes sense. I guess in addition to that, uh, or maybe related, you've done an impressive amount of research in reinforcement learning. What- what's unique about the way that Perplexity uses reinforcement

7. 19:33 – 20:49

   How Perplexity approaches reinforcement learning

   1. SGSarah Guo19:33

      learning and how does it tie into these plans?

   2. ASAravind Srinivas19:35

      We like RLHF, like, Reimforcement Learning from Human Feedback where we use the contractors to u- we collect feedback from the users on whether they like the summaries, the completions or not, and, like, we use contractors to use, like- like, do the summ- ratings themselves. And these days even LLMs can be prompted to do the work that contractors do. Uh, Anthropics wrote a paper on that. So, all these things are getting really, uh, very efficient to do. So that's sort of how we have been thinking about reinforcement learning right now. But, uh, we haven't gone beyond that to think of, like, agents and browsers and things like that. Uh, we'll- we'll probably focus more on the first part for the next at least six months to a year.

   3. DYDenis Yarats20:16

      Uh, one add on this aspect, uh, you know, full-blown, like, RLHF is, you know, definitely something we can now look into that. But there is, like, several many steps that you can have in between that significantly c- can increase your quality. So for example, I mean, uh, like, even using something like a rejection sampling, you know, like, discriminator on top, it's kind of you wanna like shift, you know, maybe you have a several samples from, you know, uh, your LLM and then you can, can rank some of them using different model,

8. 20:49 – 23:05

   Trustworthiness and if an answer engine needs a personality

   1. DYDenis Yarats20:49

      pick only those. It is in a sense kind of like one step of reinforcement learning, but, uh, uh, it helps a lot. It's very effective.

   2. SGSarah Guo20:56

      You, you guys have talked about how trustworthiness is more important to you than, um, like, I don't know, personality, the ability to play with a bot. Like, do you believe in chat as an interface? Like, where's the line between chat and search given Perplexity does support, for example, like follow-ups and things that are more conversational?

   3. ASAravind Srinivas21:14

      Yeah. We think chat UI is the, is the future. People are using it pretty heavily. At the same time, uh, if you can try to get the answer right in the first attempt, you should, right? Like, you have a responsibility to save people's time. Uh, it's not like Google doesn't do some kind of chat implicitly. Like, if you go to Google, you always get related questions, follow-ups, people also ask for and things like that, it's just sort of implicitly making you click on it and you get a follow-up question that you sort of get, like, get a chat, like, e- experience without the chat UI. So, I feel like it's- it's more, like, whether it makes sense for the particular experience you want to provide for your end user. And in our case it does. Often, you do not get the answer you want in the first attempt, and you shouldn't feel the burden to get it also. Like, sometimes you could ask a question by just asking for the keyword and then you might realize, like, "Okay, this is actually what I wanted to ask for." I've seen this in live when people use, uh, Perplexity but they just couldn't know how to phrase this question the first time, so they used multiple attempts to get to the right question. So there are things like that where as the questions get more complex, the chat UI makes a lot more sense than Google's UI. But for, like, obvious questions like if you just wanted to know whether it's, uh, gonna rain in Bay Area the next one week, uh, why do you need to keep asking more, right? Like, you just get the answer the first attempt. And we want to support both these experiences in Perplexity.

   4. SGSarah Guo22:41

      Uh, m- more, more broadly, I'll ask a dangerous question, but what do you, what do you think the future of search looks like, right? Five years plus out. Do we get... Do we still have monolithic horizontal providers if the players change? Um, do we get more embedded apps, like contextual search as a feature in different places? Are there agents that do things for you? Like, what do you think it looks like?

   5. ASAravind Srinivas23:02

      I think there's this phrase that's becoming popular,

9. 23:05 – 26:38

   Why answer engines will become their own market segment

   1. ASAravind Srinivas23:05

      uh, I think S- Karpathy was the first one who tweeted it and Satya Nadella is also using it. It's called an answer engine instead of a search engine that directly tries to answer your question instead of providing you a bunch of links or just snippets from the first link. Uh, so we believe in that. Like, Perplexity is the first conversational answer engine. Uh, like truly the first. I think nobody built it before us. I believe answer engines will become its own segment, market segment. Like, just like you have a default search engine, there- there'll be a default answer engine over time if these things really work. And the burden of, like, getting ranking in search right will, uh, reduce in the sense answer engines can do more heavy lifting than search engines. As these things get really good and, like, m- very fewer hallucinations, um, and even- even if they do hallucinate, people can still go and click on these links, they will eventually prefer this experience over the regular, like, 10 links or 20 links UI that Google has. So that- that's something I'm pretty confident about. Uh, and- and I think the sort of asking follow-up questions will become more of the norm. The number of queries in Perplexity that go to at least one follow-up has been increasing ever since we released the chat UI, so that- that'll keep going up. People will get used to this sort of experience where they're encouraged to ask one more question.... uh, and they're okay with not getting the answer right away. So that'll, that'll happen. Uh, the third thing is like actions. Like People will be more deliberate in what they search for. Um, and, and, and try to execute an action out on top of the search results they consumed. So that's definitely likely to happen. It's already happening. If you go to Google and book a flight, you just like fly from SF to Seattle, you just directly click on the book button. So that's gonna happen more frequently in the chat UI too. So you... Uh, and this will become an assistant more than just a search or answer engine. And I, I also think the fourth thing is there will be some sort of like, uh, much fewer traffic to the actual content site. Like very few links need to be consumed. Uh, in Perplexity, in fact, like we only... We don't even cite more than five links. It's, it's a deliberate decision. Like we could have... A lot of people asked us for... "Can you add, can you please add like 10 links or 20 links? Can you just show all the links together?" I, you put the summary at the top, but you also put all the 10 links, the usual, I want both. Uh, it's a, it's a decision just, we just made that, no, no, no, that's not the right experience for you. You actually need to feel the difference here. So we only made only three to four links. Or like in fact the first version shipped with like three to four links, I think, and like 50 word summaries, and we expanded the numb- summary more. So I think we, we just have fewer tabs open. Uh, we only open the tabs that we really need so that lo- all this sort of behavioral change in the consumer is likely to happen. Whether it be through us or Google itself doing these changes, it's unclear. But this, these, this is where it looks like it's trending towards.

   2. SGSarah Guo26:02

      When, when do you think we're gonna have that transition from, uh, almost like a, a pull versus push model, right? Because I think right now you go and you ask for certain information and you may ask for it repetitively, you know, if it's checking the weather every day, uh, versus a world where you have agents that are effectively understanding your intent ongoing and providing you with an information in a sort of a push-based way. How far away of a, of a world is that? Is that a year away, five years away, 10 years away?

   3. ASAravind Srinivas26:28

      I feel like we can do it now. Uh, in fact, like Google Now was an attempt to do that. If you keep checking for scores of a favorite football club, it'll automatically

10. 26:38 – 30:20

    Implications of “the era of fewer clicks” on publishers and advertisers

    1. ASAravind Srinivas26:38

       give you a push notification for their next latest match that you might have missed. Uh, they're, they tried some of these things already, so I feel like we can do these things even better now with language models. So yeah, it'll be... I think it'll happen in a year, more than five years.

    2. SGSarah Guo26:53

       And, and do you think it's gonna be fragmented in terms of each? And I know this is all uncertain, right? It's kind of predicting the future is never correct. Um, but I'm just curious how you think about, you know, is, are there gonna... Is there gonna be like a agent for your, um, Google Drive or, or an agent for your GitHub or an agent for your email? Or is it just sort of consolidated eventually into one central service?

    3. DYDenis Yarats27:13

       I mean, it's, it, it's getting pretty clear, you know, with Google, um, the first few results, like at least like my, my pattern of using Google, I see like first two, three results I basically skip most of the time because it's, uh, SEO or like some ads. Um, and that's like not ideal experience obviously for the user. And it's also, uh, kind of like why we believe might... It might be very hard for Google to fully launch this system like answer engine because it just like breaks monetization strategy for them. As, as Arvin mentioned, you know, there's gonna be like fewer clicks and Google wants you to click on, on those things, right? So that's how they make money. Basically, I think there has to be this like new paradigm where you kind of like, you get your answer quicker, but then maybe you can monetize it better on it. Maybe now, like you can help user to make decision faster and then you can show like much more, uh, targeted and, uh, you know, um, accurate ad. So then, you know, like if user clicks it, it's gonna be... Maybe they're gonna buy something or, or whatever. Um, so there is gonna be a few, few clicks, but each click is gonna be more expensive. And I feel like overall user experience has to be better because you just, you just get what you want. You don't have to do any extra effort.

    4. SGSarah Guo28:33

       In this, um, era of fewer clicks, if you, if the summaries and chat that's just giving you the answer, how does that impact the relationship of search or answer engines with publishers? Does that remove the incentive to publish information on the internet? Does it become more adversarial? How do you guys think about that?

    5. ASAravind Srinivas28:51

       Uh, probably not. I feel like... So it's sort of like whether you cite a paper or not sort of thing, right? Like you would cite a paper if the paper was really good. It's actually gonna bring back the whole concept of page rank even more. The, by the way, the concept of page rank was inspired by academic citations, uh, from what I've read, uh, like a very important paper tends to get cited. So when you're in this sort of citation-based search interface now, the better content a, a publisher has, the more likely it'll get cited by an LLM unless humans figure out answer engine optimization or LLM optimization, which is, which I... Hopefully they don't invest effort into. But in general, um, it's unlikely, uh, that it's as easy as SEO with just keywords because LLMs are gonna be much smarter in understanding relevance to a query. So I think it's just gonna incentivize people to publish higher quality content in order to get cited by an LLM powered answer. Like Substack or things like that try to do. Like you, you, you, you wanna own your content and you publish it and make sure it's high quality and you have your own like set of subscribers. Um, I've... So, so people put a lot of effort into that, more than writing tweets. So something like that is likely to happen with this interface too. Uh, but it's unclear exactly how to make all this monetize at scale, like, you know, the click-based ad engine that Google has. And I think it's super interesting problem and

11. 30:20 – 33:20

    Monetization strategy

    1. ASAravind Srinivas30:20

       it's amazing that many companies are trying this at the same time. So even if one company figures this out, like others can also like benefit from it.

    2. SGSarah Guo30:28

       And if you look at a lot of the biggest consumer services, I...... um, they took a while to figure out monetization. So for example, Google, uh, the- Google's first attempt was literally to sell search results. So they got paid per thousand search queries that they'd syndicate to others, and then they built out an enterprise appliance, li- a literal piece of hardware that you'd install at enter- (laughs) enterprises to do search inside the enterprise. And then eventually they came up with ads, and really realized that that was their future path for monetization. And then similarly, you look at YouTube and people said that would never make money, and Facebook would never make money, and all these things would never make money. And then of course they monetize eventually. How are you thinking about monetization, or is it simply too early and it's more about just getting market share and then, you know, at that point you can iterate on the right model?

    3. ASAravind Srinivas31:10

       Yeah. Um, we, we have multiple thoughts about this. Uh, nu- obviously just like Google was paid to, you know, sell the search results for 1,000 queries. Like, Bi- I mean, Bing has APIs like that too, and people asked, at least mor- more than thousands of people have emailed us or messaged us in various different ways to ask for Perplexity as an API, and we haven't done that yet, but that's an obvious monetization strategy. And the other thing is obviously prosumerization of this, uh, where, like, we are already sort of s- beginning to see our Chrome extension pick up rapidly, uh, heading towards 100,000 users. And extensions like Grammarly have this sort of free version and, uh, prosumer version which has more features in them, so there are ways to do that through the browser extension as a productivity assistant sort of thing. Uh, we already see is, like, some kind of search pilot. Every time you're on our site you can a- you can ask us to do things for you. And then there is the whole, as we keep getting more and more traffic onto our site, uh, like say hundreds of millions of v- people come to our site, let's, uh, uh, at, uh, at one point eventually, that becomes a ri- uh, ripe ground for serving ads. But, uh, we need to not make the mistake that Google did of combining ads into the core search product itself, and figure out an alternative pathway like, uh, Facebook did. And, and that might work out better for us. Subscription-based search has been tried by other companies, and that's something that ChatGPT is also trying. So we, we don't know yet if it's high margin enough. Um, and so if th- if a bigger, uh, behemoth like Google or Bing just put out the same or, like, even 80% as good as you for free, then you're never gonna make it as a subscription product. So, uh, we are likely to stay away from that pathway, but we don't know yet. And the final piece is, like, if Perplexity becomes, like, something that, uh, a lot of people want to use for their own internal data, their, their, uh, links or their bookmarks, or their company, and if we can make it easy for them to build that and become more like a

12. 33:20 – 39:09

    Advice for those deciding between academia or startups

    1. ASAravind Srinivas33:20

       platform which everybody can use, then that's likely to lead to monetize- monetization too. So there are, like, so many different rollouts possible here that, um, we don't know yet which one we'll actually go for, but... And f- in the, in the short term we are more focused on growing the product, the users, the traffic, and in fact improving the whole experience. Like, that's... I feel like Google and Microsoft will pretty much do the same thing we have right now, uh, as well as us. So we... As... And as we discussed in the first part of the podcast, like, we need to operate with more velocity, ship more things, and, uh, stay ahead of them, uh, in terms of the core value of the product itself.

    2. SGSarah Guo33:59

       Mm-hmm. Yeah, that, that makes a ton of sense, and I think, um, you know, to the point before, there, there's probably lots of paths to monetize once you have a lot of usage, and so it's more just, you know, figuring out what's native and natural to the product. I think, uh, you, you mentioned earlier that one of the things you'd learned from academia is fast iteration, and I feel like most academics I've worked with are almost the opposite, you know? (laughs) Like, I actually feel like there's a lot of pre-planning and a lot of discussion, and there's, there's less of a bias to action, and so I've been very impressed by the bias to action, um, you all have. What advice do you have for researchers who are now deciding between an academic path or joining a company, research role, or starting a company? Like, how, h- what advice would you give them given that you've gone through a recent transition that's similar?

    3. ASAravind Srinivas34:42

       Yeah. Um, firstly, uh, we both share... Uh, Pieter Abbeel is also Denis' thesis advisor, and my advisor, and he's a pretty different academic from the others. He pushes people to get results pretty fast. And so that's a reason why we both are like that. Uh, and Denis also worked in industry where you can't operate that slow. So for, for the advice to academic researchers, I feel like it's super hard to be an academic researcher right now. Especially a PhD student is one of the worst jobs you can go for in a time when AI is so hot and, uh, so highly paid, and you can do a startup or join a startup. You have to sort of give up all that and the buzz every single day you see for, on Twitter or other, other news journals, and still focus on trying to build a future. Um, so it's, it's more of a mental thing, I would say, more than, like, picking the right problem. Like, even having the, uh, the composure and the maturity to sort of stay poor and, like, work on hard problems is super hard right now. Um, very few people in the world can, have the ability to do that. Um, to, to be very honest, I, I, I only came to the United States for doing a PhD because there was no other way for me to come to the United States. Like, I could- I couldn't take a loan for a master's or something like that, so, uh, PhD is fully funded and, like, sponsored. So that, that was the biggest reason I actually wanted to come here, more than doing a PhD. Uh, if, if, if it, if I had the ability to get a job in industry, I'd have gone for that. Uh, so there's other reasons you might want to do a PhD, but, uh, if you are still... Okay, if all these things are, sort of not a problem for you and you still want to do something, I think it's best to look for alternatives to the transformer, alternatives to, like, language models, that sort of radical directions, than...... trying to improve them, because there's so much incentive for the existing companies to do that. We... A lot of people think open air has no incentive to improve GPT, the core architecture, or some... Or like the model or something. That's far from true. Like, even if they have a lot of money, they would want to make it more efficient and train even bigger models that make better use of compute. So I feel like, uh, it's best for people to sort of look at places that are kind of controversial or r- radical. And, and that would mean even questioning the transformer itself, which is actually one of the best research problems to work on. And that, that's what I would work on if I were to do a PhD. I would work on trying to be... W- write the next transformer paper.

    4. DYDenis Yarats37:18

       Fully agree with this. I think I would probably wanna also add, um, at least like from my experience, I think it's best to not go to PhD right after undergrad and kind of like spend at least a couple of years, um, in the industry. I think I, uh... That goes to my point. Um, to me, like the best researchers are those who can... Who are also very good engineers. Like you can get this like valuable experience of being a good engineer and then it's, it's basically gonna like propel your, um, PhD. Uh, you can, you can do things faster. Um, and especially, you know, like now AI becomes a little bit more like engineering than it used to be, right? So it's, it's just like these skills are essential. Other... Y- you won't be able... Especially if you want to work on like LLMs or like large scale stuff, you have to be very good engineer. But, uh, yeah, if, if, if you still wanna stay on like very academic side where you just like think like very, uh, deep ideas, then I agree with Aravind. You probably wanna completely ignore LLMs and just do something very radical.

    5. ASAravind Srinivas38:25

       There, there are some other ways to do stuff that's still useful and pretty different. Like I think Stanford has some students doing this, like state space models and, uh, flash attention. Some really good papers like that coming out which can... And flash attention is already being used to improve the efficiency of LLMs. So y- you can sort of do such things or you can go work on video or generat- video generation, stuff like that that's just still out there.

    6. SGSarah Guo38:51

       That makes a lot of sense. Well, Denis, Aravind, this was a great conversation and it's really exciting to see all the progress you're making on an important area. So thank you so much for joining us today.

    7. ASAravind Srinivas38:59

       Thank you. Thank you for having us here.

    8. DYDenis Yarats39:00

       Thank you for having us. Thanks. (upbeat music)

Episode duration: 39:10