Embedded AI: The Questions Every CEO is Asking

1. 0:00 – 1:29

   Introduction

   1. SSSteph Smith0:00

      2022 was a breakout year for AI. In fact, many have even claimed that ChatGPT is the fastest-growing app of all time. So with so much opportunity on the table, AI is the topic of conversation in every boardroom as CEOs figure out how to best integrate this new superpower. But they're also asking really important questions around data privacy, competition, cost, accuracy, and also doing this all really quickly. Because just like your customers really don't care if your product is built with Angular or React or runs on AWS or Heroku, there will be a whole host of ways that companies differentiate as they look to cleverly embed AI. As a reminder, the content here is for informational purposes only. It should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures. [upbeat music] Now, before we get into the weeds of how each company is implementing AI, we wanted to start our conversation with Zayd Enam, co-founder and CEO of Cresta. Cresta is bringing real-time intelligence and AI to the contact center. And here, Zayd reminds us of the upside of AI and what it can truly unlock. So I read that

2. 1:29 – 7:50

   How AI can enhance customer service

   1. SSSteph Smith1:29

      30 to 45%, well, contact centers have 30 to 45% attrition in the first year with agents, and in some cases, that can even be as high as 80%. So what is going on in these contact centers? Why is attrition so high?

   2. ZEZayd Enam1:43

      Yeah. I mean, that, that's a great question. The net-- employee net promoter score for contact centers is often less than zero, so generally it's not a role that folks sort of end up sticking with for a long time for a few different factors. You have sort of seasonal demand for increased contact center volume, you have relatively low wages, and then sort of a high inflation environment, and then you have overall sort of a pretty high-stress job. It's like taking phone calls, frustrated customers, angry customers in that kind of environment, sort of pretty stressful demanding job for relative low wage. And so because of that, like all those things add up and leads to like sort of a relatively higher turnover, turnover environment.

   3. SSSteph Smith2:19

      Absolutely. I feel like most people don't need a reminder of how brutal that job can get because they've been on the other side. Now, many people jump to thinking that AI should replace the contact center and customer service agents. But Cresta instead has its eye on transforming a historically low NPS job into one of mastery and creativity.

   4. ZEZayd Enam2:38

      When you look at artificial intelligence, I think there's really two ways to look at it. Um, there's one way to look at it, which is lazy artificial intelligence, and that's basically I have an existing process that my business does right now. I'm gonna take that process end to end, and I'm gonna automate it, and AI's really good at that. You can take an existing process, automate it end to end. There's another way to leverage AI, which is more like an creative approach, which is understanding how does AI-- how can AI be used as a building block? So like AI is now a fundamentally new capability. It's unlocked new things, and I can reimagine my business or my processes or how I do things, my products, based on this new capability, and how do I combine humans, products, AI together to result in something that wasn't possible before? And so when you look at the sort of difference between lazy versus creative AI, the lazy approach is to basically go say, "Hey, this is an existing process. Let me just go automate it end to end." But then the creative way to look at it is that, hey, no, my job or my role as a business is to deliver the best possible customer experience or the best possible product experience. And so like the information and the knowledge that's in my conversations with my customers is, one, like just really great ways to build strong relationships with my customers, but two, is just like a goldmine of data and information about my product, my market, my competitors, what's happening, how the world is changing.

   5. SSSteph Smith3:50

      So can you give a couple examples of that? Like, what can we do with this technology that a human alone could not?

   6. ZEZayd Enam3:56

      Yeah. I mean, so like a great example of it is like when you have like thousands and hundreds of thousands of conversations with customers, the, the nuggets of information about product feedback that you can use to sort of improve your product or improve your sort of how you position your products in the market or how competition is changing. We work with companies that are large telecommunications companies that sort of are constantly looking at evaluating how are they pricing and packaging their various phone bundles or various cable bundles-

   7. SSSteph Smith4:23

      Mm-hmm

   8. ZEZayd Enam4:23

      ... telephone bundles. And there's just a lot of nuance and data and understanding, okay, how does my customer perceive when I make this offer? Like, what is the context that they're coming in with? And that really informs their, like their market strategy in terms of what's a new product that they should launch? What's a new package or pricing that they should bring to market? Like, that's a way that a company can accelerate its development based on just like having a very, very close, like ear to the ground and like a real strong pulse of like what the customer is saying. Um, and that's ultimately the best way to build companies. And artificial intelligence is, is amazing at that because, like what's possible now with large language models, what's possible now with like sort of very advanced deep learning, is that you can summarize, you can sort of synthesize, you can pull together information and context in ways that are f- like really they're-- they can take huge amounts of data, make it super simple to understand, and get to insight and get to understanding really quickly in ways that the conversations were unstructured, not parsable. Like, they were sort of on legacy on-premise, like audio files that you'd have to like listen to one by one and listen to for 20 hours to figure out what the heck's going on. And now you have like this super advanced, like human level speech transcription to like human level summarization, human level sort of question answering that sort of helps these companies get to, get to just the next level of iteration as a business.

   9. SSSteph Smith5:34

      What you're saying is actually really fascinating. So when I thought of customer service, I thought, okay, this AI will actually be able to help converse with the customer. But what it sounds like you're saying is it also provides the business with an additional layer of understanding because it can basically go through all of this unstructured data and gain insights from it. Like you said, if someone- Or some cohort of customers is on one plan, and they notice something about the conversations that they can now parse through, again, this large bucket of unstructured data and go and basically back to the company and say, "Hey, maybe you should start a plan of this nature. Maybe you should offer this cohort of customers this deal. Maybe you should talk to them in this way." Is that basically what you're saying?

   10. ZEZayd Enam6:13

       Yeah. Yeah. And that's like the missing piece of all this is that, uh, it's a bidirectional thing, right? It's like you're serving the customer, but you're also serving the business, you know, being a lot of information in, in all this. And, and there's a really like, uh, for me, like the most inspiring example of this is actually, um, Andy Grove. Intel was, uh, originally a memory business for, for a very long time, and they started as a memory business, and then there was like what Andy calls strategic inflection point, where there was a 10X difference in the cost of memory production. And so all these Japanese manufacturers were sort of able to produce memory in like this 10X difference, and it sort of completely changed the dynamics of that market. It's funny, like the way he phrased it, but the salespeople at Intel came back to headquarters and they would... Uh, the Japanese salespeople, and they would say, "The customers are no longer respectful, as respectful as they were." Like, that was like the first signal like that, that he got that something is different in the market. And it's just inter- interesting phrasing, like the customers are no longer as respectful as it, as it, as they used to be. That was an input to him that ultimately led to him dramatically pivoting and changing the strategy of the company to become a microprocessor company, because they realized that the market is completely changing, customers are changing, customer reception to their products is changing and to their salespeople, and that they need to hard pivot the company to become a microprocessor company. It's a hard decision to make, but when you have the data, like when you understand what's going on with your customers, like you then have the data, you're informed to make the right product company strategy decisions that ultimately save the company.

   11. SSSteph Smith7:35

       So it turns out that these AIs won't just support your customers, but they can actually be an ear or a thousand ears to the ground, feeding data back to your company, informing how you can better serve them. Now, we'll come back to the contact center of the future, but now let's introduce

3. 7:50 – 8:56

   Using AI to shape data and analytics

   1. SSSteph Smith7:50

      Barry McCardel, co-founder and CEO of Hex. Hex is like a sculptor's tool for data, molding, shaping, refining data into valuable insights and visualizations in various formats, or in Barry's words

   2. BMBarry McCardel8:02

      It's a platform for collaborative data science and analytics.

   3. SSSteph Smith8:05

      Let's hear a little bit more about what the Hex team is building into their product.

   4. BMBarry McCardel8:09

      Earlier this year, we launched the closed beta of what we call Hex Magic. Magic is basically a set of AI tools built right into the Hex UI that our users already know and love. And so it lets you generate an edit code. For example, this morning I asked it, you know, "What's our count of paying customers broken down by pricing tier with sum of revenue?" And it wrote me a SQL query that did that. Or you can say, "Refactor this Python code to be a function," and it will do that for you. It also has features to document your codes, which is really useful if you're staring at like a super complex query or something that someone else wrote. [chuckles] It's like, "What is... What's going on with this?" You can ask it, it'll tell you.

   5. SSSteph Smith8:41

      Sounds pretty magical if you ask me. But it's one thing to dream up these new AI features and another to successfully roll them out. And as companies like Hex, Cresta, and soon you'll hear from Sourcegraph, do exactly that, they're inevitably running into challenges.

4. 8:56 – 11:27

   Solving the challenges on contextual understanding

   1. SSSteph Smith8:56

      One issue we've probably all experienced at some point is when technology just does not have the right contextual understanding, like being caught in a customer chatbot loop that just is not addressing your query, something that Cresta knows all too well. We've all been there when we're interfacing with a virtual agent, and it's asking you the same question over and over, and you're like, "That is not my problem."

   2. ZEZayd Enam9:19

      Right.

   3. SSSteph Smith9:19

      "And clearly my problem has not been coded into the potential set of solutions." And I find myself in those cases just writing, "Agent, agent, agent," 'cause I need someone with more of that broad context or the ability to interface in a more dynamic way. Um-

   4. ZEZayd Enam9:32

      Right

   5. SSSteph Smith9:32

      ... but I could also see how that could introduce maybe some unexpected responses. I mean, I actually saw someone on LinkedIn today post that he was talking to a virtual service agent, and they called him beautiful soul. And he's like-

   6. ZEZayd Enam9:46

      [chuckles]

   7. SSSteph Smith9:47

      ... "Who would do that? What human would, would do that?" Soon we'll hear from Barry and also our third guest, Beyang, about how we can actually improve the outputs of these models. But first, here's Zayd commenting on how the utilization of these models, if implemented incorrectly, could really impact your brand. What are you seeing in terms of in the field how different companies are gut checking what's coming out of these LLMs and interfacing with their customers? And I'll also tack on a question, what are people doing in terms of ascertaining whether they should actually disclose whether it's a bot or not? Because as we are getting closer to that idea of human level parity, should we be telling people it's still a bot, or is it actually good enough where we don't have to anymore?

   8. ZEZayd Enam10:33

      Yeah. That's a, that's a good question. I think at the end of the day, like if you are a brand, uh, you are fundamentally building trust with your subscriber or customer base, right? And your brand value is like sort of, "Hey, I can like call up this company or this co- and like they'll take care of me," or like, "I can trust them." And so like in that case, like I do think tran- leaning towards transparency is like the best sort of, uh, thing there, because long-term, you wanna sort of be known as a company that like doesn't mess around or like it's like y- you're, you're, you're direct. You're sort of... You, you tell the customers as it is. And it becomes like an interesting topic as well on the topic of accent masking, right? Because like that's, uh, that's another use case of the technology that sort of coming to market where folks can like use this technology to sort of mask accents or change accents or these kinds of things. If, if you're not upfront about it, and like there's like some edge case that comes up and it becomes like clear, it's like, do you wanna take that risk to your brand reputation, uh, or not?

   9. SSSteph Smith11:26

      So in addition

5. 11:27 – 13:31

   Giving AI the right information and context

   1. SSSteph Smith11:27

      to the right disclosures, we can actually improve the performance of these models relative to competitors, even if we're using the same underlying models. How do we do that? Through fetching the right data. Because it turns out that if AI doesn't have access to the right data, it'll just make something up. Garbage in, garbage out. Value in, value out. Now here is Beyang Liu. Co-founder and CTO of Sourcegraph, a code search and navigation tool for dev teams.

   2. SPSpeaker11:54

      It's this kind of like general purpose, uh, source code understanding engine. Our first kind of major push, I would say, is this editor extension called Cody. Um, actually we had-

   3. SSSteph Smith12:03

      Mm-hmm.

   4. SPSpeaker12:03

      We had an earlier effort that was called Code Search AI, and it was... That was more-

   5. SSSteph Smith12:06

      Okay

   6. SPSpeaker12:06

      ... of like a prototype just demonstrating some of the, the underlying capabilities. But where we're focusing right now is Cody, which is, uh, an editor extension. It's available, um, today for VS Code, and essentially what it does is it's a, it's a chat-based interface, but also allows you to like search for stuff and context in the code. It's gonna do auto-complete fairly soon. And, and the idea is that like we wanted something in our editors that took full advantage of, of the power of language models, but also kind of addressed a lot of the challenge that people have encountered with, uh, uh, large language models. You know, namely the tendency to hallucinate, uh, facts-

   7. SSSteph Smith12:39

      Yeah. [chuckles]

   8. SPSpeaker12:40

      ... when they don't really know the answer. And so that's a place where we thought we could be uniquely positioned to help because, um, Sourcegraph, you know, with all the pieces of context that we have around searching for code and, you know, finding references and veri- verifying things actually exist, we are kind of like the, the perfect fact checker, if you will, for the language model, and perfect like relevant context provider to the language model.

   9. SSSteph Smith13:03

      Mm-hmm.

   10. SPSpeaker13:03

       And so we're really focusing on this kind of like in your editor experience right now and pulling in all the context that we can provide, combining that with the, the power of the language model, and just solving all those like questions and challenges and sources of like tedium and toil that developers encounter like daily, hourly, minutely, uh, when they're trying to build software.

   11. SSSteph Smith13:22

       Yeah. I mean, I think one of the key questions that's coming up right now is if everyone's trying to implement AI, then how are you gonna differentiate

6. 13:31 – 16:25

   Tools that help build language Models (LLMs)

   1. SSSteph Smith13:31

      relative to your competitors? And it sounds like something you're pointing to is that your focus for so many years at Sourcegraph on focusing on really good code search is actually somewhat of a differentiator because what you put into the model matters a lot in terms of what you actually get out of a model, even if everyone is using the same model. And so maybe you could just speak a little more to that about the importance of having code search be integral to your product and how that maybe is a moat relative to competitors.

   2. SPSpeaker14:00

      Totally. So like internally, we've started saying, uh, this phrase, essentially like context is king.

   3. SSSteph Smith14:05

      Mm-hmm.

   4. SPSpeaker14:06

      So the, the context that you provide to a language model, uh, really dictates the quality of the, the output of the model, especially if you're asking about things that are kinda outside the training corpus of, of that model. So, you know, a, a lot of the big language models that we know about today, uh, you know, ChatGPT, Claude, um, you know, Bard, they were trained on, on very large, you know, public datasets and probably some amount of private data, but, you know, they stopped at, you know, 2021, or they're not gonna include the, the private code inside your organization. So they work very well when you're trying to do something kinda generic like, uh, "Hey, build me this app with some standard React components," or something. They do generally well in those cases. There's still a bit of hallucination because the, the language model's memory is, is kinda like human memory, where it's like not precise like we expect computer memory to be. It's just sorta like a little bit fuzzy, and then it kinda fills in the gaps and, and, and hallucinates some. But if you ask it about something that's entirely outside of its, uh, training set, it just doesn't know, and worse than that, it, it's not gonna tell you it doesn't know. It, it's gonna pretend it does know and, uh, hallucinate something that looks right but is completely off. And I think like everyone who's tried to actually play around, uh, with these models in their, in their day-to-day workflow has hit this issue. That- that's a place where Sourcegraph can come in and essentially provide like the missing knowledge gap so that instead of like talking to this language model, which is kinda like talking to like a, a, a reasonably smart person on the street, asking them a question, and them just trying to like invent an answer from what they remember in their minds. Now you arm them with like, you know, Google Search or, uh, you know, whatnot, and they can go and like not only just like, uh, use their memory to answer your question but like go look up stuff on the internet. Uh, or in our case, like in, in the, the wide, wide world of, of open source code, the recent changes that have happened there, as well as like all the private code in- inside your organization. And that makes the, the answers much more precise. Like when we think about the, the issues that Cody still has today, and it's, you know, obviously not perfect, um, almost all the quality issues come down to, "Oh, we didn't fetch the right context into the context window."

   5. SSSteph Smith16:07

      Interesting. And I mean, let's, let's double-click on that because there are other tools that help you build code using these language models. We'll just throw out a couple, like a lot of people are familiar with GitHub Copilot, a lot of people are familiar with what Replit is doing with Ghostwriter. But maybe you could actually speak to this idea

7. 16:25 – 19:08

   Constructing prompts

   1. SSSteph Smith16:25

      of fetching the right information. Like how would something like a Copilot do that, and how would something like a Cody actually differentiate in its ability to fetch the right information?

   2. SPSpeaker16:35

      I think the way we think about it, like as, as far as I know, Cody is, is the only AI-enabled editor assistant or coding tool today that, that fetches context as, as kinda like, as broadly, uh, as we do. So when you ask... You can ask Cody a question like, "Hey, you know, where, where is like the, the SAML auth provider defined in my code base?" Or, "Where's like the GraphQL search API, uh, defined?" And Cody will actually go and convert that to a couple of search queries, uh, to Sourcegraph and surface these relevant snippets of both code and documentation. Use that as like concrete references to answer the user's question. And that's in contrast to the way that Copilot works today, where most of the context they're, they're looking at, number one, they don't do, uh, Q&A yet. I, I think that's on the way, uh, with Copilot X. But currently, they don't do Q&A. It's, it's purely kinda like auto-complete driven. And the context that they fetch to do that auto-completion is kinda like recent files that you've opened in your editor. So it's kind of like this very local context, which works amazingly well. I mean, like huge credit to that team. They've built an awesome user experience. We think that the next evolution of that is, is, is providing more relevant context and essentially emulating- Like what a human kind of does when you're trying to write code, right? Like you as a human, you, you might go back through some recent history in your editor to see like, "Okay, how does that code work? How did that code work?" And use that as like a, a pattern matching reference point for the thing that you're currently writing. But more often than not, I think you're doing stuff like, you know, go to definition, find references. You know, let me see a couple of, of examples of how to use this particular API that just... I just imported. And that's Sourcegraph's bread and butter, right? Like, that's the code graph that we spent the past 10 years, uh, building, and we can do that essentially in any language, any code base at compiler-level accuracy. And so we can do that automatically. So like the, the language model Cody will essentially do the kind of like toilsome task of like, you know, fetching context and then click, click, click, clicking through the code graph to, to acquire the necessary context, and then take your user prompt and synthesize an answer from like, uh, the actual code snippets that it thinks are relevant. And I think that that's gonna lead to much better results. I think it's also gonna lead to much, uh, more kind of introspectable results. So getting beyond this like, "Oh, LLMs are magic. How do they work? Is it AGI?" You know, whatnot. It's like Cody will actually tell you like, "Hey, I read these files and these are the files I'm using to generate an answer." And if it completely, uh, you know, returns a lie or, or is wrong, you can usually tell by looking at the context, uh, that it read. Like, oh, like that... Why are you reading that file Co- Cody? That's dumb. And you can like thumbs down that and, and, and we'll take that as, as like a reference point to, to improve the product later.

   3. SSSteph Smith19:06

      Another key question that founders are

8. 19:08 – 21:09

   How do you differentiate?

   1. SSSteph Smith19:08

      asking is how they'll differentiate, and Sourcegraph thinks that open source is part of that answer.

   2. SPSpeaker19:14

      The other major distinction I would draw, uh, from a lot of other offerings on the market is we are trying to build this as much in the, the open as possible. So like Cody, the, the editor extension, we've released as, uh, open source under the Apache 2 license, and we think that's kinda like the r- right mentality to build, you know, the standard AI editor assistant that every dev, uh, should use. We think that there's kind of like a natural inclination, uh, among developers. I'm a developer myself. I prefer to use open tools, and I think it'll, it'll, it'll make it possible for us to build like a, a much more like pluggable ecosystem. Like, when you think about like, like the, the wide wide world of like context that you would... might wanna pull in to answer a question in your mind as a developer, you know, it's, it's not just the source code or, or the markdown documentation. Like, maybe you're searching through your issue tracker. Maybe you're searching through like chat messages. Maybe you're searching through like Google Docs or, or, or Notion for like the, the latest product spec. And by virtue of, of being open source, we, we make it a much better and friendly kind of ecosystem i- into which we can plug in like other developer tools and other pieces of context that aren't necessarily tied to a, a proprietary compute platform.

   3. SSSteph Smith20:22

      Yeah. I mean, something that's jumping out to me is if people aren't developers, like even thinking about writing an article, right? Imagine writing that article based on just like your last five tabs that you had open. That's gonna be very different to being able to actually channel your goal and search the web yourself, search your own notes, um, that are specifically likely relevant to the task at hand. And so I think this is like a really interesting perspective as to we know what goes into the model matters in terms of what you get out of the model, and basically what you're establishing is two different approaches, both the open source approach, which allows more people to help you extend it, but then also this approach of really finding the most relevant information that's gonna help the model give you the best answer.

9. 21:09 – 22:22

   The value of proprietary datasets

   1. SSSteph Smith21:09

      All right. So we learned from Beyang that what you feed models really matters, and that may be an opening for differentiation relative to competitors. But part of that equation is just the data that you have access to. Here's Zayd commenting on the value of creating proprietary datasets.

   2. ZEZayd Enam21:24

      The opportunity that we see, or the opportunity I think that exists in the market, is that you can do that exact same thing if you can collect proprietary datasets that sort of are unique and at that same scale, and you can actually sort of, in some ways, sort of actually sort of train foundation models that are like sort of able to do even more. So like the internet or the web- web pages are like a fairly sort of like static sort of language modeling task where like you're sort of doing a task and you're, you're trying to complete the next token or the next word. The kind of things we interact with in the contact center, which is like both dialogue and action on sort of you're interacting with someone and then you're sort of working with enterprise software and systems of record to like fill out basic things. It's both sort of this intersection of like two sets of datasets that don't really exist together, but... and we sort of are able to collect through, through, through our, our work with a lot of different companies and customers like a really, really big dataset of like this kind of, this type of work. And then that sort of enables us to then train over time these large foundation models that sort of use that intersection of that two dataset to sort of do things that aren't possible just with, with, with web pages.

10. 22:22 – 23:44

    Customization as a moat

    1. SSSteph Smith22:22

       Another wedge may be customization. How can you actually use the information from past user behavior to support the user in creating better prompts? Here's Barry's take.

    2. BMBarry McCardel22:32

       Under the hood, we are doing a ton in terms of constructing the right prompts and parsing responses back from the, the model APIs we're using. And so we, again, we have thousands of people already writing SQL and writing Python and doing data work in Hex every day, so we have a ton of information we've got. We're connected to their database schemas, so we see the structure of their data. We see past queries and past code they've written, so we know which tables and columns are most frequently referenced. We have information about the project they're building, so we know like, oh, this project is already referencing this part of the schema and that's probably the relevant data for this. We could even look at things like this is the typical way this organization is formatting their charts and infer how they might want their visualizations to look. And so there's a ton of context we get because we're incorporating this in an existing set of workflows that can help us create the right context and, and, and create the right prompt to pass to the model. And I think with all of this right now, our, again, our focus really is on augmenting the data professionals. Like we're not... This is not like, you know, trying to build some black box thing. It's, it's actually like we have a rule, uh, as we're building these features. [chuckles] It's like we don't run the code for you. Like we will generate the code, we will show it to you, but we wanna keep the human in the loop.

11. 23:44 – 26:49

    Increasing compilation rates

    1. SSSteph Smith23:44

       AI sure can feel like a black box, so designing a UI that actually guides the user is also becoming a differentiator. At least for the time being, where you're effectively nudging users to want to interface with these new tools, but also come back. Any learnings in terms of what is increasing completion rates, what is getting people to actually interface with this new, in some ways, superpower within the app? Because there is the flip side where, as you said, if it's hallucinating or if it's not helping them, or in some cases it may even be hindering their workflow, they're not gonna return to it. So how are you thinking about designing that?

    2. BMBarry McCardel24:21

       There's like some UI things we've done as an example, like in Hex, you know, we kind of have these code cells that you use, and so, you know, we put a lot of thought into like what's the right way to interact with that. Is it like a comment completion? We, we have this cool thing that, um, seems to have been inspirational to other people now, [chuckles] um, a little drawer that kind of top, pops out of the top and like allows you to like kind of edit what's been below, and it kind of tucks itself away when you're done with it. You can keep using it. I think giving people feedback on what's happening, um, some of these models, like the latency is still really high, so even something small like just being really thoughtful on like how you're exposing to the user what's going on and why they're waiting. One of the things that we've observed sort of more on the back end in terms of increasing completion rates is when you're building prompts, I think we've been-- we were tempted early on to try to shove as much context as we could in. You kind of figure like the more I can tell this model, the better. And you realize you can pretty easily confuse a model in terms of the amount of context you're passing, and so we've been really thoughtful on iterating through and finding what's the right level of context to pass through between what else is going on in the project or the underlying data schemas or, or past work they've done that's gonna best encourage. And there's a lot of techniques you can start to la- like sort of ladder up and use in terms of working with chains through LangChain, through starting to apply embeddings to better allow you to sort of craft the right context to be able to pass through that model, and I think that is really actually the hard work in many ways of i- incorporating AI into your product, right? Like it, it feels really simple, like you can go get an AP... Anyone listening to this today could go to the OpenAI website and get an API key. I- it's everything you're doing before you're sending requests to that API of like building the right context, understanding how you're iterating on how different models are gonna respond to different types of prompts and different amounts of context, how you're p- potentially even chaining together different types of models or different modalities of models together in one sequence in UX, and then how you're giving that feedback to the user. I think, again, see it feels like a, a really exciting time 'cause it's very early days in figuring all of this out.

    3. SSSteph Smith26:19

       Yeah. I mean, I spoke to someone yesterday, and what you're saying really resonates with what they said. Um, they started off by creating like a Lensa-like product, but at the end of the day, there wasn't really a moat there because anyone could kind of hook up to Dreambooth and create the same thing.

    4. BMBarry McCardel26:32

       Yeah. Right. [chuckles]

    5. SSSteph Smith26:33

       But today I, I asked him like how he's thinking about differentiating, and at least in his scenario, he was saying, "You know, I, I now link 15 different models." And so that, maybe that isn't much of a moat either, but the point is, it's not as simple as just hooking into one API. So we've

12. 26:49 – 29:49

    Where is the moat?

    1. SSSteph Smith26:49

       touched on different aspects of differentiation from customization to context fetching, but is any of that truly a moat? Where does value really accrue when your competitors can copy your UI or your latest feature? Let's attack that question head-on. Where is the competitive advantage? Where is the moat? Is it in the UI? It feels like that can be replicated very easily. Is it in the cleaning of the data, the linking of the data? Is it something else? Like, how are you thinking about the fact that basically you can kind of depend on all your competitors to replicate what you do if it's working?

    2. BMBarry McCardel27:23

       Yeah. We've, we've already seen that. [chuckles]

    3. SSSteph Smith27:24

       Yeah. [chuckles]

    4. BMBarry McCardel27:25

       Uh, down to, down to some UI elements that look very familiar. Um-

    5. SSSteph Smith27:29

       Mm-hmm.

    6. BMBarry McCardel27:30

       Yeah. I- I-- You have to take that as a given. I think the, the models themselves and being able to send a request to one of them is, is already a commodity. I think there's a few layers that we think about this at. One is we do have a pretty great data advantage. You know, already having hundreds of customers, thousands of users writing millions of lines worth of SQL and Python, we can use that sort of rich information. Um, we're not using it to train models, I should be very clear. Like, that's one thing we've been very upfront with our customers about. Like, we're not training models where they would expect to ever have, like, some code they've written be a completion for someone else, which is a problem in other places. But it's more like using that actually to personalize for that person and their team. You know, again, like which schemas you're using, which-- what has their code style been like in the past? Like, having all of that information and people who are already doing that work in Hex gives us a big advantage. Thing two is, and I talk about this with the team all the time, like it's kind of what we've already been doing, which is building really thoughtful, well-constructed user experiences and UIs. Like, you could say that there's very little moat for a lot of things we or a lot of other SaaS companies do inherently for, in their products. It's really how do you put these pieces together, and how do you build a really great user experience, both from, you know, the pixels on the screen to thinking about performance behind the scenes to things like docs? Like, that all sort of combines to being a really superior product experience, and it's stuff that we're, you know, three and a half years into taking super seriously. And so that, that is a sort of advantage and almost like infrastructure we can already stand on in terms of how we start incorporating these things in. I do think that long term, generative AI, large language models, they will change a lot of like fundamental assumptions, and I think that there will be an advantage to the companies that can be the first ones to figure out what those things are.

    7. SSSteph Smith29:17

       Just like social apps benefit from network effects, it's actually not crazy to imagine that even if certain products are built on the same models, first movers that incorporate customization may maintain an advantage. Because if you spent months or even years personalizing a model to your specific needs, would you want to do that again? But as these platforms collect more information about your preferences and potentially even train models on your data, certain companies may build a data moat. But that's also where privacy comes into play. The future of privacy and security in AI

13. 29:49 – 34:01

    Privacy challenges

    1. SSSteph Smith29:49

       is a complex and evolving issue, especially since your competitors may be on the other side of that API call. Companies are rightfully asking questions about data collection and data storage. A lot of these companies, um, that are integrating AI are building off of just a few models, right? A lot of people are familiar with OpenAI's API that came out recently. Um, but there's also that very interesting dynamic that a lot of the same companies that may even consider themselves competitors are, are using similar models. And so how did you think about that? And also, there's this kind of layered question as it relates to security and privacy, because depending on the company that you are, your code is actually potentially somewhat all the way to extremely proprietary, right? If you're talking about, like, a self-driving car company. And so how did you think about that, given that, again, this code for the, the customers that work with you is extremely important?

    2. SPSpeaker30:43

       Yeah, that's an excellent question, and it's especially pertinent to us 'cause we have a lot of enterprise customers that are, are very security and, and privacy sensitive to the point where, you know, Sourcegraph is-- One of the reasons we made it self, uh, hostable is because we wanted to, uh, enable companies that didn't wanna put their code bases in, in the cloud to still have, like, awesome code understanding, uh, and, and code search tools. I think for us, like, the, the way we think about it, like, we love OpenAI, we love Anthropic, um, Cohere is also great. I'm looking forward to, you know, what Google releases in, in this, uh, arena too. And there's just, like, amazing developments happening in open source, uh, as well. Um, like there's a team at Salesforce that produced a really good codegen model called Codegen, and it's, it's kinda competitive from, from what I can tell with, with kind of the, the Codex-based model that, you know, other tools use at the moment for, for autocomplete. And so the, the, the space is, is kinda like fast evolving, and our mentality is like, look, we have a wide range of customers from, like, very conservative large enterprises to, like, fast-moving startups that have different risk and security profiles. The language model in, in our, like, overall architecture is just one component. It's one component of the, the bigger Cody picture. The, the other, you know, components being the, the Sourcegraph code graph and, and the various other developer tools, uh, that we wanna integrate in, in kind of an, uh, open way. And so we wanna make it possible to kinda like bring your own language model to the table. You know, some of our customers have negotiated separate deals with model providers. They wanna use that model. Some have in-house models that, that they've built that they wanna use, and some are like, "Hey, can you give us something that we can self-host?" And while we don't have one that's, like, completely self-hostable yet, I think, you know, with all the interesting things happening around, you know, Llama, Alpaca, and, and that sort of thing, I think, like, within six months, we'll, we'll have, like, a viable language model that can do, like, really good Q&A-based interactions. And so our mentality has just been like, look, we wanna work with all the model providers. They're all innovating at just a, a, a crazy, uh, like, breakneck pace, and our desire as a business is to do what's best for our customers and users. And, and right now, that just means, like, whatever's the latest and greatest on the market, like, let's try to plug it in and, and see how it does.

    3. SSSteph Smith32:55

       So you're basically saying that you give them the selection or the option. Am I understanding that correctly, or are you still building off of one model? Like, when someone uses Cody, is that integrated with one, or are you basically saying, again, you're gonna give them the option to select how they wanna integrate?

    4. SPSpeaker33:11

       Uh, we'll, we'll give you the option. So right now, yeah, you can use, uh, Claude, which is Anthropic's, uh, kind of flagship model. You can use, uh, ChatGPT, which is, uh, kind of the OpenAI model. And we're looking to integrate additional models too. And there, there's also kinda, like, different models that we plug in in different pieces of, of Cody, right? So there's kinda like the chat-based models, which are often, like, the largest ones. Um, but there's also things like the, the embeddings model. That's what we use to generate the kind of embeddings vector. Uh, the embeddings vectors that we, we use to do really good kinda, like, fuzzy semantic-level, uh, code search. And we have an open source model that, that, uh, we fine-tuned, and that's actually, like, the best mod-- embeddings model that we have. Now, if you, you wanna use, like, one of the, uh, proprietary embeddings offerings too, we, we can definitely do that. But I think our, our mentality is just like the language model aspect of this we wanna make as, as kinda like pluggable as

14. 34:01 – 39:35

    Cost and pricing of models

    1. SPSpeaker34:01

       possible.

    2. SSSteph Smith34:01

       That's amazing because something that that also relates to is cost, right? Like, each of these different models has a different cost. I think a couple weeks ago, Bing, like, 5x'd their pricing overnight, right? Like, you're-- you have a dependency as well, both Sourcegraph, but also, like, that ends up filtering down to your, your customers. And so every one of these models-- I mean, I, I think we're still in the early innings, and there's gonna be so many more developed. And each one will, to your point, it'll have a different security posture. It'll have different pricing scheme. It'll probably... You know, there will be a range in terms of its efficacy or specialty in certain, uh, areas. And so that's actually-- You know, it never dawned on me that actually, you know, you could offer the access across, across the board to all these models but also kind of relay the transparent pros and cons to the customer base. Is that how you're kinda thinking about it?

    3. SPSpeaker34:51

       That's exactly how we're thinking about it. And I, I think for us it's, it's kinda like there's so much innovation happening in that space, and, and we don't want to be kind of tied to, to any one provider. And, and also I think a lo- a lot of the value that we can provide in terms of Cody and in terms of, like, integrating AI deeper into Sourcegraph is really about combining the language model with the pieces of context and the structured, uh, understanding of code that, that we have. And so it's really the combination of those two, I think, that, that, uh, makes something truly valuable. And it's funny that you mention the kind of Bing price hike. I thought that was, like, a, a big proof point, and, and people noticed. Because, like, when, when, when ChatGPT first came out, I think a lot of people said, like, "Hey, you know, this kinda replaces search engines," right? Like, "I, I could just chat with this thing, and it'll just tell me the answer instead of me having to go and, like, click through a bunch of different results, uh, and, and figure out the answer myself." But then as people started to use language models a bit more, they started to run into more hallucinations. And I, I think it was, like, the release of Bing where people finally realized, like, like, Bing released, they, they integrated, um, ChatGPT or, or GPT-4, you know, one of those, like, awesome, like, OpenAI models in. But they, they combined it With the, the fundamental Bing search. It, they just didn't, they didn't just like ship a white label ChatGPT. They combined that with, uh, Bing search on the back end, and a lot of the other AI-enabled search engines were actually using the Bing back end as well because that was... A- as far as I know, like Google doesn't have like a search API that, that, that you can use, so everyone was using Bing. And that just spoke to the, the necessity, I think, of combining kind of the language model as, as sort of like the, the reasoning engine, but you still need kind of like an informational retrieval engine to make that truly powerful, and it's the, the unison that, that really is valuable. And, and that's maybe, I, I'm speculating here, but like that maybe had something to do with th- the Bing, uh, price hike. Like it, it is not true that language models make search engines unnecessary. If anything, they make the search engines more valuable because now all that data that you can search becomes like 10X more powerful because you can use that to, you know, get to, g- get to your answer with like, you know, one tenth the effort or in one tenth the time.

    4. SSSteph Smith36:57

       Yeah, I mean, that, that's such a great point that the search engine itself is still, it still has utility to find all the information that at the end of the day feeds into the language model. Beyang brings up a really important point around price. These models are not free, and we're still in the early innings of figuring out the right business models for LLMs. How much should they cost? What value add would allow certain providers to charge way more? And is it worth integrating into multiple models to reduce price or platform risk? But something else that feeds this equation is the cost to maintain data, which is also not free. Here is Barry reflecting on the deep relationship between dollars and data. One interesting facet of all this is that there's kind of this, as we just discussed, this realization of the value of data, and I think we're gonna see a lot more companies retain their data for longer, collect more data from their products or from their users. And so there's this, again, this privacy security posture of like, "Okay, we wanna have this data. We're probably gonna keep it for longer. How do we keep it safe?" But then there's also a cost element, right? It's not necessarily free to retain data. It's a- also not free to run these models.

    5. BMBarry McCardel38:08

       It's also not zero risk. A lot of companies intentionally delete their data. Like people will have retention policies on emails or Slack, but by doing that, you're eliminating knowledge, and that knowledge could be useful sort of like in the first instance, just like I wanna go back and search. That's the thing I think a lot of big companies deal with as they start to like have record retention things. But there's also like second order value to that in terms of could that inform a model of or a, um, you know, potential different applications for how you could learn from what your organization has done historically. So I, I don't know where we're gonna wind up on that. I do think you're right that a lot of companies are realizing the value of their data and their IP and thinking about that in a deeper way. Already operating in the data space, both at Hex and previously, we already have this like appreciation for people being really paranoid about their data, people really caring about where their data goes. And so in some ways, there's nothing new for us here. I think there's all sorts of really exciting opportunities on how you could use data that customers are trusting you with, making sure you're continuing to earn that trust.

    6. SSSteph Smith39:10

       With all these considerations, it's really easy to get stuck in the weeds. So let's take a step back and imagine a future where companies have successfully deployed this technology, not as a gimmick, but in a way that successfully gets their customers closer to their goals.

    7. SPSpeaker39:26

       What differentiates software is like can you build the workflows that get to the business outcomes?

    8. SSSteph Smith39:30

       And as we close out, let's return back to where we started, the contact center. Let's

15. 39:35 – 44:40

    What does the contact center look like in 2028?

    1. SSSteph Smith39:35

       say, you know, we're in 2023. Let's jump to 2028, five years from now. Given the technology that we have today, also the change in the cost curve that you mentioned, which sounds really fundamental, what does the contact center look like in 2028?

    2. SPSpeaker39:48

       What I think it looks like is this concept of like w- like I remember like when we started the company, uh, Tim and I, we got this video from our first customer that like we, we had both, we both had not dropped out of the PhD yet, but like we were working with our first customer sort of like getting our first sort of thing and our, and they showed us a video of what their vision for the contact center of the future looked like, like this is five years ago. And when we saw that, we were like, "Holy shit," like, "Okay, everyone has the same vision. Let's, let's, like we're gonna drop out tomorrow, and we're, we're gonna go, go do this company." Um, but, but the vision was basically this idea of like a touchless close where, um, it was one of, it was, uh, one of our first customers who basically had this idea that they, they had a contact center of like people that would, didn't, that didn't sit at computers anymore. They just like would walk around. They'd have like a, a, a, a dialogue, uh, sort of a, just a, a piece that they were using to talk to customers, and all they were focusing on was the interaction with the customer and the relationship building aspect. And on the back end, like the system is like doing the, um, it's like doing the sort of automatically data entry, the filling, filling in the form, all these things. It's like taking care of all the stuff on the back end, and like the agent's not typing anything. They're not doing any data entry. They're not doing any summarization. They're not looking up the customer record. Like all that stuff is like handled by the machine, and it's handled in a way that the human can just like focus on how do I build a great relationship and resolve this customer's problem or like help finding the right product for, for their solution. And like that's naturally what humans are great at, which is that sort of empathy and connection and sort of relationship building. And I think that's what's gonna be, we'll have a much less regimented contact center, like much less sort of structured and like these are your times where you have to do after call work. This is your av- average handle time goal. These are the things. It'll be just much more focused on, hey, how do we use this center as a proactive way to strategically build relationships with key customers, and how do we build those relationships with, uh, with our folks and like take away all the mundane busy work from it? And then for the stuff that people want, like frictionless experiences, like there will be things that they don't want to interact with a human about at all, and like those are like we can provide those in a fully digital sort of frictionless manner that they can deal with if they wanna deal with a bot. For those like strategic things that they want to interact, they want to trust someone and have a interaction with someone for like a critical decision and like feel like that they can trust something, that's when we have humans available, and like we won't have those humans tied up in busy work and mundane work. We'll have those humans tied up on how they actually can help the customer, and they're not measuring their average handle time, and they're not measuring like h- like, "Oh, I have to get to the next call. I have to fill out these notes for this call." It's they're able to focus really on the customer interaction.

    3. SSSteph Smith42:10

       That's a very nice picture that's painted, and in that world, it's, it's almost as if the ... customer service agent is actually like a strategic role, right? It's not just like the, the day-to-day let me plug in, plug out, let me solve this customer's request. It's almost like, like an account manager or something like that.

    4. ZEZayd Enam42:25

       Yeah, and I think that's where it's headed, because once you have systems that handle the repetitive, like reactive things, and you move to like sort of becoming strategic and proactive, it's like how do I proactively build businesses? Like when I build relationships, I'm proactively reaching out to build relationships with large enterprise customers, right? And that's like a very strategic use of my time to like go spend time with them and like sort of build those relationships. I think the same concept applies to like even if you-- whether you have like 100 enterprise customers or 1,000 enterprise customers or like 100 million consumer customers, it's like those folks are people at the end of the day, and like building relationships with them is just as like strategic as, as, uh, as it is with, uh, your, your large number of enterprise customers. And so now technology can enable you to build those kinds of relationships in a very human way at scale.

    5. SSSteph Smith43:11

       Yeah, and, and on the customer side, I could see how it could actually be, you know, you could even say fun or entertaining with the integration of technology. Imagine that, you know, you're, you're talking to a bot, and not only does it serve your problem, but it's also in the voice of like your favorite cartoon character, and it's telling you answers in a way that you're actually like following a story or something like that. And actually, oh, because they've done all this back-end research based on these customer conversations, that actually you leave the conversation with a better deal than you had before, and you didn't have to spend three hours on the phone to get there. Like I, I actually, as we're talking about this, I'm like, "Wow." You know, you go from, as you said, a negative NPS industry, like you could actually imagine that these things could be fun and frictionless.

    6. ZEZayd Enam43:53

       Yep. Negative to positive.

    7. SSSteph Smith43:55

       Negative to positive. What a great place to end off. AI's rapid evolution is constantly challenging companies to stay ahead, and navigating the implementation of AI is bringing up issues of personalization, design, cost, and privacy to center stage. A big thanks to Hex, Sourcegraph, and Cresta for sharing their insights and experiences in this ever-changing landscape. And if you'd like to learn more about these companies or what they're doing with AI, links can be found in the show notes, where you can also find links to a bunch of our recent AI coverage. Thank you so much for listening, and I'll see you next time. [upbeat music] Thanks for listening to the a16z Podcast. If you like this episode, don't forget to subscribe here on YouTube to get our exclusive video content. We'll see you next time.

Episode duration: 44:50