No Priors Ep. 49 | With Shopify VP of Core Product Glen Coates

1. 0:00 – 2:22

   Background

   1. SGSarah Guo0:00

      (music plays) Hi, listeners, and welcome to another episode of No Priors. Today, we're joined by Glen Coates, the VP of product at Shopify, where he leads the core developer platform, including all of their AI products that we'll get into today. Before he was a leader at Shopify, Glen was the founder of Handshake, a B2B e-commerce platform that was acquired by Shopify in 2019. We're excited to talk about how AI is changing e-commerce and entrepreneurship as well as innovation at scale and leading at Shopify. Welcome, Glen.

   2. GCGlen Coates0:33

      Thanks for having me.

   3. SGSarah Guo0:34

      So, lots of fun stuff today. Let's definitely cover your personal story quickly since you're also a former founder, one of several at Shopify now. Can you give us some of your background?

   4. GCGlen Coates0:43

      I- I was a comp sci grad. I spent the early part of my career in video games, and then I had a weird left step in my career where I moved to San Diego in 2008 to run the US operations of an eco-friendly shopping bag company. You know, like, you take your own bags to the store, that whole thing. Anyway, this company did a whole bunch of wholesale. They sold a lot of bags through stores, and through that I ended up, um, getting to the idea for Handshake. And Handshake was a- initially a very, like, sales rep-focused B2B e-commerce product, but then became both for sales reps and for customers to buy online. So Handshake eventually became basically a B2B-only version of Shopify, like a wholesale-only version of Shopify. So it's not hard to join the dots from there to after, you know, building Handshake for better part of nine years here in New York, the opportunity came up to, uh, join forces with Shopify, and then I've been at Shopify for almost fi- it'll be five years in May. I started out very much focused on- on B2B and wholesale, which was obviously the point of the acquisition. I spent about a year doing that. I then moved on to focus... uh, I- I ran a- a code red on checkout in 2020, which was the first year of the pandemic, and then since the end of that, um, I've been leading the kind of core product group at Shopify, which is- um, you could think of it as all of the built-ins of Shopify, the online store, the checkout, the back office, the developer platform, the app store, um, uh, and yeah, that's- that's what I do at Shopify.

   5. SGSarah Guo2:21

      Lots of-

2. 2:22 – 4:04

   Calling a “Code Red” at Shopify

   1. SGSarah Guo2:22

      lots of good stuff. What's a code red look like?

   2. GCGlen Coates2:24

      A code red is when Toby sends an email to the entire company saying, "This thing is the number one priority," and he means it. (laughs) From an actual operational perspective, what that really means is if the team that's working on this code red asks you to help, please drop what you're doing and help. This is the number one priority. So that's how they work, but usually a code red is a symptom of some other much more systemic problem that's gotten you to that point. At least in my case, checkout code red in 2020 was, "Hey, the checkout is failing in, I don't know, three, four, five different ways, and the checkout is a pretty important part of Shopify obviously, so let's go fix that." So we, me and, you know, somewhere between, I think at- at its peak it was probably 200, 300 people working on various parts of the problem. So we all, like, scrambled for a year and, like, did what it took to, um, fix those issues, but then at the end of that year, Toby took a step back and said, "Okay, well, why did that happen? Like how did we get to the point where those problems even happened in the first place?" And then that led to some of the reorganization of the company around less... like there used to be 12 to 15 of these kind of fairly small fractured business units, and now there is actually only like three or four which is actually truer to what the product is, but of course each of those units is bigger than the ones that were before, so it actually requires you to lead in a slightly different way o- once you start grouping that many- much of the product and the people together.

3. 4:04 – 12:15

   Integrating acquisitions, entrepreneurial leaders

   1. GCGlen Coates4:04

   2. EGElad Gil4:04

      I guess, like, two related questions from your background. I guess number one is since you were acquired into Shopify, one of the things I've been very impressed with by the company is the degree to which it's been able to retain and grow the roles of founders that it's acquired. Because as far as I understand it, the majority of the management team at this point was acquired in at some point, or at least a very large fraction. And then secondly, as you buy a lot of things, you end up with a lot of disparate products, disparate platforms, disparate integrations, and all sorts of things can go right or wrong based on that. And so I was curious a little bit about the broader concept of integration, number one, from a team and individual perspective as a founder coming in, and why is it such a good home for founders? And then number two, like, once you're acquiring the- the- the tech and the- the product, how do you integrate that as well?

   3. GCGlen Coates4:45

      Yeah. It- it's really notable. So like when I look at the... so the- the- the sort of exec team at Shopify contains a number of founders, some of whom were acquired in it and some- some of whom just were founders. And when I look at like my management team, the- the people who report to me who each of these sort of nine or 10 people run R&D orgs that are like a couple hundred people each, and almost like a very high percentage of- of those people are also founders, right? And so you ask, okay, well why are all these people ending up essentially running the main parts of the product of Shopify, and Shopify is a product-first company, so that's sort of the same thing as saying running Shopify. I think a lot of this comes from, uh, Toby himself who has extremely strong opinions and has, I think, learned for the better to no longer be shy about, like, saying, "I want to aim in this direction," right? I think there was like a big culture of bottoms-up decision-making, power to the edges, like whatever you want to call that school of management philosophy which is like, you know, delegate and empower. But I think a lot of former founders are actually coming back to the place where they say, "Hey, what makes... what am I uniquely good at? I'm uniquely good at, like, having a point of view, being willing to..."... smash my head against a wall and, like, do whatever it takes to manifest that point of view in the world through building things, building teams, you know, saying the hard thing at the right moment. A- and I think Toby has kind of re-embraced that, right, in the past few years of saying like, "Hey, if I'm good at one thing, it's being extremely opinionated." And it seems that in this domain, at least for Toby, in this domain of, like, the internet and commerce, "I'm right more often than I'm not. Like, I have a higher batting average than usual. Therefore, if I can help accelerate this company through, like, being decisive, that's gonna be good." And so, he sort of sets the example and a lot of us, you know, see that and we're like, "Hey, we- we remember being founders. We remember what it's like to have all bucks stop with you, and if there's a way that we can apply that level of decisiveness here and accelerate teams through, like, the otherwise, like, you know, design by committee bullshit that kills a lot of companies, then, like, that's a net positive, right?"

   4. EGElad Gil7:12

      The other piece of int- of buying in companies is sort of integrating different systems and platforms. And often you see people do one of two things. They either let things run independently, that was WhatsApp for a long time at Meta for example, or you integrate in all the infrastructure, and usually that makes things more performant, you can have features that cross over, et cetera. But the flip side of it is sometimes you see these giant projects that just stop a company from functioning, right? Like at Google there was a famous sort of identity layer that was built and for a year and a half people just didn't ship things, right, uh, and other areas. And so how do you all think about integrating in acquired companies and acquired technologies? But more generally, as you build and build and build and build, there's a bin- a big mung of stuff. How do you sort of consolidate that down in a performant way that's actually speedy to execute? And then, I guess, later on maybe we can talk about whether AI has played a role in that or not.

   5. GCGlen Coates7:59

      Yeah. It's- it's a really good question. It- the- the answer is it depends, like most problems. Usually when you encounter these problems, you sort of have to look at the stack. Y- you usually don't want duplicates at the same layer of the stack, right? Um, and the funny thing about this is sometimes you actually don't notice that two things are in fact at the same layer of the stack until much later on, right? So as an example, inside of Shopify there is a, uh, there's the checkout which takes, you know, someone's shopping cart and says, "Okay, let's take the business rules of this store and convert that to an agreement for a sale with taxes and duties and shipping and all that stuff." So that's one engine. And then at different points in Shopify's history other things that have been built that are sort of like adjacent to that but not quite the same thing, right? So like, a thing for creating, like, an invoice or a thing for, like, even editing an order after it's been placed. And a lot of these things kind of get built 'cause- 'cause you start with a small problem. Example, I just need to be able to add a discount to an order after it's already been placed because, you know, someone calls up and says, "Hey, I forgot to put the discount code on. Can you please enter it," right? And so someone starts a project over here which is, "Oh, we're just gonna support editing discounts onto orders," and you don't realize that what you're actually doing is creating a- a second version of the negotiation engine, right? It's just that you started with a very, very small part of it. And then that thing kind of grows for a few years and you're like, "Oh, it's not just editing discounts in, it's also editing the shipping costs, it's also editing another product onto the order that they forgot to add." And before you realize it you're like, "Oh my God, we're reinventing the checkout over here," right? And by the way, in the meantime all the teams that work at Shopify are, uh, finding themselves having to build to support both of these things, right? So someone builds a new feature and they're like, "I'm gonna add, I don't know, international pricing," and then that team is like, "Oh shit, I have to build it into the checkout but I also have to build it into that order editing thing. Oh my God, and I also have to build it into that invoicing thing." And now it's like, to your point, Elade, it's like, okay, now I- every- the thing that should've taken a month to build is now taking six months to build because the layer of the stack that's underneath me has five things where there should just be one, right? And so sometimes it's hard to know what to do, right? Because sometimes the right answer is let's collapse those five things in the stack into a single thing, which is it sounds like what Google tried to do. But then you're like, "Oh my God." It's like trying to take the foundations of a house and, like, rearrange them while the house is still on top and, like, that's, "Oh God, this is gonna be really hard because I- I can't just make everyone move out of the house," right? The- the real answer to how do you solve this problem is one is, um, developing extremely good intuitions for when you are accidentally creating duplicates at the same layer- layer of the stack so you just never get into the problem in the first place, and then the second part is being willing to actually notice when you have created the duplication and eat the vegetables of saying like, "Oh shit, whoops, we did it." Okay, now it's time to eat the vegetables and collapse these things together because if we don't every team that exists above us in the stack is gonna pay this duplication cost, like, in perpetuity. And the problem is it's not even just your R&D teams that pay the cost. The saddest thing is that usually it's a combination of your R&D teams and your customers paying the cost because the reality is that they u- not everyone will do the thing of building the five versions, they'll probably build like the top two or three most important ones and then all the customers who depend on numbers four and five will go to use the thing and they'll be like, "Oh this is super weird, this thing doesn't work. Why doesn't- why is this thing so different to the other thing? I thought they were the same thing." And then they look under the covers and they're like, "Oh, I know why it's different. Because actually this part of the stack is all fucked up."

   6. EGElad Gil12:12

      How have you approached that from a AI perspective?

4. 12:15 – 15:51

   AI adoption

   1. EGElad Gil12:15

      So I think like, um, Shopify was a very early, very early adopter of AI. And in general, whenever there's like a really sharp technical founder driving a company, they've been a earlier adopter, right? And Tobi obviously fits that mold. But it's an area where I think you kind of have to play around with it, experiment, et cetera, to really understand the capability set. But at the same time, to your point, you want kind of centralized infrastructure approaches. So how have you thought about how to do that adoption as well as, um, how, how have you thought about that in the context of launches like Sidekick and Shopify Magic and things like that?

   2. GCGlen Coates12:42

      Ye- I mean, yes, this is the, obviously the space where everything's moving the fastest, right? Like, the world of invoicing is not moving at the pace of the world of AI (laughs) , right? Um, and so this is-

   3. EGElad Gil12:53

      Depends on what you're invoicing, but yeah. (laughs)

   4. GCGlen Coates12:57

      (laughs) Um, yeah, it's, it's been an extremely interesting last 12 months of, like, looking at where can we apply different models to parts of Shopify. I think the high-level idea is, you know, Shopify's real focus and real mission is enabling entrepreneurs, right? And the meta strategy of everything we do is, how do you simplify the process of starting and running a business so that more people who would otherwise not get over the hump, get over the hump and have an opportunity to try building whatever that business is? And, like, the historical strategy for that at Shopify was build really easy-to-use software. Use software that is super simple when you start and it kind of reveals complexity as you need it. Like, as your business grows and you have harder problems, like, the tools sort of, like, appear at the right time. I mean, the world is littered with the corpses of, you know, the companies that tried to serve SMB and enterprise at the same time, and this is kind of the hard problem of Shopify. How do you be simple but also powerful and scale up? AI is this really interesting moment where it's like, okay, well, Shopify built its position in the market largely by being the easiest-to-use thing as you build a business and scale, but in imperative mode, right? It's the, "I'm gonna give you all these switches, and we're gonna give you the best switches. And we're gonna show you the switches at the right moment in time, but these a- this is the best switches," right? And AI brings us into the world of, like, "Okay, well what if we can, um, give you the, the driver in the car who knows how all these switches work and can help you along without having to learn all the switches?" It's probably one of the most powerful opportunities for more people who would otherwise be daunted by having to learn the switches. Even though we try to make the switches as good as possible, there's all these people who can't quite figure out the switches, but they could talk to the, they could talk to the co-pilot, and they could explain what switches they want hit, but they just can't do it themselves. And so, um, I think we have an amazing opportunity f- to increase, like, the amount of entrepreneurs in the world who actually get a shot, right? And so that's the kinda headline, and then from there you go into, "Okay, well which models do we apply to which problems?" You know, you look at individual problems like, okay, how many people get stumped just writing the descriptions of their products 'cause they're like, they're not good copywriters? And they're like, "Oh, this is embarrassing. I can't write a good description for my candle. I'm not gonna launch my store." You know? And there's so many instances like this where it's like, okay, well, can you give someone that extra bit of skill that gets them over the hump where they're ready to hit the big green launch button, you know?

   5. EGElad Gil15:48

      Yeah. That's really cool.

   6. SGSarah Guo15:50

      Glen, it's, it's really

5. 15:51 – 17:33

   Deciding when to ship AI products, evaluations

   1. SGSarah Guo15:51

      exciting how many obvious places there are to apply that to make it easier for the merchant or the would-be merchant, but I think one of the things that has struck me that you and Tobi and the team has done is actually shipped a lot of that quickly when a lot of very large organizations are like, "Oh, that makes sense. The outputs are non-deterministic, um, by nature. We have a process for measuring and evaluating quality of our products. Generally, that process does not apply." Like, how did you get to, "This is good enough and we can ship it?"

   2. GCGlen Coates16:23

      Well, well, one of the principles that we, uh, currently hold, this might change in the future if confidence intervals go high enough, but one of the principles that we have for all of the Shopify Magic features today is that they're allowed to propose changes but not commit changes, right? So they can, like, generate text, but they're not gonna save it without you actually reading it and saving it. We might suggest a reply for the user that's, you know, someone writes in like, "Hey, what's your shipping policy?" We can, like, suggest the reply based on us knowing what's in the store and, like, running that through an LLM, but you have to hit Enter, right? And so one of the things that w- like, human is in the loop essentially right now is one of the, um, one of the ways that we're kinda mitigating risk here. And obviously human in the loop is great because it gives you the feedback cycle. You actually get a three-part signal. You get which suggestions are e- accepted clean, which suggestions were accepted but then with minor edits, and then which suggestions were outright rejected. And that's an amazing loop to be able to improve things through. So we're always getting better, but that sort of human in the loop, human must click Save thing is a, is a big part of the strategy.

   3. SGSarah Guo17:32

      Do you think that Shopify

6. 17:33 – 18:50

   Shopify’s risk orientation

   1. SGSarah Guo17:33

      has a different, like, risk orientation than other companies of its size? You still need guardrails, and even if they're suggestions, like it's not, it's not perfect, right?

   2. GCGlen Coates17:44

      It, correct, yeah. I think Tobi, Tobi and Kaz and myself, we're all, like, fairly risk-hungry people. I mean, I think that's just a little bit of founder culture is, like, you want to take risks. Like, you don't really wanna be safe. You get bored. You get annoyed when things get too safe. But there's always this balance of, like, we like taking risks that are risks to us. We don't like taking risks that could hu- like, break someone's business. And so if there's a way to do a thing that's, like, very risky for us...... but, but is, the risk is mitigated for the actual merchant. Like, we're very for that. And I think that's where that kind of human in the loop, human must click save thing, is- is part of that, right? And who knows, like, maybe a year from now, maybe two years from now, like, we get to, like, you know, "Hey, you're, you seem to be accepting our suggestions without edits 99% of the time. Do you wanna just go into, like, full auto mode?" Like, maybe we get there eventually. We're not there yet, but, like, you can sort of see that the hill climbing will eventually take us somewhere close to that, right?

   3. SGSarah Guo18:49

      Can you

7. 18:50 – 26:05

   Changing the core Shopify data model, enabling AI features

   1. SGSarah Guo18:50

      just describe for listeners, um, some of the stuff that you think is coolest at additions, if that's, um, you know, image gen or semantic search or, or even any of the foundation's work?

   2. GCGlen Coates18:59

      The, the foundation's work is the thing that I'm most excited about, but I'm a commerce nerd, so, like, this is gonna seem like very boring to the audience, but, like, just humor me for a sec.

   3. SGSarah Guo19:07

      It's a pretty nerdy audience, so it depends.

   4. GCGlen Coates19:09

      Yeah. The data model that is the most, most, most central to Shopify is the products data model, right? Like, how do you represent the products that you're selling? And Shopify has for, you know, basically, like, 10, 15 years had... the products data model hasn't changed that much. It's, like, fairly simplistic. It's actually an amazing testament to how big of a company and how big of a... you know, the fact that you can get to, like, 10% of all US e-commerce on, like, a fairly simplistic data model is actually kind of amazing. But, you know, like, Shopify has, um, been a little bit on the weak side for dealing with, like, very large and complex products, so products that have a lot of options, a lot of colors, a lot of sizes. When it gets complex, we don't do as well, and so we're updating our data model with support for just a much larger number of variants. And again, this is the nerdy part, this is one of these crazy things about tech. The reason this is so hard is because it forces you to go from unpaginated to paginated APIs when you start making the variant counts very large, and it's just a breaking, it's just a breaking change for, like, the entire app ecosystem, and it's just... Breaking changes are hard, right? So I'm really excited about just the fact that we're unlocking that data model. We've actually also embedded in, um, not just like, "Hey, you can have more variants and more options," but there's also now a standard product taxonomy that comes with standard categories and standard attributes so that, like, when a merchant creates, like, um, a, a T-shirt, we're gonna auto recognize, "Okay, this is in the standard category T-shirts, and that category comes with all of these standard attributes," and we're gonna use AI to, one, detect what category it should be in, two, detect what the values should be. So we're gonna auto de- we're gonna try and guess, "Okay, seems like the colors here are green and yellow based on the images you uploaded. Seems like it's made out of cotton based on the description you had." And the great thing about that is when that merchant takes those products out to their own storefront, all of that data is gonna make the search experience better. But more importantly, when they take those products out to, like, Google and Facebook and Amazon and all those places, having that structured metadata around their products makes them way, way, way more discoverable, which is gonna lead to them ranking higher in search results and basically making more money. So this is one of those weird things where, like, getting the data model right and having the data quality be very high actually causes these very, like, important effects up funnel for the a- for the business. So that's really cool. Um, uh, we are also, uh, releasing, um, the first version of our, uh, image, uh, image editing with AI thing. Um, so y- you can take product images, you can replace backgrounds. Basically, you can take, uh, uh, product imagery and get to a very, very, very professional, uh, standard, uh, on it without having to go through, like, very expensive photo shoots. Basically, a lot of what we do is, like, how do you help someone who's, like, actually just, you know, my mom at home trying to start a business? How do you help that person look, like, as professional as, like, the world's largest companies? And then, um... Yeah, what else am I really excited about in the addition? I think you just mentioned something, Sarah. Now I'm having a mental blank.

   5. SGSarah Guo22:26

      Uh, I mean, I just have... Like, I'm a search nerd, and so I think it's, like-

   6. GCGlen Coates22:29

      Oh, yeah.

   7. SGSarah Guo22:30

      ... related. Yeah.

   8. GCGlen Coates22:31

      So here's a, here's the crazy thing th- So I just learned what LBD means.

   9. SGSarah Guo22:36

      Like a little black dress.

   10. GCGlen Coates22:37

       Exactly.

   11. SGSarah Guo22:38

       Yeah. Of course.

   12. GCGlen Coates22:38

       I didn't know what LBD meant until recently.

   13. SGSarah Guo22:40

       (laughs)

   14. GCGlen Coates22:41

       Um, but, uh, very soon Shopify stores will actually understand the meanings of things as you s- So today's Shopify search is very, very literal keyword search-y. There's a video of me going around internally being, like, infuriated when I go to... There's a store that I actually buy things from, and I went on there and searched for, um-

   15. SGSarah Guo23:01

       LBD.

   16. GCGlen Coates23:02

       ... s- not LBD (laughs) . I searched for sweater, and because that store, uh, only lists their thing as sweatshirt, I got, like, zero search results. I was like, "I can't, this cannot still be true in 2023. Like, this is insane that this is happening." So that's a really easy example, but like, you know, if you go to a store now and they haven't put little black dress in any of their keywords, if you search for LBD, it'll actually do the right thing. You can even do things like, um... I saw a crazy example from the team yesterday. They were like, "Um, Christmas-themed shoes," and it would actually correctly go and find all the red and green shoes in the store. So that's kind of an edge case example, but it's like, you, you know, searching for things like, um, something to wear to a wedding, something to wear to the beach, will actually do the right thing, whereas before it would only, it would never do the right thing unless those keywords happen to be in the product descriptions, which will never happen. So that's, I think, again, that's one of those things where it's like how can the s- search experience on someone's tiny little storefront actually be as smart as, like, what you would get on Google?

   17. EGElad Gil24:05

       That's really cool. And are you doing, like, RAG or embeddings or specific technical approaches, like, to that, or...?

   18. GCGlen Coates24:10

       Yeah. Yeah, yeah. So a lot of what we have to do is figure out what the right embeddings are to use, how to fine-tune the models. Like, obviously there's categories in Shopify that are, um, more pr- like apparel, fashion, gift, homewares, um...So spending the time to, um, assemble the datasets and do the, uh, fine-tunings and pick the right embedding models that are best for this category. But it's also been really interesting with, like, especially some of the multimodal models that have emerged in the last, I don't know, two, three months. Um, experimenting with how much, um, how much weight to give, like, the text descriptions versus the images versus the, um, taxonomy attributes, and figuring out what mix of those things generates the best results is... I mean, we're still working on it honestly, but it's a pretty cool and exciting space.

   19. EGElad Gil25:04

       That's really cool, yeah. 'Cause I guess like, uh, a lot of people increasingly, I feel, are adopting GPTV or the Vision Model, which allows you to upload images or understand or interrogate them. And so it seems like there's a lot of information just resident in, like, visual imagery that people aren't really making use of, but now you can translate into a textual understanding, and therefore tie that into everything else that's associated with their merchants. That sounds very exciting.

   20. GCGlen Coates25:26

       And, and it's, uh, it's an amazing thing for the buyer experience. Uh, I mean, like, to your point, like, the buyer experience is really cool when you search for things to wear to a wedding and it does the right thing. But actually think about it from a merchant experience point of view. If you're like, "Hey, I've been running a store for five years and I've got, like, 10,000 products in there, and now you want me to go back and backfill, like, 10,000 products worth of five attributes for a product? Like, Jesus Christ." Being able to actually apply the models to do something that turns out to be, like, a 90% correct guess is an amazing way to help bootstrap people into, like, the current moment, you know?

   21. EGElad Gil26:04

       Mm-hmm.

8. 26:05 – 28:47

   What’s missing from LLMs for merchants

   1. EGElad Gil26:05

      What do you think is still missing from a capabilities perspective to substantiate your perfect AI world for Shopify?

   2. GCGlen Coates26:12

      (laughs) Well, look, I mean, I think the, the exciting and frustrating thing about LLMs right now is that you can get an LLM, like, agent thing, like a Sidekick-esque thing. You can get it to, like, 75% in, like, 10 minutes, and then it's like this brutal hill climb to get it to, like, 95%, like, over time, right? It's just like- it's like catnip for, like, hackers 'cause they're like, "Oh, my God, I just, in 10 minutes I got to a thing that's, like, pretty good." And then you're like, "Yeah, yeah, wait, buddy. Wait, wait for the-

   3. EGElad Gil26:44

      Yeah. (laughs)

   4. GCGlen Coates26:45

      ... next bit. The next bit's real interesting," you know? (laughs)

   5. EGElad Gil26:47

      Yeah, the last 5% I think is the next five years of tech or something.

   6. GCGlen Coates26:51

      Right.

   7. EGElad Gil26:51

      So...

   8. GCGlen Coates26:52

      Right, right, right. And, and the irony is, is like, you know, people talk about, they're like, "Oh, yeah, this LLM is hallucinating right now." And you're like, "Yeah, yeah, you know that's its job, right?"

   9. EGElad Gil27:00

      (laughs)

   10. GCGlen Coates27:00

       "You know that every time it emits a token, it's hallucinating, right?" It's just that you like some of the hallucinations more than the others, right? Um, and so I, I think that's, that's been the challenge for us. Like, it's even in the name, right? Like, Sidekick is, it is supposed to feel like your companion on the journey, like your coach, your assistant, your, the person who's maybe seen- seen this movie before a little bit. Getting it from 75% to 95% is, is the process we're working through, and it's, and it's so exciting because, you know, we're doing stuff on our side where we improve the training sets. We... You know, we're building more and more conversations with feedback from users that help us dial it in more. But then every five seconds, you look around and, like, there's a new model that you can take all your training set an- all your evals, and then it might be another step function. So it's kind of innovation really rapidly happening both inside and outside our R- our R&D team, and you really know... You never know what we- each week is going to b- bring really, you know?

   11. SGSarah Guo28:05

       And you can be pretty optimistic in that, um, like, especially if you look at any one of these problems, I think there's a lot of focus on the core model capability, and there should be, but, like, you're like, "Oh, well, embeddings models are getting better," right? And, um, and people are working on, like, synthetic data generation tools and working on tooling for the entire RAG pipeline, and so I, I think it's, it's... Yes, like, you hit the, the pain of reality of like, "Oh, no, I have to go beyond the demo," but a lot of other people are, um, working on some enabling stuff. So, I still expected a lot of the experiences to get better fast. Counting on it from Shopify.

   12. GCGlen Coates28:39

       Yes. Yeah, yeah. Well, I mean, we're- we're at the cutting edge, you know?

   13. EGElad Gil28:42

       Yeah. Is there anything external to Shopify, um, that you think is especially

9. 28:47 – 33:22

   Most interesting AI developments in the industry

   1. EGElad Gil28:47

      interesting right now in the AI world? Be it startups or things that people are working on or projects or...

   2. GCGlen Coates28:53

      This is literally every single person has probably said this, but I think the, like, the Rabbit thing at CES was pretty interesting. Um, I mean, I'm, uh, I'm literally wearing the shirt right now. Like, I'm a teenage engineering nerd, so like I was, I was hyped on the hardware. But I think one of the interesting problems with, like, LLM-based applications and agents in particular is, like, what's the actual interface they're interacting with, right? Like, in the case of Sidekick, right? There's a couple different places. Like, like, what is Sidekick using, right? Is Sidekick actually using the admin API under the hood? Or is Sidekick actually reaching into the pixels of the web app and clicking around in the web app and doing stuff, right? Like, that's a pretty important question. Like, what is the interface that the agent is actually learning and interacting with? And, um, I thought the really interesting part of the Rabbit presentation was that they decided to treat the actual GUI as the interface, and to try and, like, have a model that became very good at interacting with essentially web apps. If it actually works, the strategic brilliance of that is they instantly have the world's largest app store because the world's largest app store is just the web, right?

   3. EGElad Gil30:12

      Mm-hmm. Yeah.

   4. GCGlen Coates30:13

      Um, now who know- who knows if it'll actually work, but it's- it's an incredibly interesting strategy.

   5. SGSarah Guo30:18

      I was talking to Jesse from Rabbit about this, and I was like, it's a very... It's a controversial, interesting strategic decision to say, like, "I'm going to interact with the applications themselves versus use APIs." And his last company tried to unify actions on APIs, and his view was very much, um, like, "Well, there are implications on the...... uh, like as you said, reach and ability to use different types of data, and, like, whether or not you own your destiny if you are relying on those interfaces. And so, as you said, if it works, that's super interesting.

   6. GCGlen Coates30:51

      Yeah. And it's e- even philosophically and from first principles, it sort of rings true to me, right? Because, you know, a lot of these experiences are, especially with s- like when, when I actually think about the Sidekick UI, the side- Sidekick actually runs Sidecar in the admin next to you, right? Even the name Sidekick is like, "Hey, I'm here with you. You know, we're both here together and we're both using this Shopify thing to try and build your business," right? And so the idea that Sidekick would literally be interacting with the pixels in the same way that you are, it sort of almost, like, smells right in a weird way. Um, and, I mean, obviously, the nice thing about it is, as long as the model can actually learn the UI, then every time you ship updates to the UI, if the model's smart enough, like, it's just like, "Oh, yeah. My smart body keeps... It, it also can use this new button that just appeared today," right? So yeah, I think it's an interesting approach. We'll wait and see what actually happens. I also know people in the industry who are like, "Yeah, yeah. It's a cool idea. It's gonna be insanely hard to pull off." But who knows? Again, this thing changes every week. The other thing I think is kind of interesting, which is actually like Shopify relevant is... I, I guess I thi- I'm curious what you guys think about this, but like the, the sort of, uh, the, the triangle of like Perplexity, Google, ChatGPT of like... It's basically like LLM native search and, like, ChatGPT really good at the LLM interface thing, not amazing at search yet. Google amazing at search, not really figured out the LLM thing yet. Perplexity is kind of squarely in the middle and saying like, "We think LLM native search is a really interesting, like, hill in the middle of this thing."

   7. SGSarah Guo32:40

      Mm-hmm.

   8. GCGlen Coates32:40

      I'm sort of watching this from the sidelines and kind of interested to see where it plays out. I think embedded in that strategic tussle is the question of what do people actually want from search? Do they want hard facts? Do they want opinions? How much lossiness are they willing to tolerate in order to get the compression of the LLM took 10 search results and summarized them for me? But maybe they hallucinated the summarization, right? It's, it's a really interesting place. Like what do people actually want, you know?

   9. EGElad Gil33:16

      Yeah, I think it's a really interesting open question and related to that too, you start... I've, I've started to see, um, the

10. 33:22 – 38:20

    What users want from LLMs and search

    1. EGElad Gil33:22

       degradation of some performance of some of the really early players in the market where there's a lot of qualifications or safety, or you start to see the LLM go out to try and gather information and it just really, in some cases, either slows down or makes the information worse 'cause to some extent, the reason you're interrogating an LLM is you're, you wanna get to an answer. You don't want a web search or you don't want that, you know, poor synthesis of bad data from the web. You want the good synthesis of bad data from the web. And so yeah, it's been fascinating to watch. And to your point, I think a lot of people, um, have started to adopt Perplexity for that specific use case because it has that middle ground of some form of IR plus LLM in a traditional sense. So yeah, I agree with you. It's, it's, it's gonna be fascinating to watch how all the directions this goes. And I think the reality is people almost forgot that people from search, they actually want an answer. They don't, they don't wanna do a search process. They wanna get to a result in many cases or a list of results. It's almost like people forgot the user need in some sense.

    2. GCGlen Coates34:19

       Yeah, and it, it's funny when like, I mean, some searches are so specific, but some are almost like shared knowledge of humanity. So just like one, one thing I found myself doing a lot like a year ago, um, when I was looking at, okay, how can we improve search at Shopify both on the storefronts but also on the shop app and just kind of playing with this idea, one of the tests that I found myself running over and over again... So at the time I actually had, uh, this happen. I had a, a friend's, uh, daughter, it was her birthday and I was like, she was... I think she was six at the time. And I was like, "I wanna buy a great classic book for a six-year-old girl." And as an experiment, I would go to Google and I would like say, "Classic children's book for six-year-old girl." And like the search results you would back, get back on Google Shopping would just be like terrible, like absolute disaster, right? And then you would go to Amazon, type the same thing in, you would get like nonsense back, right? You went to ChatGPT and you said, "Give me 10 great children's books for a six-year-old girl." What would come back would actually be amazing. Like it would be like really like the best 10 recommendations, right? And so I was like, "Huh, there's something here," right now, like ChatGPT terrible if you ask it for, "Hey, I want a pair of Nike AF 1s in size 10.5 in a store in New York City," but really, really, really good at the general knowledge question of like great children's books, you know?

    3. EGElad Gil35:50

       Yeah. It's funny 'cause I worked at Google years ago and then when I was at Twitter, one of the teams that worked for me initially was search and, um, you know, you tend to segment these things as you undoubtedly do at Shopify into, uh, types of search, right? Is it navigational? Is it certain types of information that you're looking for? Is it something else? And so, you know, you could imagine that in the LLM world, you effectively wanna do almost like one boxes like you have at Google where you trigger off as certain types of keywords or phrases and then therefore you end up with a different result. And you should be able to especially have integrations with search engines to be able to serve the "I want these specific Nikes and where should I get them," right? 'cause that, that's almost like a form of navigational query in some sense, relative to a commerce objective versus a, just give me some knowledge. So it feels to me like a lot of stuff will come as long as people don't actually think that LLM has to do everything, unless it can also function as a router or something. So it's really fascinating to think about.

    4. GCGlen Coates36:39

       Yeah, and, and then that's when you get into the world of, like, things that are, like, matters of fact and things that are matters of opinion, right? Like, "Is this shoe a Nike?" is not a matter of opinion, right? Whereas, "Is this book great for a six-year-old girl?" That's a matter of opinion. And how do you construct search systems that can actually sit correctly at the midpoint of the world of facts and the world of opinions, right?

    5. SGSarah Guo37:05

       I, I think also going back to Elad's point on, um, use cases, like, one version of this, um, like, eh, from a scenarios of the future perspective is it fragments, right? Sometimes I want facts and sometimes I want opinions, especially opinions like my own already.

    6. GCGlen Coates37:21

       (laughs)

    7. SGSarah Guo37:21

       I mean, like, a date on how people consume media definitely looks like that, right?

    8. GCGlen Coates37:26

       Yes.

    9. SGSarah Guo37:26

       And, and so I think it's very easy... I mean, I was a investor in a search company called Neeva and I think very highly of the Perplexity team. I think it's very easy to look at the chokehold on distribution that Google has through its partners, uh, and say, like, "That's very hard." But we're at this very, I think, special moment where the technology and perhaps the, like, user behavior, the, the expectation, if we take out, like, slices of it where the expectation is very different and get people to think about pieces of search... 'Cause it's not at all clear to me it should've been one market to b- Or it's permanently one market, right?

    10. GCGlen Coates38:00

        Mm-hmm.

    11. SGSarah Guo38:01

        You guys probably think about owning commerce search, um, and so...

    12. GCGlen Coates38:04

        Right, but even within commerce there's so many different... Um... Like, the way people search for, for clothes and the way they search for industrial parts are, like, not the same thing at all, right? So it's... Even within that world, there's- there's so much nuance of the way

11. 38:20 – 38:48

    No Priors social

    1. GCGlen Coates38:20

       that people express what they're looking for.

    2. SGSarah Guo38:23

       Glenn, awesome. Thank you for being here. That was great.

    3. GCGlen Coates38:26

       Thanks, guys. Appreciate it.

    4. EGElad Gil38:27

       Good to see you. Thanks so much.

    5. SGSarah Guo38:29

       Find us on Twitter at No Priors Pod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.

Episode duration: 38:48