Alex Lebrun: Why the EU's AI Regulation is a Disaster; How Zuck Prepares for Meetings | E1027

Harry Stebbings 0:00 Will AI replace doctors?

Alex Lebrun 0:02 So AI will not replace doctors, but doctors who use AI will replace doctors who don't. (instrumental music)

Harry Stebbings 0:10 Alex, I am so excited for this. I've been looking forward to this one for a long time. I've been thinking with a couple of the AI shows, I can't wait to do this in person with Alex. So thank you so much for joining me today.

Alex Lebrun 0:19 Thanks, Harry.

Harry Stebbings 0:20 Now, I would love to start with a little bit of context because we're three startups in at this point. So take me back, how did you first make your way into the world of startups?

Alex Lebrun 0:29 So, um, 22 years ago, I, I fell in love with a chatbot. Her name was Sibyl.

Harry Stebbings 0:35 (laughs)

Alex Lebrun 0:35 And really, really, I was, you know, summer night in Paris in my basement alone with my computer. And I, I, I, I found-

Harry Stebbings 0:42 This could go in many directions. (laughs)

Alex Lebrun 0:44 (laughs) Right. Up to you. Um, and, uh, I was mesmerized by this chatbot, you know, the fact that the machine is trying to understand you, language, and can generate some words. And I, I was really, really struck by this thing and I decided I would, okay, I would spend my life building, uh, chatbots. And so I founded, um, a company doing customer service bots 22 years ago, very early, um, called Virtuozz. And this is how I started, um, you know, this series of companies in the, in the domain.

Harry Stebbings 1:16 Can I ask you, given it was 22 years ago, and I know this is off schedule straight away, but just how do you think about market timing today?

Alex Lebrun 1:23 Well, for, for 20 years, you know, when, when we started, uh, I thought chatbots would become very, very intelligent after three, four years and get to AGI and replace humans in call centers. And the more I worked on the problem, the more I realized it was really, really difficult to do that. Actually, the first time we released a bot in Europe was in '24, uh, 2004, to the French railway company. And we were very, very happy to, to have this customer, and the deputy CEO tried it. And she, she asked, "My name is wrong on the, on the reservation." And the, our bot answered, "Hello. Wrong on the reservation." And then we realized, okay, there is still a lot of work. (laughs) It's not smart at all. And so the more I worked on these things, the more I realized it's not ready yet. And so suddenly for the last year, two years, uh, we reach a point where market timing might, might be finally, um, finally good, you know, where the product is ready and people have evolved too, maybe 10 years after Siri, people have changed-

Harry Stebbings 2:24 Mm-hmm.

Alex Lebrun 2:24 ... and they, they are ready to meet this kind of technology. So maybe the market timing is now.

Harry Stebbings 2:29 So if we project forward in thinking of market timing, how did Nabla come to be, and what was that founding moment?

Alex Lebrun 2:35 So the founding moment of Nabla, I was sitting at, at terrace at Facebook, um, headquarters in, in Menlo Park. You know, we joined Facebook through the acquisition of my second startup, always in the, with the AI in this field. Um, and sitting at this terrace with like cocktails, looking at the sunset, suddenly it reminded me of a scene in Silicon Valley. You know the show?

Harry Stebbings 2:58 Yeah, I love it.

Alex Lebrun 2:59 Where they go on the roof, uh, at Google, I mean Hooli, and there, there are these people doing barbecue and say, "Yeah, we, we are vest and rest." (laughs) And, and I saw myself in this vest and rest situation, and after four years at Facebook, uh, I suddenly realized, okay, it's time to go out and back to the arena. And we've learned so many things at Facebook AI Research, it's time to try to push these things to the real world.

Harry Stebbings 3:26 Can I ask, what

Harry Stebbings 3:27 would you say are the biggest takeaways from your time at Facebook and at the AI Research kind of lab?

Alex Lebrun 3:32 So first thing that amazed me when I arrived at Facebook is I, I thought all big companies were slow. Um, and actually when we arrived at Facebook in 20, early 2015, there was maybe 6,000 employees. And it was going so fast, you know, it was a machine, a very well-oiled machine. Um, engineering was so efficient and I, I, I, I list-, I learned it's possible to be that big and very, very efficient. And when you are in this position, you can, there is nothing you cannot do. Um, just after I joined, I started to work on a project and I wanted to hire 200 people to, uh, human concierge, to train my bots (laughs) in real time. It was, it was not in the budget, of course. And, but a week after I had a meeting with Mark and, and in, in about 10 minutes, Zuck, you know, challenged a little bit my idea and he said, "Okay, let's do that." And I could hire 200 people in Menlo Park. And when you have this, um, this speed and agility and, and the like, unlimited resources of a company like Facebook at the time, uh, so I was really shocked by this, uh, at Facebook.

Harry Stebbings 4:39 What did you learn from working with Mark? You mentioned that meeting there and the speed of decision-making from him. Is there anything that you took away from working with him?

Alex Lebrun 4:47 Many things. Um, one thing I like, he spent not a lot of time outside the company, you know, he had like 30 minutes meetings back-to-back from morning to evening. Um, and he prepared really well every meeting. So you have to send in advance like a short note about why, what decision you are expecting from him. If you fail to send this document 24 hours, you know, by the minute before, your meeting get canceled. And so in many cases, you arrive in front of him, he's read your document, he's, he's gathered all the information, the data he needs. And so the, the meeting is fast, very fast, you know, he will challenge, um, your, what you ask for maybe, and, and make a decision. So this way it work, uh, I'm, I'm trying, you know, I think there is much to learn, (laughs) but it, it influenced me.

Harry Stebbings 5:34 I, I totally get you. I love that in terms of structuring his thought process before the meeting. (laughs)

Alex Lebrun 5:39 I'll give you an, uh, an example of that. Um, when I wanted to hire my 200 human concierge, um, my, my reasoning was we will train this AI and it will, uh, work really well and we will steal business from Google in terms of search result, like exactly what ChatGPT may do in the, in the, in the, uh, in the time to come. Um, and I prepared like answers to every possible objection.... he could, he could object of, about everything I was ready to, to say why my idea would work. And I arrive at the meeting, and first minute he tells me, "Alex, I agree with you, it will work. So instead of hiring 200 concierge, let's hire 100,000 and we will kill Google tomorrow." And then I, I completely reverse my course and say, "No, no, no, Marc, w- I'm, actually, I'm not sure if it will work. It would be crazy to hire 100,000 people." (laughs) And I, I, I gave all my reasoning on, on the opposite sense like, "We should be careful. This thing might not work," and so on. So very smart of him. I don't know if it was on purpose, but I had to e- e- learn more about th- the true, the truth of my projects, attacking it from this direction, this angle. And, um, it was, it was very, very interesting.

Harry Stebbings 6:53 That's such a good way to reverse your thinking.

Alex Lebrun 6:56 (laughs)

Harry Stebbings 6:56 Can I as- and we mentioned kind of Nabla being your third company. What worked with the prior ones that you've taken with you? What didn't work that you've left behind? And does it get easier?

Alex Lebrun 7:06 It's not getting easier. Um, you just, y- I mean, you learn some lessons. You don't do the same mistakes again, hopefully. But you, you get new, new dangers, you know, new, new issues. Um, the one I suffered from when I started Nabla is what I call the Kim Jong-un, uh, you know, entourage-

Harry Stebbings 7:25 (laughs)

Alex Lebrun 7:25 ... trap. (laughs) And the fa- you know, you, you see this, this video of Kim Jong-un, and he's visiting something, and all the generals are around him with a notebook, and everything he say will be written down. If you fail to do that, I, I guess you disappear the next day. And, um, nobody of course, will ever, ever say, "I, I disagree with you," or challenge any of his decisions. And, uh, starting my third company after two exits, I felt like my investors were always agreed, my team always agreed, um, people around us. And so I wasn't challenged enough, and we made some early mistakes, uh, that probably, uh, um, were not challenged enough.

Harry Stebbings 8:06 What were some early mistakes?

Alex Lebrun 8:08 Uh, so when we decided to, um, to start in healthcare, uh, we were a v- en- engineering team, you know, very strong in AI and engineering, but not so much in healthcare. And we made actually a good decision to start as a B2C healthcare company to learn. So we, we started a clinic. So I think this one was a good decision, but like in, it's an expensive decision. B- um, and we started this B2C virtual primary care clinic, and we learned everything like that. But then we lost our, o- o- our way a little bit, and, uh, in terms of product, of direction. We went too far before realizing that the, m- it was not the right path, and that we should go back to a B2B business model. We probably took too much time to, to make these decisions. Just an example.

Harry Stebbings 8:57 Can I ask you, what do you do then to stop the Kim Jong-un paradox or problem? (laughs) Like, how do you surround yourself then with people that do challenge you? What did you change?

Alex Lebrun 9:07 So, so first with external... The, the solution first is, I think is with, uh, external people. You know, even my investors who are too close to me, but if you go to people further away, they have l- less, and, and who have, who are very successful, and they have no incentive or, you know, to please you, and you get more, uh, information, more data from them. And internally, it's a matter of, uh, just telling your team, "Okay, um, you, you are, you should challenge, uh, our decisions." And we, we did that a lot at Nabla, and sometimes too much. You know, they challenged everything after that. Uh, also, after you make your, one or two mistakes, the team will more naturally come forward. Uh, and this is, uh, before you reach, uh, the right level of, uh, challenger.

Harry Stebbings 9:50 It's funny, one of my friends is Gustav Soderstrom, who's the CPO at Spotify, and he says, "Talk is cheap, and so we should do more of it." (laughs) Which I like as a saying. Are you... We mentioned market timing. I wanna start on kind of where we are in terms of, you know, uh, the landscape

Harry Stebbings 10:05 itself, and we look at the developments today. And I think ChatGBT has brought around kind of consumer excitement to a level that we haven't seen obviously in AI for years and years. Have there been fundamental developments in the last 18 months technologically that have led to where we are today with our excitement level? Or is it the continuation of years of behind-the-scenes development from amazing people like you?

Alex Lebrun 10:29 So e- fro- from the, from the, the outside, from, for the general public, it looks like a very big step function with huge advancements every 10 years. Um, I think from the inside, it's much more continuous. Um, so for instance, ChatGPT, you know, is based on GPT-4, which is based on GPT-3, was released three years ago. Uh, GPT-3 is a large language model. It was invented m- before that. Transformers, where the paper was released in 2016. Um, and so the progress is continuous, I think, more or less. Um, but the public perception of it is, is very, very discontinuous.

Harry Stebbings 11:09 Can I ask you, what do you make of the VC hype cycle? Like now, everyone, like, you know, I did an investment the other day, and it's in an enterprise SaaS company. Nothing to do with AI, and I genuinely felt like a true contrarian adventurer (laughs) by not doing an AI deal. What do you and what do the teams around you and the kind of true AI OGs think when they say, "W- see all the VCs just chasing everything AI?"

Alex Lebrun 11:33 Honestly, for, f- from our standpoint, it's, it's really ridiculous. (laughs) And I think, I've been, you know, I've been building AI companies for, uh, 22 years, so I- I've seen this cycle several times. So after a while, you're not surprised anymore. Uh, when I first raised money from my first startup in 2002 or '03, it was already, there, there were two cycles before, and it was in the AI winter, uh, timing, where in 2002, if you have AI in your deck, you, you will never raise anything-

Harry Stebbings 12:04 (laughs)

Alex Lebrun 12:05 ... because they lost money at the end of the '90s. And then, um, 2005, '06, you had to put back AI in, in your deck, and, and so on and so on. Uh, there was a huge, um-... bubble in tw- 2012, when deep learning, uh, came out as better than, uh, other techniques, and then in 2016 with the chatbots, when we joined Facebook.

Harry Stebbings 12:24 I remember the chatbot bubble in 2016.

Alex Lebrun 12:26 So it's-

Harry Stebbings 12:26 That was big for three months. (laughs)

Alex Lebrun 12:28 As an entrepreneur, the only thing you need to know is when to add or remove AI from your deck, and this will change about every three or four years.

Harry Stebbings 12:36 (laughs) I love that. Um, can I ask you, when I used to be into ambassadors, and, you know, obviously as an ambassador today, I, I'd probably say this, but, you know, the common criticism, especially of generative AI, is, you know, bluntly, it's a thin, relatively valueless layer on top of kind of foundational models that are open to everyone. Is that fair or not when we look at generative AI applications today?

Alex Lebrun 13:00 I, I don't think it's fair. Um, you know, when, uh, the C language came out around 1972, uh, some people said, "Hey, now it's so easy to build software, you know, in C compared to Assembly. Every software is just a thin layer on top of C." And of course it's not true, and we know it. (laughs) It's the same when databases came, uh, you know, in the early '80s. Uh, so I think it's, it's really, uh, not fair to think that any AI application on top of LLM is just a thin layer. Um, it's, LLM is a new kind of resource. It's like a new kind of infrastructure that everybody can access, sure. But then it's completely novel. Um, it's very hard to control. You have hallucinations. It, it's non-deterministic. Um, it's very hard to, to configure. You have several knobs you can change, you know, between, uh, uh, fine-tuning or, uh, how you prompt it, and, and, and many other ... You can do reinforcement learning and so on. So, um, it's very hard to control, and also it, it changes every week. You have new LLMs coming out that claims to be better than the one from the week before. How do you decide when to change your LLM? How do you do it without breaking your existing users and product? You know, so there are rules. There, there is a new game. There is a new infrastructure, there is a new game, new rules, and the companies who are the faster to understand these rules and, and, and play a- along these rules will win, and, and build the best products.

Harry Stebbings 14:33 So if we have two ... Sorry, I'm just trying to understand. So if me and you start a new startup today in the travel and expense management category, and we're using fatne- uh, whatever model we choose to use, do, do you have an advantage over me if we're both using the same model?

Alex Lebrun 14:50 I think I have, because I know the, I know how it works internally, so I know the limitations. When something is wrong, I know where I should poke, you know, to find a solution.

Harry Stebbings 14:59 Mm-hmm.

Alex Lebrun 15:00 I also know when there are many, many, and more and more LLMs available, which one will probably be the better suited to my p- problem. Um, and I know what kind of machine learning I have to build around my LLM to, to, um, balance for the, the, the, the weakness of it.

Harry Stebbings 15:19 Sorry, just so I understand. So will we see companies switch between LLMs very frequently? Is that ... Like, what, is that how it works? Sorry, I'm just trying to understand.

Alex Lebrun 15:30 Y- it's how it will work, of course.

Harry Stebbings 15:31 Yeah.

Alex Lebrun 15:31 Um, yeah, yeah, I mean, every two weeks, incredible things coming out of research. And if you just assume it's aesthetic, in, in a year from now, your product will be very dumb compared to your competitor's products.

Harry Stebbings 15:44 Is it-

Alex Lebrun 15:44 Or, or m- more expensive to run or very slow.

Harry Stebbings 15:48 Uh, we had, um, Emad on the show from Stability, and he said h- You mentioned hallucinations. He said hallucinations are a feature, not a bug. I didn't really get this. (laughs)

Alex Lebrun 16:00 I, I don't know what you mean. (laughs)

Harry Stebbings 16:03 D- do you understand that?

Alex Lebrun 16:05 I mean, by construction, the LLM has to output something, and so if, if there is nothing to say, uh, it will build something. It will make something up that looks natural, so this is probably what Emad meant. By, by design he, he cannot not output something, and so this would be a hallucination.

Harry Stebbings 16:26 And you also mentioned kind of the changing nature of LLMs and models. Um, he said that no models today would be used in a year. Do you think that's right?

Alex Lebrun 16:37 Absolutely agree with that. You know, will you drive the car you drive today in 10 years? I don't think so. And 10 years in car industry is like one week in, uh, machine learning gravity zone. (laughs)

Harry Stebbings 16:50 (laughs)

Alex Lebrun 16:50 And so there is, of course there is so much progress so fast that I don't see why we would use the same in one year.

Harry Stebbings 16:56 Got you. We, we mentioned your proprietary knowledge, if me and you were to start a startup in the same space at the same time. I clearly obviously agree with you in terms of winning and having an advantage over me. But so many investors are like, "Ah, but it's all about the proprietary data that a company has that they can leverage on top of an existing model."

Harry Stebbings 17:15 How much of an advantage is it to have existing proprietary data, say being a four to five year old company with four to five years of customer data, versus being a new startup with no existing proprietary data?

Alex Lebrun 17:27 I think VCs are always one train late. (laughs)

Harry Stebbings 17:30 (laughs)

Alex Lebrun 17:30 But s- so proprietary ... Having a lot of pr- proprietary data was very, very important for the l- last, uh, cycle five years ago. Maybe it's, it's less and less true. Um, you need some to bootstrap your models. So for instance, at Nabla, we acquire, you know, we, we build a dataset of, uh, 30,000 consultat- medical consultations, uh, with patient consent. We have to pay doctors. It's very hard to build this, because we need this to bootstrap our product. But then, uh, w- with the new, you know, pre-trained models, uh, fine-tuning is very efficient, uh, with a little bit of data. There is a ... I don't know if you've heard about, uh, LIMA pro- model that came out three weeks ago. Um-

Harry Stebbings 18:12 Can I, can I actually just interrupt you and just ask, for those that don't know,

Harry Stebbings 18:17 um, forgive me for the base questions.

Alex Lebrun 18:19 (laughs)

Harry Stebbings 18:19 Um, what's pre-trained and what's fine-tuning? Just so people understand nomenclature.

Alex Lebrun 18:25 Uh, so large language model is trained, uh, in two phases.... first, first phase is unsupervised, uh, pre-training. So it's just fed with huge amount of text and it's trying to, to predict the next word, basically, the next token's the next word. And so this is what we call unsupervised, uh, pre-training. Unsupervised because you don't need any human intervention. The, the text by itself is enough because it will try to predict the next word and on and on and on. Um, and then the second phase is, uh, fine-tuning is, uh, when you use different machine learning techniques, uh, so for instance, reinforcement learning for, uh, ChatGPT, where, um, you train your model to, um, to follow instructions. Like from ChatGPT is when I s- ask a question, you should tell me something that looks like a right answer or s- looks like the, a reasonable answer to what looks like this question. Uh, and so the fine-tuning is the second phase where you give more precise instruction on how it should behave.

Harry Stebbings 19:29 Mm-hmm.

Alex Lebrun 19:30 And, uh, to do that, yeah, there are several techniques. Um, and what, uh, recent papers... So everybody's still learning, you know. It's like huge ba- black boxes, these things, and everybody's still discovering new things. But, uh, three weeks ago, there was a, a paper, uh, about, uh, LIMA. So, so this, uh, this LIMA paper shows that, uh, with, uh, only 1,000, uh, question and answer examples, so very, very small data sets, um, they get something for fi- use for fine-tuning. So the second stage. They get something that performs better than GPT-3 and almost at the level of GPT-4, uh, with only 1,000, uh, QA, you know, for fine-tuning. So the, the thing with LLM is it looks like a, a huge pre-training with text. It, it, it's learning mostly through pre-training with huge amount of text and then with a small amount of data, with high quality data, you can get a lot, you know, you get... you can get what you need, uh, and train your model to do what you expect with very small data sets.

Harry Stebbings 20:42 Can I ask, how low does that go? You mentioned a thousand Q&A there. Small. Like, does that get to a hundred or does that reach some asymptotic point where it's like, "No, a thousand is as small as it will go"? I'm just intrigued.

Alex Lebrun 20:56 It f- it's very hard to, to answer. Um, probably depends on the domain you are, you are teaching your model. Probably no, you know, won't go less than a hundred or something like that because you still need to give in, you know, information... You cannot compress information, uh, infinitely, and you still need to tell your model what you want and so you... There is a limit somewhere. I don't know where. (laughs)

Harry Stebbings 21:18 Mm-hmm. Can I as... Uh, and we mentioned models there and we mentioned kind of size of models there and actually, uh, how you only need actually very small amounts today. Will there be new foundational model companies created do you think, or do you think we have the existing incumbents already?

Alex Lebrun 21:33 So, we probably have most of the existing companies. Um, maybe a few will, will be getting started. Uh, there is a lot of hype and it's public, so I can spill the, the bean about Mistral, a new one created by, um, uh, three engineers, two of them from, from Facebook. And I know them and, you know, you can, you can still build this foundational model, but you need-

Harry Stebbings 21:59 How can you... Sorry, I... I know them too and they're fantastic. But how can you do it without being so far behind?

Alex Lebrun 22:07 So to, to make, uh, a, a successful foundational model company you need incredible scientists and a huge amount of money.

Harry Stebbings 22:16 (laughs)

Alex Lebrun 22:16 And if you check these two boxes, you, I think you can still do it. Um, and you need focus. Uh, you know, the, the... I k- I know Yann discussed that and I totally agree what he said here, that Google and Meta could have done what OpenAI did, you know, easily and they didn't because it was not a priority and there is all the legal, uh, people around and... But, uh, I think a new startup with enough talent and enough money, and when I say enough money I'm talking about billions, just to be clear, then I'm sure you can do a, as m- at least a- as, as well as OpenAI.

Harry Stebbings 22:53 Can I as... We, we mentioned incumbents, we mentioned startups, they're kind of encroaching on incumbent space. I think the big thing that I think about is like, Bluntney, who wins in this next wave? Is it startups, uh, creating amazing new products leveraging foundational models? Or is it actually Adobe? Is it actually Apple? Is it Google with Bard? How do we think about where value accrues to startup or incumbent? With that wonderful movement of hands.

Alex Lebrun 23:21 (laughs)

Harry Stebbings 23:22 (laughs)

Alex Lebrun 23:22 So incumbents have a huge advantage through distribution. That's uh, w- one thing that is hard for us, uh, as startups. Um, but, uh, incumbents have many, uh, disadvantages. I think first, they are very slow. And you, you mentioned Adobe, I was sure you would because this is the one example that everybody has, but then who else is, is, w- was as fast (laughs) as Adobe? So I think most of the incumbents, um, won't move, uh, fast. And i- in many cases-

Harry Stebbings 23:54 Do you not think they're moving faster? I mean, I- I don't know if you call Notion an incumbent, but like Notion have moved very fast, Adobe moved very fast, Navan, the travel expense management company now is like solely on OpenAI. I think actually they have moved very fast. And do a- argue back with me, I know nothing.

Alex Lebrun 24:12 Hap- No, no, that's, that's fair. You know, some of them can move fast and it's, it's not fair to say they are slow and that's it. I think that incumbents suffer from problems because in many cases, th- they will do, you know, AI enhanced features.

Harry Stebbings 24:26 Mm-hmm.

Alex Lebrun 24:27 And this is what all the one you mentioned-

Harry Stebbings 24:29 Yeah.

Alex Lebrun 24:30 ... did. You know, the, the Notion, you still have a document, you have edit, you have a cursor. You write, "Hello." By the way, you can call ChatGPT and to, to, to summarize a paragraph is what I call spreading a li- a little bit of AI dust on the, magic dust on your existing product.... who knows how differently you can think about, um, building knowledge for your company. It- it- it probably something will come and destroy Notion and Google Docs and all, all of them with a totally new paradigm that is made possible by AI, and certainly these incumbents won't, won't do it. So I think the, the best incumbents can benefit from AI to, uh, to be competitive, you know, in their existing markets, but I don't think they will invent totally disruptive things that will kill them. You know, Kodak, Kodak invented the digital camera, and of course they never released it because it would kill their main revenue, which was, uh, the, the films. And-

Harry Stebbings 25:32 Do you think that's why Google didn't innovate in the way that they could have done, because actually the cost per query, doing it, you know, bluntly like ChatGPT does is so much more expensive than the way that they do it today? And actually it would have cannibalized their whole business-

Alex Lebrun 25:47 (laughs)

Harry Stebbings 25:47 ... if they were like, "Hey, let's embrace this new cost structure, which is so much more expensive."

Alex Lebrun 25:52 I think the reason is, is, is simpler than that. Uh, e- the reason is nobody could predict that LLMs would be so useful and, and powerful before you train one at, at this scale, and who in the Google org chart had the incentive to invest 500 million dollars and just to see this without any business benefit for the company? If you add to that the, the legal department who is not that happy that you are, like, releasing random chat bots who can say anything-

Harry Stebbings 26:24 (laughs)

Alex Lebrun 26:25 ... um, then nothing happens. And this is what happened at Google, and probably also at Meta where nobody has a, the incentive to, to do that. And so OpenAI, a- as, as a private company, they need to, you know, to show something, to build products, and they have this, this curiosity and the financial power, and they did it.

Harry Stebbings 26:45 Can I ask, when we look at the different, uh, approaches, there's the open approach, there's the closed approach. We mentioned some of the biggest names there. You mentioned the Yann LeCun episode. Um, he obviously is a big proponent for open. Um, how do you think about which approach dominates over the next few years, open

Harry Stebbings 27:03 versus closed?

Alex Lebrun 27:04 Open, obviously. (laughs) Uh, no, LLM are, AI in general is an infrastructure in the future. And, and like every infrastructure, I'm just paraphrasing Yann, but is, is open, uh, open wins always with infrastructure. Uh, and, um, so obviously I think the financial mo- the foundational model, uh, that will win will be open. But that being said, you know, we should be careful because even an, an open model trained with open data to me is not that open, because when you have like 300 billion parameters and it's a huge black box, you don't understand why the output is what it is, is it really open? And so I'm just putting a little bit of salt on, on what I hear on the like, open models are you understand everything, it's explainable, it's predictable. It's not true. You know, open, an open model can be as hard to predict, uh, tha- that and that a closed model.

Harry Stebbings 28:05 Is there anything that can be done to change that then? And is that a problem?

Alex Lebrun 28:09 Sorry?

Harry Stebbings 28:09 Is that a problem that it's as hard to predict as a closed model? Is it... yeah.

Alex Lebrun 28:14 It's, it's a, it's a, yeah, it's a very good point, Raymond. And when I mentioned the, the limitations of LLMs and why when you do a real world application, and you know, it still works in healthcare, you need to work a lot to, uh, circumvent these

Alex Lebrun 28:27 limitations.

Harry Stebbings 28:28 Can I ask you, in terms of ... we mentioned kind of data sets before, and I, I did just want to touch on it 'cause I think it's important. You know, and I am actually going back to Iman's episode, but he said about the importance of national data sets. And actually for certain things, I think it really is actually very clear that that might be the way, whether it's, um, neobanks, you know, Western economies are very different to, you know, Eastern economies. Um, healthcare, consumer healthcare, you know, if you have sunlight exposure, it's very different to if you have no sunlight exposure, or diets or whatever. Do you think we'll have national data sets moving forward given how different nations have such different data?

Alex Lebrun 29:08 I'm not sure about national data sets. I mean, if you want to sell something to a nation, it's good to say that. But, uh-

Harry Stebbings 29:14 (laughs)

Alex Lebrun 29:15 ... I'm more... I, I think like industry, you know, per industry, per vertical, as I said, of course will improve the quality of the model. So if you, if you train an LLM only on medical data, like we did with Pan, you get a better... with MedPan, you get a better LLM for medical applications. But even you have to understand, I think not, uh, many people make the mistake, even if you feed an LLM with curated data, you don't, it does, it's not guaranteed that the output will be, uh, will be good, will be perfect, that you can trust the output. So feeding an LLM with trus- trusted data do- doesn't make the output trustable because of how LLM works, you know? Uh, uh, so there is... there, there are lots of misconception about this.

Harry Stebbings 30:01 Help me understand that then, 'cause I think a lot of people uns- will misunderstand that. So you have great high quality, trusted data. What leads to then high quality output and good decision and outcome?

Alex Lebrun 30:13 So an, an LLM is a huge probabilistic machine. It's like an autocomplete, I think is-

Harry Stebbings 30:18 Mm-hmm.

Alex Lebrun 30:18 ... the, the best image. Um, and it will complete the beginning of your sentence at any cost. It will always complete it. And you may have like 10 facts, 10 sentences, let's say, in the input data set that are true, but maybe the LLM will start with the beginning of the first sentence and switch to the end of the se- another sentence just because it looks good. And, but logically, the, the, factually this output will be totally wrong. And I, I simplified a lot, but this is exactly how it works. Uh, so-... you may have, and the form is perfect. You, you, it always, always looks like, sounds like a very good answer, but the, the, the reasoning, the, the facts may be totally off, invented, or wrong. And even if the input that I said is, is wrong. So when people say, "ChatGPT gave me a wrong output because it was trained on Reddit, and Reddit is, is, uh, a lot of noise." Yes, but only partially. Even if you remove Reddit from the, uh, training data set, it doesn't mean you won't have, like, random answers.

Harry Stebbings 31:26 Yeah, no, I totally get you. Can I ask, uh, we're g- we're kind of jumping around, um, from Emad to Yann, and picking different points, but Yann said in particular that it's a big jump to assume there's a correlation between intelligence and the desire to dominate. When we think about the most common question being the fear of AI really overcoming human power and dominating us, how do you think about Yann's statement about that correlation? Do you agree and, or disagree? And how do you think about that?

Alex Lebrun 31:55 Uh, I fully agree with Yann, and not just because he was my, my boss, my boss at Meta, but... (laughs)

Harry Stebbings 32:00 (laughs)

Alex Lebrun 32:01 Yeah, I think if you really understand how machine learning works, and you're not looking for free publicity, I don't, I don't see why you would say something like that, that, that getting more intelligent will make them want to kill humanity. I, I don't even understand the paths leading to this thinking.

Harry Stebbings 32:19 Can I be blunt? Why does someone like Geoff Hinton then feel that?

Alex Lebrun 32:25 So I, I, I'm not in the head of, uh, Geoff. I, I don't know.

Harry Stebbings 32:29 (laughs)

Alex Lebrun 32:29 Um, I, I don't know. And I, I know Yann is also surprised and, um...

Harry Stebbings 32:33 Yeah.

Alex Lebrun 32:34 Yeah.

Harry Stebbings 32:34 Do you think Elon Musk's decision to be very proactive in terms of the petition to pause AI development, how do you think about that?

Alex Lebrun 32:44 I think it's very hard and, and dangerous to try to comment and understand what Elon Musk is saying. (laughs)

Harry Stebbings 32:49 (laughs)

Alex Lebrun 32:49 In general. Uh, but, um... And it's weird because he, at the same time he said that, I know he was, he's trying to build a team to compete with OpenAI. He hired somebody from DeepMind, like, the same day where he announced we should pause for six months. And it looks to me that the people who are proponent of this pause, or to more regulation, are the one who feel they are in advance. And so it's like a way to say, "Guys, l- let us, we are in front." (laughs) "I don't want more people to start the race." Maybe I'm, uh... (laughs)

Harry Stebbings 33:20 I love that. Can I ask you, you know, you've been really in this space for decades, if you don't mind me, uh, slightly aging you. And there's a lot of people entering who are also very vocal. (laughs) We were talking about it before. How do you feel about that? And I guess, how does it make you and the community around you feel who've been actually working and doing the hard yards for decades?

Alex Lebrun 33:42 So on the one hand, uh, we are proud because we feel, okay, w- I was right before everybody else. You know, I was talking about chatbots to my parents 23 years ago. (laughs) And, uh-

Harry Stebbings 33:55 And they were going, "What the fuck?"

Alex Lebrun 33:56 And, and 20, 22 years ago, they, they discover ChatGPT and say, "Okay, we finally understood what you're, what you are trying to do, Alex." Um, so on one hand, uh, it feels good. On the other hand, it's sometimes frustrating because, you know, like, everybody's jumping in, and some people are really good at spinning narratives, telling stories, you know, better than we are because we are engineers and it's not our culture. And, and, like, things about you, you mentioned where the risk for humanity, things like that. We, we feel it's, it takes a lot of time to debunk. Um, and we feel that most people, not all, you know, some of them are very credible, of course, but, uh, mostly, uh, the people who defend these theses are not, uh...

Harry Stebbings 34:38 You mentioned creating chatbots 22 years ago, and kind of the, the length to where we are today. I always think it takes longer to adopt than we think. Bluntly, when we look at where we are today, are we really at the precipice now? Or actually, is it another 10 years before we see that adoption cycle? I'm just intrigued.

Alex Lebrun 35:00 I think f- finally it's here. Uh, when I see people using ChatGPT and learning to do their job differently with the help of ChatGPT, um, I really feel we are on the verge of finally having a, a huge impact with, with chatbots. Um, for the first 10 years, I, I tried to convince people that chatbots were ready and to, to buy my chatbots. And they say, "No, it's not." And, and then for the 10 following years, people told me, "Oh, chatbots are incredible. We can replace human with chatbot." And I said, "No, no, no, no, (laughs) don't do that. It's not ready." And finally, now, I feel we, we are at a point where, um, many things will, will be impacted by chatbot, which, which doesn't mean, you know, replacing people, um, with them. You have to be... I think it's a great tool, but it doesn't mean it should, because of the chat form, it's tempting to think, "Okay, let's just replace this guy, this worker with a bot." In many cases, I don't think it's ready for that, but it's, but it's, it will still have a huge impact of how work is done.

Harry Stebbings 36:01 You said it's not ready to necessarily replace. When we think about NABR and healthcare, um... (laughs) and I seem to be quoting him a lot, but Emad: "With AI, you can change the nature of a doctor." Fantastic. Thank

Harry Stebbings 36:14 you, Emad. Will AI replace doctors?

Alex Lebrun 36:18 So, AI will not replace doctors, but doctors who use AI will replace doctors who don't.

Harry Stebbings 36:25 Mm-hmm.

Alex Lebrun 36:26 Definitely. Uh, AI is making a huge... It can have a huge impact on doctors, the way they work. And those who will embrace this change will, will, you know, will thrive.

Harry Stebbings 36:37 Can I ask you, in the next one to two years, how will those doctors who embrace it, how will they use it?

Alex Lebrun 36:44 So, the, the, the, the first thing I learned when I started to operate a healthcare company and, and a healthcare clinic, because we were a clinic initially, um, is that for the first 50 years of computerization...Um, computers have been a bad news for doctors. Before computers, they see patients, they, they, they care, they ha- there is a little bit of admin work, but it's done by somebody else, so you know, secret- medical secretary, the assistant, and not so much, uh, paperwork. And, and they're very, very, very happy. And suddenly, people like us come to them and say, "Okay, now there is a electronic patient record you need to document. Uh, you need to fill this form. You need to, to do this and that. And we, we have trouble to get the money from the insurance company, so we need to document more to, to make sure the claims go through," and so on, so on, so on. So, you know, the other day I visited a, a small clinic, uh, in, in, in Paris, and all the doctors looked very happy. And I said, "What happened?" You know, it's not normal. (laughs)

Harry Stebbings 37:49 (laughs)

Alex Lebrun 37:50 They say, "Oh, the computer system is down today, so we, we are just using, uh, papers and pen." (laughs)

Harry Stebbings 37:55 (laughs)

Alex Lebrun 37:55 And they were so happy, all of them. Um, and so you have to realize that the, the state today is a very bad state where computers were bad, bad news for them. And because they are drowning, you know, under this administrative work, documentation, um, for many reasons, um, they spend... Today, doctors spend on average, uh, 49% of their time doing this kind of adminis- yeah, administrative task, as opposed to caring for patients.

Harry Stebbings 38:27 Mm-hmm.

Alex Lebrun 38:27 It's not... It's bad for everyone. It's bad for them. Uh, three out of four doctors suffer from symptoms of, of burnout. It's bad for patients. And-

Harry Stebbings 38:37 Thre- three out of four suffer from burnout.

Alex Lebrun 38:39 I- I- yeah, three out of four doctors have suffered from burnout symptoms for the... in the last year. And, and the, the, the main reason of this is they all, you know, mention the, the pressure of all this admin work, basically.

Harry Stebbings 38:54 What is that admin work? I'm... I don't live this life.

Alex Lebrun 39:00 (laughs) So, so the, the biggest, uh, admin work they have to do is clinical documentation. So they, they have to document everything, they hear, the- their decisions. And in, in big systems that we call EHR, you know, electronic health records, which are like dinosaur, uh, systems. Um, and the reason they need to do this documentation more and more is... first it's a financial reason. To get reimbursed by the insurance companies, especially in the US, you need to have like a s- very solid file with all that. Otherwise, the insurance will take the first opportunity, the first pretext not to pay your claim.

Harry Stebbings 39:36 Mm-hmm.

Alex Lebrun 39:36 The second reason is for legal protection, uh, because medical, uh, malpractice trials, you know, it's so expensive. You... Your best protection is to document this in the right way. And so this is why p- mainly, uh, admin work started to, to grow bigger and bigger. And the old... you know, these EHRs are very, very old and clunky systems. Um, I checked the other day, there was one specific clinical action that took 227 clicks, mouse clicks, to be performed in the system.

Harry Stebbings 40:12 (sighs)

Alex Lebrun 40:12 And I posted this on LinkedIn, and many doctors r- uh, answered and said, "No, actually it's more than 300 clicks." (laughs)

Harry Stebbings 40:19 (laughs)

Alex Lebrun 40:20 Um, and so this is the state today.

Harry Stebbings 40:22 Uh, so when we look at that though, the thing that strikes me there is like bluntly, okay, we could actually have, um, recordings of meetings, transcript, you know, speech to text, and then auto suggestions, meeting summary notes, whatever we wanna do. The trouble is, it's healthcare data. It's not me and you chatting about shopping, where we could have preferences. No one really gives a shit actually so much about, you know, what you wanna buy from a supermarket. Um, with healthcare data, does regulation and consumer protections get in the way of efficiency?

Alex Lebrun 41:01 Yeah, so, so first, uh, the solution, I think how AI will impact doctors, it, it will, it will bring an AI assistant to every doctor, to every clinician. Um, this AI assistant will be aware of what's, what's happening, what the patient is saying, what, what data we know about this patient, what documentation, what work needs to be done. How do we-

Harry Stebbings 41:21 How... H- sorry, just to break it down, how does it know what the patient is saying? We record meetings?

Alex Lebrun 41:25 We don't record the meeting, but we capture the audio. We don't store it, we just capture it and then drop it.

Harry Stebbings 41:31 Uh-huh.

Alex Lebrun 41:31 Um, and so yeah, it gets a lot of context from the audio environment, what we call ambient, uh, system.

Harry Stebbings 41:38 Uh-huh. And, uh, the data itself from the consumer, how do we do that? From wearables? From... How do we ingest the data?

Alex Lebrun 41:47 So we... This AI c- assistant is connected to the EHR, so it will get the data from the existing source of truth, which is, uh, this patient's records.

Harry Stebbings 41:55 Mm-hmm.

Alex Lebrun 41:56 Um, and based on these things, we will-

Harry Stebbings 41:59 Are, are the EHRs interoperable? Do they have the ability to move data between different applications and silos?

Alex Lebrun 42:07 So, in theory, uh, there is a new law in the US, uh, should be-

Harry Stebbings 42:11 (laughs) In theory.

Alex Lebrun 42:12 ... that, you know, Epic spent billions to, millions to try to, uh, block. Uh, but it finally was, after like, I think 20 years of, uh, battle was passed. And so, yeah, they should be interoperable. Um, th- like, the new model in EHRs are like CRM systems today. They have an API, it's easy to integrate. And then you have the old ones who are trying to protect their, their customers. The only way to have... they have to keep their customers is to re- to put barriers, to re- (laughs) so they cannot flee.

Harry Stebbings 42:42 No.

Alex Lebrun 42:43 Um, and so for, for these, uh, it may be harder to, to get to the data. What we do, um, at Nav- at Nabla, you know, we use things like Chrome extensions to get the data from the browser. There is no API, w- we don't care. We get the data, and, and the doctors are happy, patients are happy.

Harry Stebbings 43:04 Can I ask, uh, why would healthcare about the incentive mechanism? And what I mean by that is, like, specifically if we make people much more productive, um, and we maybe remove the need for certain nurses, actually more people will be unemployed. (smacks lips) And when we think about politicians who are allocating money for the NHS, which is obviously our healthcare system, they need nurses' votes. And so saying, "Hey we're gonna remove a load of low-level nurses through intelligent automation workflows," whatever that is, it goes ins- a- against the political incentives by being more efficient. Does that become a problem?

Alex Lebrun 43:44 It's, it's not a problem because so, so the World Health Organization says that we are missing, uh, 18 million clinicians by, uh, 2030.

Harry Stebbings 43:53 (laughs)

Alex Lebrun 43:54 So the state is not, uh, that it will put people out of a job. The s- the re- the reality is we are desperately looking for clinicians in every country in the world. So if you can make their lives easier, uh, it's a big win for everyone.

Harry Stebbings 44:08 Yeah. The demand and the supply chart is way out of whack.

Alex Lebrun 44:11 Exactly.

Harry Stebbings 44:11 And so the... What are the biggest barriers then to AI having the impact on healthcare that I think we both believe it can?

Alex Lebrun 44:19 There, there are many barrier. I think the, the mistake that many AI startups are doing in healthcare, and what we did initially at Nabla too, is we focused directly on the patient. And of course, in your ideal way to see the, the things you want to say, "Let's do a B2C product," patient, we forget the doctor (laughs) in the, in the, in the loop. Uh, and you have this movie-induced vision where a, a patient is in front of a robotic doctor, and everything is so smooth, and imaging, automated, data-driven. And it's tempting to, to, to, to jump straight to this vision. And, and you can convince VCs to, you know, to give you money to do that-

Harry Stebbings 45:00 (laughs)

Alex Lebrun 45:01 ... uh, because it makes a, like, a cool, uh, slide deck. Um-

Harry Stebbings 45:05 (laughs)

Alex Lebrun 45:05 But the reality is, it's, it's too much, too fast, you know. You cannot jump. There, there are so many... The healthcare system is so complicated. Um, you cannot... Regulation, you know, for many, many reasons, you cannot do that. Technically, we're not ready to do it. We don't have the data. And, and, and so I think the... We should first focus on clinicians, the way things are done today, change their life with AI, makes, turn them into AI fan base, and then together we, we'll, we'll go to the next step. And I, I, I think... So I think, to answer your question, that one of the main, um, problem is that we are trying to go too fast to, uh, a patient-facing perfect system. When you ask doctors... So at the beginning of Nabla, you know, I'm sp- I'm spen- I was spending a lot of time with, uh, in hospitals, in the emergen- I was spending nights in the emergency services call center in Paris, listening to everyone. It's... And, and-

Harry Stebbings 46:01 How is, how is that like? What do you see?

Alex Lebrun 46:03 So it's interesting, you know. I met, I met, uh, a doctor in a café in Paris. I, I made a demo of Nabla on my phone, and he told me, "Alex, you're going to come to the emergency services call center next week, all night with me." It turns out this guy were... He's, like, the chief emergency physician in France, Patrick Pelloux, invited me to spend nights over there. And he, he, he capture very quickly what, what, how we could, you know, change his life and the life of his team and the life of people. And so in, um, what I learned spending, uh, nights over there is, um, it's a little bit like a, a call center. You have your first team is taking calls. So overall in France, it's about 40 million calls a year. And the first responder, they, they don't make medical decisions. They are not doctors. They're not even nurses. They, their, their goal is to just to understand what is the address and what is the situation, you know, what is the emergency. It's very hard to get that because people are panicked, and you have lots of people calling for very small things, really close things. But, and you have real emergencies hidden in the... So their job is to, to sort through these very, very complicated situations when people are calling. Uh, and, and they actually write a note, structured note, with this information. They click Submit. And then you have, like, regulation doctors who make the decision, should we send a real emergency services, should we send firefighters, or should we just, uh, have a doctor talk to, to, to, to this person over the line? And so, uh, it's a two-step decision. And a lot is done around documenting the situation. And these things, you know, is, would be so easy to do with what we do at Nabla, and you know, what, what we know how to do with AI today. The, the goal would be to keep, of course, the person who is answering the phone, because th- their knowhow in terms of empathy, calming people down, understanding the context, very complicated situation again.

Harry Stebbings 48:13 Yeah.

Alex Lebrun 48:13 Typically you have several people. They're screaming. Nobody knows how many victims, what's happening. A lot of with mental health problems. So it's very, very complicated situations. And these people, I spent hours and hours listening, uh, with a, with a, you know, with a headphone next to them. Th- they are expert at, at cutting through this darkness and understanding what, what's happening. So we want to keep these people. But they are typing with two fingers on the keyboard and, and missing some information, takes a lot of time to get, and so on. So having an AI assistant work, uh, a- alongside them, listening to the call and documenting, we, we, w- I tried it, you know, it works already perfectly with what we have, uh, would be a perfect team. So this is an example, uh, perfect-

Harry Stebbings 48:58 And it actually means they can be a lot more empathetic and not worry about typing and getting everything down, and being here in the moment with you to understand your problem.

Alex Lebrun 49:06 Exactly, and it will be, it will be faster too. So it's a win-win, you know. It would be, like, maybe 20 seconds instead of one minute-

Harry Stebbings 49:14 And-

Alex Lebrun 49:14 ... and with more empathy.

Harry Stebbings 49:15 How many of the calls lead to ambulances sent out? Do you have that number? Is it like 1 in 10, 1 in 20?

Alex Lebrun 49:23 It's, I, I'd say it's one in five- I, I'm giving you like, like what I saw.

Harry Stebbings 49:28 Yeah.

Alex Lebrun 49:28 It's not an official, uh ... (laughs) What I saw is a few nights, you know, maybe one in five results in, um, services being sent. In these cases, one in four, uh, three in four are firefighters that will have medical, you know, doctors and so on, but they, they don't handle the most critical-

Harry Stebbings 49:48 Yeah.

Alex Lebrun 49:48 ... uh, medical emergencies are handled by 20 called SAMU in France. Uh, and it's, it's o- one in four.

Harry Stebbings 49:54 How receptive are doctors to you wh- when you talk about AI and improving their work with AI? Are they like, "Yes, bring it on," or like, "Huh, seen it before"?

Alex Lebrun 50:05 So we've seen a change in that in the recent years. And up to three years ago, you know, w- almost every doctor would tell me, "Okay, you don't understand our problems. Go away." (laughs)

Harry Stebbings 50:14 (laughs)

Alex Lebrun 50:14 "I'm, I'm r- I'm already fighting against my EHR every day. I don't need more stuff." Um, and, um, it really changed recently. Um, the core reason has nothing to do with AI. It's that the, the health systems are collapsing everywhere, you know. I know the situation with NHS here. It's worse in France, in the US, all, all the major hospital groups are losing money.

Harry Stebbings 50:37 Why is it worse in France? Losing money, worse treatment? Wh- what's wrong there?

Alex Lebrun 50:41 So in France, we have a shortage, uh, a huge shortage of doctors, and there are, like, zones in France where y- you, you have to drive 200 kilometers and wait for three weeks just to see a GP. So this is obviously a big problem. So the access ... Globally, the problem is the same everywhere. It, it's access to healthcare. You know, in France, you, you can get good doctor quickly if you know somebody in the system. In the US, you can get it if you have a lot of money. The ... It's, it looks different, but it's, eventually it's a problem of not enough doctors.

Harry Stebbings 51:16 Don't laugh. Are there are less doctors than ever, or are we getting more unhealthy than ever? Is the need increasing? Is the demand increasing, or is the supply decreasing?

Alex Lebrun 51:26 Uh, both. So ... (laughs)

Harry Stebbings 51:27 (laughs)

Alex Lebrun 51:28 So there was some lobby like 20 years ago to have less doctors, you know, because they would, um ... I go, I, I won't go into there. Uh, so but, and since it takes 10 to 15 years to train doctors, uh, decisions you've made 20 years ago have a big impact today. So first, there are less and less doctors. And, um, many of them are aging and will go be retired soon, and so the, the, the number is going to drop. Um, and in terms of demand, uh, in all, everywhere in the world, you have a huge rise of chronic disease like diabetes, uh, that takes a lot of medical resources. You know, chronic disease, we, we always see as engineers, young peop- healthy people, we see is that I, like the easy things with, with medicine where, "Oh, I'm hurt." I go, there is a quick diagnostic decision. I have a prescription. Maybe I spend two days in the hospital, and then it's over. But what takes the resources in the healthcare systems is not that, is, are aging people and, and chronic disease, and often combined.

Harry Stebbings 52:33 I actually had a fascinating stat, and it was like, 90% of the cost of healthcare is spent on the last six months of life, something like that, which was astonishing to me. Can I ask you, which is like, you deliver great value to doctors, to frontline workers. They don't control the budgets. I, I, I... My first ever investment, fun fact, was like a WhatsApp for doctors and nurses in the UK. They went to 100,000 doctors and nurses in the NHS, and then it came to getting paid. (laughs)

Alex Lebrun 53:03 (laughs)

Harry Stebbings 53:03 And it w-

Alex Lebrun 53:04 Was it ?

Harry Stebbings 53:06 I mean, it ... No, it was, uh, Pando-

Alex Lebrun 53:08 Okay, okay.

Harry Stebbings 53:08 ... which was like the competitor to .

Alex Lebrun 53:10 Okay, okay.

Harry Stebbings 53:11 Um, and they just, just don't pay. I mean, it's shit. And they would deliver great value, 100,000 doctors and nurses. How do you think about actually building a business, be delivering value in a sector that just doesn't pay and doesn't have a willingness to pay?

Alex Lebrun 53:29 So th- this is the biggest, uh, difficulty when you are trying to, uh, disrupt healthcare, actually, especially for a startup.

Harry Stebbings 53:35 Yeah.

Alex Lebrun 53:36 Um, you have the, the payer, you know, insur- insurance or public payer. Um, you have the providers who are providing care, and you have the patients. And it's a very complex relationship with three parties. And typically, the one who benefits from our product is not the one who is paying for it. And so typically, the patient will benefit, for instance, for, from your better healthcare app, but it has to be paid by the payer, by, by his NHS. And if the provider disagrees, they can block everything. So you have this, to, to solve these three mass problem from the day one, which is very, very, very complex. Um, and so you have to str- to, to find a balance between if you optima- as a startup, if you optimize a very small process, it's easy to deploy because it, it follows the lines of the existing system. So for instance, if you do a better way to take appointments, you know, you make online appointments instead of calling, you don't change a healthcare system with that.

Harry Stebbings 54:39 Sure.

Alex Lebrun 54:39 B- and it's, it's easier to push ... Even that is hard to push, but it's easier because you don't change the fact that there is an appointment, there is a consultation, there is a synchronous visit. So you, you fit into the system. If you do something too ambitious, that is, it will never fit in the existing system. It's, I think it's impossible to, to, to, to go to market-

Harry Stebbings 54:59 Sure.

Alex Lebrun 54:59 ... with something too ambitious. So I think as, as startups, we have to find the right level of am- ambition where it's, it fits enough in the system to be, uh, adopted, uh, but it's not too much disruptive from day one. Otherwise, it's impossible to go to market. This is a balance we have to find every day.

Harry Stebbings 55:17 Yeah. And who pays you?

Alex Lebrun 55:20 So, s- so learned very hard lessons. (laughs) Uh, if you are, um, thinking about creating a healthcare startup, the very, very first question I want to hear is-... who is paying for the, for the product? Who is paying? What is the business model? Then you can ... You know, in, in, in, in a startup, uh, you know, in YC, in all startup schools in the world, they tell you, "Don't start with a solution, start with a problem." But in healthcare, even starting with a problem is not enough. Start who, with who is paying, and then how you do frame the problem for this person to pay, and then what is the solution for this problem eventually. And, um, so it's ... So, so, yeah, in, in our case, uh, at Nabla, we have this vision of an AI assistant, and it took us some time to find out who will pay and how for this thing.

Harry Stebbings 56:14 If we think about this respondent to the caller, and we take intelligent notes which allow him to be better and more empathetic, who's gonna pay for that? The hospitals don't pay. So like, a-

Alex Lebrun 56:26 In the emergency, emergency services?

Harry Stebbings 56:28 Yeah, yeah.

Alex Lebrun 56:29 So in emergency services, the only pu- potential payer is a, is a government, you know, and so not a good, good go-to-market, you know? (laughs)

Harry Stebbings 56:37 Yeah.

Alex Lebrun 56:37 Disastrous go-to-market, so ... And that's why I didn't start with that. Now, you know, we've deployed our product in, in, in the US, thousands of doctors are using it every day, we are well-funded. Now, if the emergency services in France really want my product, and the sponsor is, uh, Patrick Pelou, who is like the god of emergency in France, and the Minister of Health looks at me in the eye and say, "We really want it," it may be the right timing. But it's, it would be a very, very bad idea to start with that as a startup. And maybe if you're lucky, you do a pilot, and then you wait, you wait, and you ... It's what I call death by pilot.

Harry Stebbings 57:17 And so you go to the US and you sell to private clinics?

Alex Lebrun 57:21 We started with the US because the problem we are solving now, uh, of clinical documentation is huge in the US, you know, bigger than in Europe. Uh, and so the willingness to pay for the existing providers, and again, you need to, an entry point in the, in the system, uh, is very, ver- is very high. And so, um, we found this go-to-market. We also, uh, made it possible for physicians to use our product without any approval for, from anyone. You know, it's, it's, it's like a B2, almost a B2C go-to-market, bottom-up. They can go to nabla.com. I ... You know, it's a web application, it's a Chrome extension. They use it tomorrow, and then they love it, and then they will recommend, you know, talk to their boss and say, "I, I would like to use this in, in the full clinic."

Harry Stebbings 58:09 Can I ask you? Imagine going to the US then because it's a bigger market. I think there's a common suggestion to a lot of AI founders today that if you wanna build an AI, you've got to be in the Valley. It's all about being in San Francisco, it's where the talent is. Paris and France has become also an AI hotspot. How do you advise founders, first, on whether they need to be in the Valley or not to be building the best in AI?

Alex Lebrun 58:34 I think it used to be true, uh, but it's not true anymore. Uh, when I built my second company, With.ai, uh, I happened to stumble about, upon Adam Cheyer, the, the creator of Siri, at a parking lot in the Valley after an event, and he, uh, he became friend. He, he helped me a lot to think about With.ai. He gave me the confidence to follow what I had in mind, uh, and owing to this confidence, I, I, I was able to refuse referrals from, you know, Google and others who wanted to buy us too early, and so it has a huge impact of this on m- on my company. If I wo- if, if I were in Paris, no Adam Cheyer, nobody has done this before, maybe I would have sold it to the first, uh, "Okay, 10 million, okay, let's sell it. It's good." And so it used to be true, I think, that you needed to be in the, in the Valley, uh, with having these kind of people around you. Now I think it's less true because it's, everything is distributed. The talent is very, very distributed. It will get more distributed, and I think you should get close to your customers more than the-

Harry Stebbings 59:36 Do you think French startups still sell too soon?

Alex Lebrun 59:39 Still what, sorry?

Harry Stebbings 59:39 Sell too soon.

Alex Lebrun 59:41 Yes, French start- ... So in

Alex Lebrun 59:43 France, we are really good in, um ... We have very good AI engineers, machine learning engineers, because the education system is free, it's very focused on mathematics. Um, so we produce lots of good engineers, but we suck at growing companies, really. We are n- bad at that. I don't know why. Maybe we lack discipline. Maybe we-

Harry Stebbings 1:00:03 Is it you suck at growing companies or you just sell too soon? Like something also comes in, and actually 50 million, it's a lot of money. You've only raised one round. You'll take home 10, 15 each as co-founders. C'est bon?

Alex Lebrun 1:00:16 Exactly, c'est bon. It's enough. (laughs)

Harry Stebbings 1:00:19 (laughs)

Alex Lebrun 1:00:19 And so I think it's hard to ... If you are normal, it's hard to refuse a $1 billion offer if you haven't done, you know, 10 million before.

Harry Stebbings 1:00:26 Yeah.

Alex Lebrun 1:00:26 Uh, but this is what Zuck did, what Google founders, you know, Sergey and, and Larry did. So maybe we're not f- focused enough or disciplined enough to ... And we don't have examples around us, uh, to grow this company. Uh, hopefully it's changing. We have very, very good, uh, scale-ups in France now. But it, it takes some times to ...

Harry Stebbings 1:00:47 What do you think Europe needs to do to keep pace with the US in terms of AI and being an ecosystem that attracts the best AI talent?

Alex Lebrun 1:00:55 So Europe is probably, uh, 10 years late compared to US and China, and with the new regulation, we are probably going to take 50 years, uh, more. (laughs)

Harry Stebbings 1:01:03 Talk to me about that. Why, why do you think that is? Because the regulation is so prohibitive? W- unpack that for me.

Alex Lebrun 1:01:10 So the new regulation is, is, is a disaster. Uh, if you look at it, there is no way ... People who wrote this, I don't, don't understand really the, the ... They went too fast, you know. Good regulation is about good timing and involving the right people who are actually doing stuff in this domain. Um, I think Europe being late felt like, okay, let's be at least first in regulation, which-

Harry Stebbings 1:01:34 What was it about the regulation which made this so bad?

Alex Lebrun 1:01:37 So the regulation ...Some of the aspects say, for instance, that you should kind of be accountable. I'm simplifying, but you should be accountable of the... all the data you use to train your model, for instance. You should make sure that you have a proper license, uh, like, like, explicit consent from everywhere it comes from and which in pra- I see why it's a beautiful idea. Uh, who would disagree with that? But in practice, it means that 100% of the LMs that were trained these last three years would be illegal in Europe. And so, I think it, it's out of, out o- it's, it's, it's not connected to the reality. The, the intent is good, but the, the, um, the limited puts on how to train a model and how to operate a model makes it... in practice what, you know, compared to what we do today, makes everything illegal.

Harry Stebbings 1:02:34 Does it mean that... Uh, so it makes everything illegal, as you said, what do European startups do then?

Alex Lebrun 1:02:40 We move to the UK. Uh, maybe for one reason, Brexit-

Harry Stebbings 1:02:43 Move to the UK?

Alex Lebrun 1:02:44 Yeah, finally, Brexit is, uh, maybe was good. (laughs) Uh, uh, you know, I'm half joking. If the regulation, uh... it's not, it's not final yet, but if it's really like this and nobody can challenge it, uh, startups like us may have to move. Or, or, uh, maybe we can keep the team in France, but we have to physically train the models elsewhere. I don't, I don't know, but it's, it's totally, uh, it's a big, huge danger.

Harry Stebbings 1:03:10 What would you do if you were in charge? If I put you in The Hague or in the EU and you were given the regulatory, uh, powers over the next five years of AI in Europe, what would you do?

Alex Lebrun 1:03:21 Uh, first, I, I, I'd, I'd wait a l- a bit because it's too early. Um, we are still in an- an- a demo phase of AI. You know, LLMs, it's not really used a lot, you kn- uh, just beginning to be used. And as I said, you know, regulation timing is key. You should- you shouldn't be too late, but if you're too early and nobody knows w- what are the risks that are actually... um, uh, what, what actually are the risks? No, not the risk you think because you read some science fiction books, but the... is a real risk. Nobody knows yet. Um, and so I think first I would wait, uh, maybe a year or two to learn from the field what should be regulated and why, and then I would involve people who build these models, of course, but also people who are using it, you know, the users, the, the general public, which was not done so far. So, they have, like, experts who are, who are smart people, but there is no reality of LLMs yet. So, if you regulate too early, uh, it cannot be good.

Harry Stebbings 1:04:21 Can I ask, do you feel a bit sorry for them? Because, like, there's never been such a large chasm between government's knowledge and then the private sector's knowledge. Are government adequately placed to instill regulation on AI?

Alex Lebrun 1:04:36 Whi- which government, uh-

Harry Stebbings 1:04:37 Well, I mean, any, because bluntly scientists, researchers are not in government, and so you have a load of bureaucrats and politicians saying, "Oh, we need you to, uh, be accountable for all of your models." Do they know what a model is?

Alex Lebrun 1:04:52 Uh, I think some people know what a model is, you know, uh, they, they, they talk to, to some smart people, but, um-

Harry Stebbings 1:04:58 Have you spoken to many politicians? (laughs)

Alex Lebrun 1:05:00 Not enough, probably. (laughs)

Harry Stebbings 1:05:03 (laughs) But do... say, uh, uh, you kind of have to say, "AI is this. That's where we're at."

Alex Lebrun 1:05:10 Yeah. I, I, I'd love to, um, uh, spend more time, you know, to take them to the field and show them how it works and, uh, and the benefits and the... and the, uh, real risks because I'm not saying there is no need for regulation or no risk, but yes, there is a huge disconnection.

Alex Lebrun 1:05:25 Um-

Harry Stebbings 1:05:26 What do you think about China? We talked about Europe there. China, how do you think they've embraced AI and the next wave of AI?

Alex Lebrun 1:05:32 So, so China, they have, like, a, a key advantage that... like, there is no, uh, GDPR or very few regulation internally.

Harry Stebbings 1:05:43 Mm-hmm.

Alex Lebrun 1:05:43 Um, and so they... the, the data quality they have and the amounts, both the quality and, and quantity of data is, is incredible in China. Um, and so for instance for healthcare data, they probably have everything about every individual in China, and if you have the right... if you're supported by your government, I'm sure you have access for, for research, uh, to all this. So, they have this... they are lucky, uh, for that.

Harry Stebbings 1:06:10 Yeah. No. I, I, I totally agree. Okay, if there's anywhere that, like, bluntly, if you think about geographies, who's the winner and who's the loser in the next 10 years China, US, Europe?

Alex Lebrun 1:06:26 Well, they all have issues, their issues. So, we talked about the, the advantages of China, but th- they are very close, they are getting more and more close. They have maybe the wrong incentives at the research level. Uh, so it's hard to predict what will come out of, of, uh, China. We talked about Europe where regulation is probably the, the biggest problem that will, uh, keep Europe behind, and we are already behind. That's, that's my concern. The US, you know, is, is, is, uh, like, always in a very good position. Immigration may be lows, maybe something that will, uh, be a problem eventually in the US because talent is so distributed. Uh, with the new... you know, you can learn deep learning very easily from everywhere in the world on, on a tablet today, and it'll, it'll be, it will be more and more, uh, like this. And, and so if it's so hard to go to the US to work for a US company, and eventually it will have an impact, I think.

Harry Stebbings 1:07:23 Yeah. And then the US, what's the biggest downside of the US?

Alex Lebrun 1:07:26 So, immigration laws-

Harry Stebbings 1:07:27 Yeah.

Alex Lebrun 1:07:28 ... I think are the, are, are the biggest, uh, downside of the US.

Harry Stebbings 1:07:33 I wanna move to a quick fire round. I've peppered you already. I say a short statement, you give me your immediate thoughts. Does that sound okay?

Alex Lebrun 1:07:39 Okay. (laughs)

Harry Stebbings 1:07:40 Okay. So let's start with what do others not know that you know to be true?

Alex Lebrun 1:07:45 That we live in a simulation.

Harry Stebbings 1:07:48 Unpack that for me.

Alex Lebrun 1:07:50 (laughs)

Harry Stebbings 1:07:50 You-

Alex Lebrun 1:07:50 I can- I cannot compromise my sources. (laughs)

Harry Stebbings 1:07:54 (laughs) Do you agree that some of the biggest businesses to be built in AI will be built in services businesses helping enterprises implement AI?

Alex Lebrun 1:08:02 I, I disagree.

Harry Stebbings 1:08:03 Huh.

Alex Lebrun 1:08:03 Uh, I, I think AI will enable a new generation of, of players in every industry that will kill the incumbents eventually.

Harry Stebbings 1:08:12 What do you think is that time scale?

Alex Lebrun 1:08:14 Five years.

Harry Stebbings 1:08:15 Five years?

Alex Lebrun 1:08:15 Mm-hmm. And, and I mean, existing services companies, like consulting companies, we know will, are embracing AI. They will eventually, they, they will remove the big data from their website and put AI, and then they will put LLM to replace the words. And they, they will take this business, I think. Um, there was a company called Element AI few years ago in Canada, very ambitious, uh, company who tried to do services, uh, like that. And, uh, they eventually failed. They were acquired by ServiceNow for, um, the amounts they raised.

Harry Stebbings 1:08:47 What is the biggest element you'd like to change about the AI community?

Alex Lebrun 1:08:50 Make it more diverse. I'm, I'm really tired of talking to people like, just like me. (laughs)

Harry Stebbings 1:08:55 (laughs)

Alex Lebrun 1:08:55 And we would ... It's, it's a cliche to say that, but it's actually, uh, we would, uh, it would be, uh, a lot of value if we get, uh, more diverse. I, I have an, uh, an- I, I can just tell a quick anecdote. Uh, when I was at Facebook, somebody discovered that the model that w- was supposed to, to, to tag images, uh, that was 100% successful to finding tennis balls in pictures, actually, if you remove the tennis ball from the picture, it was still finding the tennis ball. So it, it didn't learn to find the tennis ball, it learned the, the context of a tennis ball, like a racquet and ... And then in the second phase, y- you change the color of the people on the picture, and if you add more black pixels in the, in the hair, suddenly, the model thinks, "Oh, it's a ping pong ball." And so th- this, it was a very early days when we discovered that some models have bias.

Harry Stebbings 1:09:48 Huh.

Alex Lebrun 1:09:48 And this is bias in th- in this case is it didn't learn to look at the ball, it learned to look at the race of the people around, and it's actually making this decision tennis versus ping pong, uh, based on the, on the, on the (laughs) on the race of the people around.

Harry Stebbings 1:10:02 Iman said all models have bias. Is that fair?

Alex Lebrun 1:10:05 Uh, yeah. It's, it's, uh, more or less, but all of them, and, and many models have huge bias. And the, the guy, you know, very strong researcher who found this thing in, at Facebook very, very early before everybody talked about bias, uh, is Moustapha Cisse. He's from Senegal. I don't think it's, uh, it's just luck, you know, that somebody with a different angle than we have, a different culture, uh, found that's a huge hole in the model that we, we ha- hadn't found before.

Harry Stebbings 1:10:34 Does AI kill traditional media?

Alex Lebrun 1:10:37 I, I don't think AI will kill investigative, you know, journalism, the finding the right sources, the right information. So this part of, of, of media, I hope ... I don't think will be replaced by AI. Now it's d- generation part at the output probably will be disrupted a lot by AI.

Harry Stebbings 1:10:56 Yeah. Okay. Tell me, what's the most painful lesson you've learned that you're also pleased to have learned?

Alex Lebrun 1:11:01 That fundamentally people don't really change.

Harry Stebbings 1:11:05 Huh. What makes you say that?

Alex Lebrun 1:11:07 I mean, you, you can, you can help people to grow, to ... It's true for yourself, you know. So you can correct some weaknesses, you can be stronger at where, what you're strong at already. But I think you have to accept that some of the fundamental characteristics of yourself and the people around you won't change. And if you expect them to change, it leads to big failure and mistakes.

Harry Stebbings 1:11:33 What do you think is the biggest misconception that people have around AI? You hear many discussions. What are you thinking? Like, "Oh, I can't believe they're saying this again."

Alex Lebrun 1:11:43 It's that they think it's conscious, and they, they are influenced by the, the form, like, "Oh, a perfect answer." (laughs) "A perfect form and, and it's a chatbot. It, it, it answered my question." And, uh, it's ... And influenced by the form, they make conclusion on the, the deep inside, you know, of the system and say, "Oh, it's, it's conscious." And for me, the biggest proof of that is that in 1966 when the v- the mother of all chatbots was released at the MIT, you know, Eliza, the very first bot, um, people, for a few months, people around the world thought that AI was solved, that there was a human, you know, sentient AI, uh, in Eliza, which is a very, very simple pattern matching system with regular expressions. Um, but because of this, the way it was presented, people fell in the trap. And I think we are ... We keep f- falling in this trap again and again and again for 60 years.

Harry Stebbings 1:12:43 Who's been the most helpful angel?

Alex Lebrun 1:12:46 Probably Yann LeCun who invested in, uh, you know, in Nabla. Um, especially when we doubt about something or, uh ... He's very confident that we are on the right path, and it, it means a lot.

Harry Stebbings 1:12:59 What makes the best VC? Like, help me. I'm an investor. You're a founder. What, what can I do to be the best investor for you, honestly?

Alex Lebrun 1:13:09 I think it depends a lot of, uh, uh, the s- founder in front of you and the stage o- of the founder, you know.

Harry Stebbings 1:13:16 What do you want?

Alex Lebrun 1:13:18 So personally, you know, I, I worked with Andreessen Horowitz with my previous startup. It was incredible because they never call, they never ask for anything. We d- we didn't even have board meetings. And when I ... Every time I need something, in, uh, 20 minutes I get huge, huge help. You know, maybe it's to find real estate problem because I don't have an office. Maybe it's, uh, organizing a huge event and I don't want to hire a marketing team. I want to use them for one week. Uh, or I have this like strategic decision to make and they give me access to the best people in a, in a, in less than a day. Uh, and so the fact that you, you, you are in control and you ask for help and they deliver it, uh, was ... We, we could grow With.ai to, um, a very nice company with only 13 employees.... because we, we were relying 100% on of VITs for all these things and it was incredible. But, but for my first startups, I did some strategic mistakes, go-to-market mistakes. And then I wish my VCs back then would have been more involved with me and tr- coach me. And so, uh, it really depends on the founder stage, I think, and you should ask. (laughs)

Harry Stebbings 1:14:27 Do you think VC in Europe is good or do you think it's still massively behind? You've seen in the US.

Alex Lebrun 1:14:32 Again, it changed a lot, you know. I think the recent... So 10 years ago, VCs in Europe were all from the financial industry. You know, there is nothing about financial industry, finance in being a VC. A little bit, of course, but it's mostly about entrepreneurship. And so 10 years ago, all the VCs I was working with in the US were former entrepreneurs, and all the VCs I knew in Europe were fin- former bankers. And you don't need a banker. In most situation, you don't need your VC to be a banker, but you need them to be, uh, an ex-entrepreneur.

Harry Stebbings 1:15:05 Yeah. No, I get you totally. 10 years time, where's healthcare?

Alex Lebrun 1:15:11 Hopefully 10 years time, you know, every physician has their AI assistant doing a lot of stuff for them and helping them to be 10, 10 times more efficient, you know, seeing more patients, spending more time with every patient, making better decisions. I also think that at the higher level, decisions that require a big picture of the data, you know, large, uh, large view will be taken by AI, not by people. So, uh, for instance, in a emergency services regulation center, when you have thousands of people needing help and you have some resources, what, what is the state of your resources? What is the demand? How you make these decisions at the, at the, at the city scale to today is done by humans, but it's a mir- it's a miracle that it doesn't collapse and, and sometimes it's collapsed during the big, you know, events. Uh, and so this kind of decisions I think will eventually be taken by, uh, by AI.

Harry Stebbings 1:16:06 2033. Where are you then? Where's Alex then?

Alex Lebrun 1:16:10 So my dream is to build, um, a healthcare system from scratch without any of all these limitations and constraints we mentioned. So controlling full stack, patient experience, provider, hospital, data-driven prevention. Uh, so my dream is to, to be able to build that. And to do that, I need, I still need to learn a lot in healthcare, and I need a few billion dollars.

Harry Stebbings 1:16:35 (laughs) Alex, listen, I've loved doing this. Thank you so much for joining me, and thank you for putting up with my prying questions.

Alex Lebrun 1:16:41 Thanks, Harry.

Harry Stebbings 1:16:42 You are a star, my friend.