No PriorsNo Priors Ep. 26 | With Weights & Biases CEO Lukas Biewald
EVERY SPOKEN WORD
100 min read · 19,804 words- 0:00 – 8:16
Lukas Biewald's Journey in AI
- EGElad Gil
(music plays) We've talked to many practitioners who are pushing the state of the art. This week on the podcast, we're exploring the dominant ML developer tool, Weights & Biases. Elad and I are sitting down with CEO and co-founder, Lukas Biewald. He has a knack for creating companies that support pain points in ML development. His first company, Figure Eight, addressed the problem of data collection for model training, and his second company, Weights & Biases, has created an experimentation platform that supports AI practitioners at companies including NVIDIA, OpenAI, Microsoft, and many more. Lukas, thanks for doing this. Welcome to No Priors.
- LBLukas Biewald
Thank you. Great to be here.
- EGElad Gil
Lukas, you studied at Stanford, where I assume you discovered your interest in machine learning, and under one of our previous No Priors guests, Daphne Koller. Can you talk about when you started working in AI and learning from Daphne?
- LBLukas Biewald
Yeah, totally. A- as a kid, I was obsessed with playing games, and I got really into Go, and I was super into the idea of- or thinking about how would computers win at these games. And so, I actually sent Daphne an email, maybe as a freshman, being like, "Hey, can I, can I work with you? Like, I'm really interested in games. I wanna learn how to, like, beat Go." And, and Daphne wrote me actually a pretty polite email being like, "That's not what I do." (laughs) "Go away." A few years later, I, I took her course, and I was actually, I studied math at Stanford, and I have to say, Daphne cared about a thousand times more about teaching than even the best professor in the math department. And so, it was really just eye-opening. Like, I just loved how much she actually cared about teaching, and it got me really excited about the AI that was working there. And I went on to be a research assistant for her, and the funny thing at that time was, like, nothing really worked. Like, it was just before kind of, you know, Google was thought to be really, like, PageRank at the time was the thing that was making them work, and I think later, you know, it became clear that machine learning was a big, a big part of that. But really, when I was doing ML, it was, like, searching for applications that were working, and Daphne was actually really obsessed at the time with a thing called Bayes' Nets, which you don't hear about too much anymore, 'cause I don't think they ever really, um, you know, worked for many applications. I hope I'm not offending anyone, but that's my, my understanding. I actually think, you know, the, the thing that I really took away from Daphne that, that really lasted with me was, um, I mean, she's just one of the smartest people I've ever encountered, and she had this incredible clarity of thought and an intolerance for sloppy thinking that, that's just like really served me well. And I think that's so sort of separate from machine learning. You'd, you'd see like other professors would come and give like guest talks and, you know, they would say something that's kind of lazy, and like, we'd all just be sitting there just like waiting for Daphne to like eviscerate (laughs) them. And I think her, her personality has, has mellowed a little bit o- o- over time, but I, I kind of miss... I just miss that sort of like aggressive clear thinking, um, and I, I really admire it.
- EGElad Gil
I don't think we got a taste of that, but we did talk about whether or not probabilistic graphs are, are coming back a little bit. How did you... how'd you go from, you know, Stanford to founding Figure Eight?
- LBLukas Biewald
Yeah, you know, it's funny. I actually really struggled doing research with, with Daphne. Basically, the things that I tried just barely, barely worked. Like, you know, I, I published a couple papers that I feel kind of ashamed of, where it was sort of like, go from like 68% accuracy to 70% accuracy on a task nobody cares about by throwing like 1,000x the compute. (laughs)
- EGElad Gil
(laughs)
- LBLukas Biewald
And by the way, like, kind of guessing the most likely answer is probably like 64% accuracy, so, um, you know-
- EGElad Gil
(laughs)
- LBLukas Biewald
... it just, it, it felt honestly kind of pointless and sad. Like, I love the idea of, like, computers learning to do things, but it's hard to sort of sustain the enthusiasm for that when everything you try just completely, you know, doesn't work, and even the things that do work, you kind of wonder if you're like p-value hacking, like, "Okay, I tried a thousand things," you know? So, I guess something's gonna be like a little bit more accurate than, than a baseline.
- EGElad Gil
What tasks were you working on? Did you... did you end up working on Go or games or anything?
- LBLukas Biewald
No, Daphne was... Daphne is not interested in games, let me tell you. (laughs) And it's actually another... I kind of admire that, that perspective too, as much as I love games.
- EGElad Gil
I'm a Go nerd, so I'm curious.
- LBLukas Biewald
Oh, you are? Oh, me too. I, I... yeah. I, I love Go. Yeah, Daphne was very not interested. She really was practical, and so I worked on a task that you really don't do now (laughs) called, um, word sense disambiguation, where you're trying to find out like, okay, I have... the, the word plant, actually, if you look in most corpuses, 'cause they're government-generated often at the time, plant typically will mean like the power plant sense of plant, or cabinet often means the sort of president's cabinet sense of cabinet. And so, you're kind of trying to figure out like, what is the meaning here of these words, and, and then applied it to, um, to translation. It's a cool task. I mean, and, and actually it turns out, I think that these... again, nobody kill me, but my, my general sense is that these sort of like linguistic-oriented strategies really don't work that well. It's kind of like, by feeding more data in and, and sort of like working on outcomes you can figure these things out much better. So, um, a little bit of a dead end, and, and actually, you know, I was so frustrated by that, that I, I just really wanted to work on something that people cared about. I actually turned down an offer from Google because they didn't tell me what I would be working on, to go to Yahoo because they, they were like, "Okay, you can work on, you know, search rank ranking in different languages." And, but that actually turned out to be incredibly fun, right? 'Cause it was super applied. It's actually a task that works really well, and, and Yahoo is kind of in the infancy of switching from hand-tuned weights to machine-learned weights, and they really had no one... not many people actually like working on deploying this stuff. So, I was like writing code to translate machine learning algorithms into C code and then check it... like, we would check it into our literal code base and run this kind of like semi-hand-generated C code in, in production. So that was, that was super fun, but, you know, the thing I learned there actually, which I think... I'm not the only one that learned this, but I just felt it. I wou- I would go from like country to country trying to switch from hand-tuned weights to an ML model, and like, I was sort of the messenger here, so like sometimes it would work and sometimes it wouldn't. And so, like, people were either really happy with me when it did work or they'd be really pissed at me when it... uh, when it didn't work. But I kind of realized...... actually, the model that I'm building is, like, the same for each country. It's the, the training data, though, is different. So some countries would take the training data collection process really seriously, and they'd get a great model, and some would just, like, really half-ass it, or like, you know, have these crazy, like, issues in the data collection, and then the model wouldn't work. And so I just really kind of viscerally felt how much the, the training data process mattered, and I kind of felt like, you know, why don't they let me get involved in the training data process? Like, that would be a better use of my time than building these models. And so, I wanted to make a company where the people doing the ML could actually have control over the training data collection process, and, and really get, like, visibility into it. Because, you know, at the time, I think the thinking was like, "Oh, this is sort of like a manual task. That's like more of like an operations team should deal with this." And, and they would like- Mm-hmm. ... they would do this thing where you, you'd like make this giant requirements document, and it's so like waterfall. Like, it would be like- Yeah, it wasn't iterative. Oh, it wasn't iterative at all. And it'd be like, you'd make like a 50-page document, and like you know that the people doing the labeling are not, like, reading that document, but you kind of need that to, like, cover your ass if they did, like, label something, you know, not the way you want. And it would've been so much better to be like, "Look, we're trying to rank search results. Like, put yourself in the mindset of, like, someone, you know, who's, like, looking at this, like is it good or bad?" Versus trying to lay out in like excruciating detail what makes something relevant or, or not relevant.
- SGSarah Guo
I think also at this time, like when, when you first started, um, I think originally it was called Dolores Labs and then CrowdFlower and then-
- LBLukas Biewald
Yeah.
- SGSarah Guo
... eventually Figure Eight. Like, I think I met you in your Dolores Labs days or something. (laughs)
- LBLukas Biewald
I know. I remember, yeah.
- SGSarah Guo
Yeah, yeah. And at the time, there weren't really, um, solutions for data labeling externally, right? Some people were using Mechanical Turk from Amazon to sort of run jobs on untrained workers. There wasn't, like, scale. There wasn't, you know, there was none of these services.
- LBLukas Biewald
Yeah.
- SGSarah Guo
And so you got really early to this idea of starting, like, a data labeling company, and that that was actually very useful for machine learning. And so it'd be great to hear, like, you know, what were the early days of, of that like, and what was the industry like, and how did you get all that running?
- LBLukas Biewald
Yeah, I mean, it's funny, right? Because back then, I was coached actually
- 8:16 – 18:54
Startup Evolution and Machine Learning
- LBLukas Biewald
quite a lot by, you know, Travis Kalanick, who's the, you know, famous now for, for doing Uber and other things. But he was like, "Don't tell anyone that it's, like, AI, like VCs, like, don't wanna hear AI," which was actually good advice, um, at the time, and it was good advice in the early days of the company. And-
- SGSarah Guo
Sorry to interrupt. I think one interesting side note on that, just from a Silicon Valley history perspective is Travis used to have these effectively, like, hackathons or meetups at his house called the Hackpad, and, uh, you know, I think you used to go those, you know, a bunch of friends of mine used to, and so a lot of startups actually had some impact or influence from Travis in those days, like, due to his fact of, like, you know, being another founder in the scene and kind of getting everybody together. And so it's kind of an interesting moment in time we're in history, and to your point, back then, like, AI wasn't really as popular as, as it, as it became later, so it's, it's kind of an interesting, like, side note.
- LBLukas Biewald
Well, I mean, not only was A- AI not popular, but, like, startups weren't popular, right? Like, my family didn't, you know, understand about startups, and I, I had graduated Stanford. You'd think I'd have all these great, like, connections, but it didn't feel like that. Like, I, I had no one who knew how to, like, raise money from VCs. I didn't know any, you know, VCs, or I didn't really know any, like, entrepreneurs, honestly. And we had this website for Dolores Labs in the early days, just trying to get customers, and I put my, my personal phone number. I actually remember I was like the first user of Twilio, because I needed to make a phone tree, and so I used Twilio's software, and then, like, all three of the founders came to my house to, like, help me, like, make that-
- SGSarah Guo
(laughs)
- LBLukas Biewald
... phone tree, like, work better, which is kind of amazing. It was like, you know, like, you know, one of those, like, you know, 20-something, like-
- SGSarah Guo
Yeah. (laughs)
- LBLukas Biewald
... um, you know, grungy apartments in the Mission. And then, uh, and then Travis called in, but, you know, it's funny, 'cause the phone tree, we were just trying to pretend like we were a big company. And Travis called in, 'cause of the phone numbers on the website, not 'cause he wanted to buy anything, but he just, like, thought it was, like, awesome. And so I'm just like, you know, I pick up my phone, and then there's just, like, this guy there and he'd just be like, "Oh, man."
- SGSarah Guo
(laughs)
- LBLukas Biewald
"Like, this is so cool," you know? I'm like, "Okay, like, who are you?" (laughs) You know? It's like-
- SGSarah Guo
(laughs)
- LBLukas Biewald
It's like, "Do you wanna, like, get coffee?" And, uh, and that actually turned out to be incredibly, uh, like, helpful. But then I, I think, like, the thing that was so different back then is that the people doing ML, there just weren't that many. Like, there were people, like, heavily investing in ML, but there, but it wasn't that many. And so what happened was, you know, we got, like, eBay as a customer, which was really mattered at, at the time, and we got, like, you know, Google as a customer, and Bloomberg, and then there just, like, wasn't anywhere else to go. So, like, you know, my board was always, like, recommending, like, read Crossing the Chasm, and, and we tried, like, a million different ways to, like, you know, grow the company. And, you know, I don't know, I hope this doesn't sound defensive, I mean, maybe I was just a bad CEO, but we had, like, years of, like, struggle because there was no chasm to cross, right? There was, like, nowhere else to go. So we tried all these different things to, like, you know, build more complete solutions for our customers, and it just didn't work. And then kind of all of a sudden, um, you know, autonomous vehicles got popular, and that really actually suddenly caused our revenue to, um, you know, start to, to grow really fast again. But it was like an eight-year lull of, like, you know, really no growth, right? So it's hard, 'cause we started off fast, got everyone real excited, you know, kinda got like womped for just, like, years and years and years. Actually, we had all these competitors. They all went away. So at some point, we had, like, no competitors left, right? 'Cause, like, everyone had, uh, had gone out of business. And then it was a funny experience, 'cause, like, Scale came along and totally ate our lunch on the, in the self-driving market, which is a market, like, I knew and loved. And so, you know, I, I was so excited to sell the company after, you know, so many years of struggle, you know? But then, like, right after that, we see, like, Scale just, like, skyrocketing in revenues, like, oh, man, like, I wish we had just, like, you know, maybe held on a little bit longer. But then, you know, it gave me the, the space to start Weights & Biases. So, you know, who knows? I, I, I wanna be like Daphne Koller and evaluate my decisions, like, accurately and, and critically, but it also does seem like, you know, I've had some good luck along the way.
- SGSarah Guo
Yeah, no, the market's shifted so dramatically, and I think, to your point, self-driving was the first time that you suddenly had a bunch of systems that scale, that people needed data labeling for. And then, of course, now we have this LLM wave, but it's all very, very recent, and I think a lot of people basically view ML as a sort of continuity and everything's always been kind of rising in a sort of almost linear way. And in reality, it's this very bumpy set of discontinuities in terms of the set of technologies and markets that people are adopting it in. And so, it's not continuous, it- it's a discontinuous thing, and nobody thinks about it that way. When you started Weights & Biases, you said something along the lines of, "You can't paint well with a crappy paintbrush, you can't write code well in a crappy IDE, and you can't build and deploy great learning models with the tools we have now. I can't think of a more, any important, more important goal than changing that."
And that's, I think, like, when you announced that you were starting Weights & Biases. And so I was just curious, like, what lapses in capability really got you going on, um, 1B? And can you also just, uh, you know, many of our listeners, um, know what it does, but for those who don't, could you explain what the product does and how it works?
- LBLukas Biewald
Sure. Yeah, so it's kind of constantly evolving, right, because we're saying that it's like a set of tools for, for people doing machine learning. We're best known for our first thing that does experiment tracking, which keeps track of, like, how your models, like, perform over time as they learn and train. But we also have a lot of stuff around, like, kind of data versioning, data lineage, you know, production monitoring, model registry, kind of the, the sort of end-to-end stuff that you need to do machine learning reliably. And I think the thing that happened to me was I had been running CrowdFlower for years, and I, I always loved machine learning. But I was, like, really starting to get out of date. Like, deep learning came along, and at first I was kind of skeptical of it because people are always saying, "Oh, I have a better model that's, like, magically better." And they, they're like, "Wrong, wrong, wrong, wrong, wrong. It's just, like, really, like, data." And then I'll, and but then, they were right, right? So there actually was a sort of a better modeling approach that worked, and I kind of realized, you know, when I was in my early 20s, I was really judgemental of, you know, the people in their late 30s that hadn't, like, adapted to machine learning at the time, 'cause it, like, rule-based systems were kind of all the rage-
- SGSarah Guo
Mm-hmm.
- LBLukas Biewald
... when the different generation was, was growing up. And I was like, "Wow, you know, I am actually getting out of date myself." Like, I'm saying these kind of wrong things that were true 10 years ago and are not true now, and I honestly felt, like, really bad about myself. And so, I did a couple projects to try to, you know, get up to speed. I started teaching free machine learning classes and, and deep learning classes to kind of force myself to, to learn the material and actually, like, interned briefly at, um, OpenAI where I was just like, "Look, I will just do whatever, you know, work you want." Just I want to be like, I need, I know that I need, like, an accountability (laughs) partner essentially to force me to learn stuff even though I love to learn stuff, it's like my favorite thing, but I always need accountability partners for anything I do. So I sort of used the students as an accountability partner and OpenAI. And then what was happening was I was showing my old co-founder, Chris, like, all the, the cool stuff, and he's, like, a really good engineer, and I'm, like, actually a really, like, bad engineer. Like, I'm, like, really lazy, and I, like, try to write the, like, you know, I'm just, like, like, people... My co-founders make fun of me all the time for, like, "You don't really know how Git works." And I just openly, I have no idea how Git works. I just sort of mash the Git keyboard-
- SGSarah Guo
(laughs)
- LBLukas Biewald
... until, like, I kinda, like, (laughs) you know, get in a bad state.
- SGSarah Guo
(laughs)
- LBLukas Biewald
And then I, like, call Chris and beg him to, like-
- SGSarah Guo
The CEO rebased. (laughs)
- LBLukas Biewald
(laughs)
- SGSarah Guo
The CEO rebased. (laughs)
- LBLukas Biewald
Yeah, I just, I don't know, I don't un- do- I mean, I don't understand it, and it, and it's like, my, my co-founders just find it, like, baffling-
- SGSarah Guo
(laughs)
- LBLukas Biewald
... that I wouldn't understand it. But-
- SGSarah Guo
(laughs)
- LBLukas Biewald
... I think it's, like, um, for them, you know, it's like, they're like, "Wow, this guy, like, needs some basic tools," you know? Like, 'cause you know, they're like, "Okay, like, reproducibility, like, why don't you just use Docker?"I think that's-
- SGSarah Guo
(laughs)
- LBLukas Biewald
... sort of the ops mindset. But I'm like, "Man, I don't understand Docker." I just feel like I install it on my, like, laptop, and then it's always, like, taking up memory and stuff. I, like, uh, I don't, like, don't really know what it's doing-
- 18:54 – 29:54
Open Source Models Implications and Adoption
- LBLukas Biewald
went to my board, and I was like, "I think there's, like, a real existential threat here." And I think they were like, "Hey, you know, we don't, like, see it in the data. Like, are you sure? Like, maybe you're being paranoid." And I guess I do feel sure. And I, I don't wanna say I'm, like, the only one or, like, paint myself as the hero. Like, you know, my co-founder is also seeing this and, you know, people talking about it. But it's sort of, like...You know, this threat is like now, right? And we have to actually, like get the whole company to, to do this thing, because it doesn't show up in any of our like metrics yet, but I just really believe that, you know, our customers are rational and they're gonna do a thing that like makes sense for them. And so, I see a lot of my colleagues being like, "Oh, there's gonna be like lots of different models." And it's like nice if it were true, but like what I see everyone doing right now on July 27th is using GPT. (laughs) Like I can see like 95% of the people out there, you know, using GPT for these ML tasks. And so it's like, look, we gotta support that. And so we really rallied the whole company behind it and, uh, we pushed out prompts. We'd also... This is really my, my co-founder's... My co-founder Shawn had really put a lot of effort into making our stuff really flexible because he's like, "You know what, Lucas? Like there's gonna be like changes, you know, coming. We don't know exactly what they are." But like, you know, kind of from the beginning we really tried to build very flexible infrastructure. So this was kind of a moment where we could really sort of like flex that and get out a, um, you know, a product for, for monitoring stuff. And you know, now it's like, you know, kind of it's our, our number one priority is getting out more tools for this new, this new workflow.
- SGSarah Guo
Out of curiosity, because, you know, there's a lot of debate right now in terms of proprietary models versus open source models, and, um, I think there's a really great quote. I think it's from Harrison from LangChain, which is, you know, no GPU until product market fit, right? You should first like figure out if the thing works at all or if there's a customer need, and that means using GPT. And then once you prove it out, you know, you may use GPT-4 or something for very advanced use cases, and then you kind of fall back to 3.5 where you start training your own model for things where you just want cheap sort of high throughput things happening. And it increasingly feels to me like people, the most sophisticated people who are at the farthest sort of cutting edge on this stuff, are kind of doing both, right? They, they use GPT to prototype and then in some cases they're, they're training their own instance of LLaMA 2 or whatever they're using. Do you think that's where the world is heading or do you really think things kind of collapse onto some of these proprietary models, like over time? Like it's six months from now, it's a year from now, it's two years from now. I'm just sort of curious about how you think about adoption of open source.
- LBLukas Biewald
You know, it's funny. I, I feel like lately what I've been telling people is like, I'm just trying to see the world clearly and as it is today. I can't predict the future and I can barely keep track of, you know, what people are doing (laughs) today when I consider it like my, my full-time job. So I, I'm like scared to prognosticate like what, you know, might be coming. But I, I think you're right that that's what's happening now. I think like there are like a bunch of things that could change, right? Like I think like, you know, GPT is way far out ahead and it's hard to fine-tune it, not even possible with, with GPT-4. And I think that that is like a little... That's not like a technical limitation, I guess sort of like a business model, um, you know, limitation. So that might change. I think that there's a lot of hidden costs to running your own model. I think people are really enamored with the idea of running their own model. And I've, I've kind of seen this before where I think at the end people do rational things, but it kind of takes them a while. So I'd rather sort of support what looks like the rational workflow. I mean, I think the insane thing, must be crazier to be an investor in this world, is like very, very few people have LLMs in production. Like there's probably more companies that have raised money as like LLM tools than companies that have LLMs in production, which is like insane. It's just like an insanely saturated tools market with very few people getting things out. But it's because it's so-
- EGElad Gil
When you... Lucas, when you say, when you say LLMs in production, you mean my own that I have fine-tuned, that I serve myself?
- LBLukas Biewald
No, sorry. I mean like G- like GPT, like using GPT in production.
- EGElad Gil
Oh, really? Okay.
- LBLukas Biewald
Look, I mean you, you may be like closer to this than me, but I-
- SGSarah Guo
It's a small handful, yeah.
- LBLukas Biewald
I'm like desperately trying to find them because like these are our customers. Like we, you know... Our stuff is just like... Our ethos is like we want to help people do things in production. So it's like if you're not in production, we're not relevant to you. So I, I like... I mean, back in January, February this year, we were looking for design partners that had stuff in production and boy was it hard to find, right? Like, you know, now there are more, but even when you, you know, you find people that are sort of like claiming to have this... things in production, it's sort of like, well it's like, you know, it's coming. Like, you know, we have like all these like sort of like prototypes, you know, running. And so I think it'll change. I think it's changing quickly, but I think it's a, it's a funny moment where... I mean, I think if you actually looked at the TAM today of like tooling for like LLMs, like I don't know, I, I betcha it's, um, it's small. And I think also, I think VCs maybe sometimes have this, this funny window where you see like all the companies that are using LLMs, but the enterprise adoption has been slower. I mean, despite the fact they talk about it like constantly, like constantly, like everyone's talking about it. But in enterprises like... Boy, I, I don't know if I've like used a product of like any enterprise that actually like was backed by a, um, an LLM. And there's a bunch of things that make it hard. It's like, you know, it's kind of unfair 'cause they've... this stuff has only been out for like six months or so, but it is like... I think the adoption may be, may be taking a little longer in the short term than people think.
- SGSarah Guo
Uh, I think that's a really key point because ultimately, you know, ChatGPT came out eight months ago and that was kind of the starting gun for all this stuff in my opinion. And then GPT-4 came out in March or something, right? Which is three, four months ago.
- LBLukas Biewald
Mm-hmm.
- SGSarah Guo
And if you look at enterprise planning cycles for large enterprises, it takes them six months to plan something, right? And so people often ping me and ask about adoption of these sorts of things and it's like, well Notion is seeing... you know, has adopted it in interesting ways already. Zapier has adopted it in interesting ways. But it's basically these technical founder led companies that jumped on it really early-
- LBLukas Biewald
Mm-hmm.
- SGSarah Guo
... relative to everybody else. And the big enterprises are gonna take another year or two because it's... they're just in their planning cycle still around this stuff. They just started really thinking about it and how to incorporate it and what to use it for. And then they're gonna have to prototype and experiment for a while and then they'll push it into production. And so that's why I was kind of asking a little bit about the future. I just feel like it's so early.
- LBLukas Biewald
Yeah.
- SGSarah Guo
And we're, we all talk about it, again, as if it's this continuous industry cycle, but it's really not. It's a disruptive new technology. And so, you know, I think a lot of it's still to come in really interesting ways.
- LBLukas Biewald
Oh, totally. And there's tons of product issues too, right? Like, you know, like Notion and Zapier both have these really compelling demos, and they're both products that I use, but then I actually don't use the LLM, like, piece of them myself, and I wonder... I have no insider knowledge of the level of adoption, but I think they're, I think they haven't gotten it, like, perfectly right yet, despite, like, a lot of thinking and, and really smart people working on it.
- SGSarah Guo
Mm-hmm. Sure. For the Core 1B product, you know, you folks are being used for a wide variety of areas around autonomous vehicles, financial services, scientific research, media and entertainment. Is there any industry in particular that you think you were either surprised by adoption of the product or you're really excited to see sort of how people are using it?
- LBLukas Biewald
Yeah, I mean, the one that stands out for me, because this is the one that's really different than, you know, my Figure Eight days, is pharma. So I, I actually think this is kind of flying under the radar a little bit, but every pharma company is making major investments in, in ML, and not just on the sort of, like... I mean, they do have these operations to sort of, like, sell more, you know, drugs to, to doctors that uses sort of, like, light ML, but I think the thing that's really exciting is, like, the actual testing of drugs, you know, before they, they have to test them in the physical world. And that's, like, obviously working, you know, super well, and I think... I, I, I've seen this before too with, like, autonomous vehicles and stuff. It's like, there's a big lag there, right, before you get something through, like, all the clinical trials. So, no drug developed by ML has gone through clinical trials, but if you look at the behavior of all of the big pharma companies, I can tell that it's working because they're hiring hundreds of people, right? Like, you know, like, companies will hire, like, a few people for, like, an experiment, but they're all gearing up to, like, operationalize this stuff, and that just gets me really excited. I mean, I could well be wrong, I suppose, and I don't really have any insider knowledge except for the seats that get bought on (laughs) you know, Weights & Biases, but when I see that, I, I get pumped 'cause I, I just, like... You know, the drugs that they're working on, you know, the diseases that they're curing, it's, like, the ones that, like, you know, like our relatives have, right? Like, you know, Alzheimer's and Parkinson's, and these are kind of horrible things, and I think there's just a huge promise in being able to do physics, like, inside a computer versus in the world.
- EGElad Gil
Yeah, I think there's a... I think that this is a really important point too. It's actually commonly said, like, no, no machine learning-developed drug has actually come to market today, but it's a backwards-looking metric in a very slow industry, right?
- SGSarah Guo
Yeah.
- EGElad Gil
Like, the clinical trials cycle is very long, and, and so, um, I'm actually, like, quite, uh, quite optimistic on this.
- SGSarah Guo
Yeah, and I think that's, uh, that, that stands out in pharma because i- it's very under-discussed, but there are certain venture funds that have done incredibly well financially in pharma where, there's one in particular I can think of that never shipped a drug until the COVID era, and they were in business for 20 years.
- LBLukas Biewald
Wow.
- SGSarah Guo
And they made all this money, and they funded all these companies, and none of their biotechs ever launched anything in the market.
- LBLukas Biewald
Wow.
- SGSarah Guo
So, I think that's a, that's a broader sort of issue with pharma, and we can talk about that, I think, some other time. (laughs) But it's, it's kind of interesting how, how little biotech has actually delivered. And there's been amazing deliveries, right? In terms of different drugs and things. But it's, it's actually more common than just the ML side, I think.
- LBLukas Biewald
Hm. Yeah.
- EGElad Gil
Lucas, you, okay, so pharma is something you're excited about and you think has promise and, and growth and, um, at least seats of 1B. Figure Eight, like you talked about, you know, Yahoo, eBay, like, it's a very small set of people. Who else do you see in the Weights & Biases, like, customer base now? Like, how has that changed since... It's, it's actually incredible to me that you've been, you know, working on this from the entrepreneurial side since 2007 'cause it's like, you know, pre, pre even deep learning revolution, right? And so, uh, I imagine, you know, you've got a much broader user set now.
- LBLukas Biewald
Oh yeah, it's so cool. I mean, the coolest thing about running Weights & Biases is the customer set is everyone. I, I really think every Fortune 500 company is doing something with ML that they, like, actually really care about. And, and it's always surprising, right? Like, we work with, you know, most of the big game companies. Like, I'm not a big gamer, so, like, I, you know, like, I'm vaguely aware of, like, Riot Games and, like, Unity and stuff, but, you know, th- but they do all this cool stuff with ML to, like, you know, make the games more fun, to make, like, you know, models in the games, and this is, like, big investments they really, really care about because, you know, again, we're sort of the last step in your journey as to what good tooling for your ML team. You kinda need something to work, so you hire an ML team. You get into production, then you, like, run
- 29:54 – 40:27
ML Impact in Various Industries
- LBLukas Biewald
into problems, then you come to Weights & Biases. So, like, we see stuff, you know, after it works. And, and, like, you know, like agtech, like, we work... You know, the big agricultural companies, I had, like, never heard of some of them when they showed up, and then they're, like, these huge, you know, businesses that are actually using ML to find ways to do, like, cleaner farming. Like, a lot of the reasons, you know, you, you spray a whole field with, with pesticides is just 'cause it's, like, so expensive to do something smarter. And so, you know, I think, I think that, like, crop yields and the, you know, the, the cleanness of the, the farming practices are about to, like, dramatically, um, improve. Like, we, you know, we worked with John Deere for years back from my Figure Eight days to, you know, Weights & Biases, and they're, they've deployed sprayers that only target the weeds in, in fields. It's deployed. It's like, you know, I remember, like, for years seeing pictures on the wall and them showing me, like, prototypes, and then one day they're like, "Yeah, you can, like, buy this." You know? (laughs) And it's, it's cool 'cause, like, this intelligence stuff, it's like software, right? So it's like it's not like a machine you just, like, press copy and then you have, you know, more of it. And so, so yeah, I mean, we see that. We see, like, a lot of, um, you know, I mean, fintech probably obvious to you guys, but, like, they are kind of I think always out in the forefront, you know, of this stuff for lots... I mean, like, there's, like, consumer-oriented stuff that you'd recognize, like, you know, making chatbots not annoying, right? And then there's, like, you know, kind of more, you know, financial forecasting and, and things like that. But yeah, I mean, it's funny, we, we don't do any vertical-based marketing because there's not one vertical that's, like, dominant enough to, to warrant it, and our customers bounce around between verticals so much that I think the common thread here is people doing, like, ML and data science versus any particular application, which I just think is super cool. That means it's sort of like table stakes, you know, for everyone.
- EGElad Gil
You, you know, made jokes, I think jokes about, like, not being a, a terribly good engineer, and now the Weights & Biases messaging is very much about developer first.... right? Can you talk a little bit about how you think about, like, y- you know, and it actually, it is, like, uh, a- as far as I understand, it's, like, one of the most broadly adopted tools by developers working on ML. How do you think about, like, developer adoption versus, like, researcher adoption, and what did you do that worked?
- LBLukas Biewald
Yeah, I mean, it's like developers and researchers, they kind of blend together. But I think that, I think that what happened in the sort of MLOps space is that you got a lot of... Well, the early companies had to sell to executives, which I totally understand. Like, that's what CrowdFlower had to do. And the, the problem there is you kinda get stuck in these, like, multimillion dollar deals, and, like, you just can't get out of that. Like, you can't switch to, like, a PLG motion. And so the early companies, I think, are kinda stuck, right? With, like, these products that, like, CIOs love, and the, you know, engineers hate. And that's just, like, (laughs) I just didn't want to do that with, with Weights & Biases, no matter how big the market is or how, like, juicy that is. And the good news is it's, like, not a good market. Like, a developer-oriented sale is better. When you, when you look at, like, developers versus ML researchers, that line has really blurred in the time that we've been doing it. And, and I think that, like, there's sort of, like, subtle differences. But, you know, when NVIDIA came along and these chips worked for deep learning, it just, like, broke the entire stack. Like, it was like a first time that in, in, in my career where I'm, like, running into, like, like, linker errors. I'm like, "What the fuck is a linker e-" Like, I vaguely-
- EGElad Gil
(laughs)
- LBLukas Biewald
... like, remember this, you know, (laughs) from, you know, like a CS class I took. You know, like... And, um, and so it's like, I think that ML researchers really had to be- kind of become software developers. And then at the same time, you know, the, the AI class is the most popular class, so, like, all these software developers are smart and just kind of become ML researchers. So I think that line has weirdly blurred. But then I, I think there's a funny thing that also's been happening where, like, every DevOps person on the planet rebranded themselves as, like, an MLOps person all of a sudden. And so you get all these companies that come out of... Like, every MLOps team then realizes they could raise, like, a shitload of funding, you know? And so, like, you got, like, every, every major company, their MLOps team, like, went off and, like, raised money to, like, make a new product in the market. Which I think from an investor, that's logical, right? It's probably they have a good thing. But they're just, like, not good at connecting with actual developers, right? 'Cause they're... Actually, like, DevOps is, is like a little bit of a different discipline, where you're sort of obsessed with reliability. Kubernetes seems, like, simple to you. And that's just not, like, the experience of, like, an ordinary, you know, developer, like, you know, like, like my co-founders or, or me. And so, I think the, the joy of Weights & Biases is we're kind of making software for, like, ourselves. And I think it turned out that, like, maybe in the median of my three co-founders was actually the, the target audience for us here. I think I skew more towards, you know, an ML researcher barely, but, you know, if I had to, like, pick one end of that spectrum and... You know, my co-founder, Chris, probably skews more towards software developer, and Sean's probably somewhere in between.
- SGSarah Guo
One of the things that's common to people, or to developers, is that they love to write their own tools, and they tend to really enjoy using open source over closed source solutions. How did you think about the open versus closed source approach, and how did you think about y- you know, making something that's valuable enough and good enough to overcome that natural inclination to just do it yourself?
- LBLukas Biewald
Well, it's funny. Like, I think the tools thing, I've always felt like, I've always felt, like, kinda proud of making tools for developers. Like, that's always felt, like, really good, because I think developers sort of know what quality is. Like, I mean, it's like I- I kinda like making a tool for someone that could make the tool (laughs) themselves, 'cause it kinda raises the bar. And that's definitely... My grandfather was, like, a pattern maker, which is like a sort of... You know, like the person who makes a pattern for the machinist. And he had the same attitude of like, "Look, I'm making this stuff for, like, other engineers," and that, like, there's, like, an honor in that. So I definitely feel that pressure and love it. The open source versus closed source thing was really just, like, we didn't know how to make an open source business. So, so, like, we kinda started off closed source 'cause we just, we actually wanted to have, like, a working business.
- EGElad Gil
(laughs)
- LBLukas Biewald
And it, it's had a, like, a pro... Th- there's been a major pro, which is that all our competitors are clo- are open source. And what that means is that they don't get to see how users actually use their software. And so I think our software is a lot more ergonomic, because we have, like, metrics on what people actually click on. If people aren't clicking on a button, we remove it. If people, like, you know, pick an option all the time, then we know to, like, make that the standard option. As we've grown and you kinda can't just, like, rely on anecdotal user feedback, that, I think, has made our product, like, a lot better. Like, people find it, like, nicer to use. At the same time, I understand why people wanna go to open source stuff, but honestly, I feel like it's a little bit of a DevOps mindset also. Like, I mean, DevOps people, like, they f- they're obsessed with, like, you know, open source. And usually, like, the MLOps people we talk to in companies really want, like, an open source piece, which is why our client is open source, everything that actually runs in your servers is open source. But, like, I don't know, like, ML researchers aren't so precious in my experience, generally. They just, they kind of want to get a job done. And I think they're kinda happy to, like... that we have, like, a stable, like, business that generates money in, like, a normal way and, and isn't going anywhere. Or at least that's what I tell myself. (laughs)
- EGElad Gil
I think this is... Like, the, the part about, like, the need for, like, ongoing telemetry and application feedback, like, there are a, you- you know, zero to marginal number of open source applications that have har- actually succeeded. I think part of it is, like, the sort of, you know, hierarchy of honor of, like, the deeper in the stack you go, like, do people really wanna work on, like, web UI in the open source, or just, like, random business logic on a relational database? Like...
- LBLukas Biewald
Yeah.
- EGElad Gil
It's not as sexy and exciting to, like, go put your, like, GitHub badge on. But I think the piece that you described is actually really important, where, you know, you work on complex workflows. And if it's something that, like, somebody can just run in infrastructure and, like, you know, you, you get data back on, like, config files-
- LBLukas Biewald
Yeah.
- EGElad Gil
... or YAML or whatever, like, that might, that might work in terms of, like, one person's architectural point of view or some framework. But I really don't think it works at the application layer for, for these two reasons, right? Like, one, total lack of feedback, and two, sort of the lack of interest in the, I don't know, technical brownie points you get for it.
- LBLukas Biewald
Yeah.
- EGElad Gil
Do you still pay attention, I'm sure you do, actually, to, like, annotation? Like, what do you, what do you think happens to the data a- data annotation space in, like, you know, the land of LMs and RLHF and such?
- LBLukas Biewald
You know, I'll be, like, honest actually. Uh, this will be, like, totally honest. I find it, like, incredibly stressful 'cause I still feel bad that we lost to Scale.
- EGElad Gil
(laughs)
- LBLukas Biewald
Like, I still, like... It's just, like, lingered with me, and I, I admire Scale. Actually, I know how hard that, that business is, so I have just, like, deep admiration for their, like, execution. But as a competitive guy, I kinda can't get over it. So, I'm, like, always inundated with questions from VCs, like, whatever any annotation company's raising, I know about it 'cause everyone, like, calls me. But I, I honestly try... I know I should be closer to it, but I try to stay away from it just 'cause it causes me so much anxiety to look at what's going on-
- EGElad Gil
(laughs)
- LBLukas Biewald
... that I, uh, I just can't deal with it.
- EGElad Gil
What were some of the things that you did differently with the second company? I feel like, you know, I've started two companies, and with the second one, there's all sorts of lessons I applied immediately. Were there two or three key takeaways that, when you started Weights & Biases, made the second time around easier? Was it harder? How did you think about, you know, key, key, key learnings or how to apply new things?
- LBLukas Biewald
Yeah, I, I mean, I think, like, one thing was, like, extreme clarity about who we were serving. So, I'm, I'm surprised I don't hear this more 'cause, like, the, the... Weights & Biases started with a, with a customer profile, and I think it's actually a nice way to start a company because, you know, especially as, like, a founder, you have to spend so much time with your customers. You have to seek them out. Like, picking a customer that you love I think is a really good thing for your, like, mental health, you know? And so, that was, like, a big thing. And then I think, like, I think I've just been a more confident person in myself. Like, any time I start thinking, like, "Okay, like, long-term or short-term?" It's just, like, you always want to think long-term. Like, everybody wants you to think short-term. Like, everyone's gonna push you to think short-term. They wouldn't say it like that, but it's like, you know, it's like people can see, like, ARR growth. They can see, like, user growth. They... It's harder to see, like, product quality, right? And so I think, like, I think I'm a competitive guy who likes, you know, metrics and likes accountability. But I actually think that can get counterproductive for me, where, you know, you start, like, sacrificing
- 40:27 – 43:44
Advice for AI Company Founders
- LBLukas Biewald
short-term things to grow these external-facing metrics, and I just really try to fight that myself. I think everybody, like, chases... Every entrepreneur chases, like, short-term, like, ARR numbers, like, in quarter, but then it, like, hurts your growth rate the next quarter.
- EGElad Gil
(laughs)
- LBLukas Biewald
It's like, it would actually be better always to, like, push out deals, but, like, nobody thinks like that, right?
- EGElad Gil
(laughs)
- LBLukas Biewald
You can't think like that. But it's, it's... I don't think it's totally rational.
- EGElad Gil
Is there any advice that you would give to founders who are running their first AI company or just getting up and running?
- LBLukas Biewald
Yeah, you know, the advice I always give is like... It's, like, the generic advice that everyone says. It's, like, even truer than you think. It's even truer than, like, I know, even though I, like, deeply believe it. So it's, like, caring about, like, if you're making something people want. Like, everybody knows it, but, like, no one cares about it enough, right? Like, people just... They get distracted. They do other weird stuff. Even I do it. I understand. But, like, you should care more than you think, no matter how much you think. I've never met anyone that cared too much about that. And then spending time with customers. It's like, it's so critical. Everyone says they d- do it, but I don't really believe it. Like, I feel like I'm obsessed with this. I mean, like, getting, like... When you're an early company, getting, like, three customer calls in a week? That's, like, tough, man. I mean, you gotta, like, scrape and claw and, like, beg to get those meetings, and you know, like, two of them are gonna, like, cancel. So, I don't know. People tell me, "Oh, I met with, like, 30 customers this week," or something. It's like, "Really? Did you?" Like, I don't know. I, I, I-
- EGElad Gil
(laughs)
- LBLukas Biewald
... try, like, really hard to get customers' attention. (laughs)
- EGElad Gil
(laughs)
- LBLukas Biewald
Like... So, I don't know. I have this feeling that nobody does enough of that, but I don't really know. I think people are all lying to each other about how much, like, actual kinda customer meetings they're doing. And then it's like, you know, when you get to a customer, it's so precious. It's just like... Man, like, show up prepared and, like, ask the tough questions. Like, I think, like... I feel like one thing about me is, like, I always, like, default to, like, wanting people to like me, and it's a terrible trait in a, in a CEO. You know, it's like a... I feel like I have all these, like, coping mechanisms for myself to, like, not just, like, kind of flip into that mode. But I think it's good for customer discovery 'cause I'm always, like, so afraid that they secretly, like, hate my product, you know, that I, I get, like, really insecure and-
- EGElad Gil
(laughs)
- LBLukas Biewald
... I'm just like, "Okay, like, you know, tell me, like, more. You know, like, like, are you sure this is really, like, working for you?"
- EGElad Gil
(laughs)
- LBLukas Biewald
I actually think it does actually help in that one (laughs) important, like, entrepreneurial process-
- EGElad Gil
(laughs)
- LBLukas Biewald
... to lean into your insecurities with your, with your early customers.
- EGElad Gil
Uh, Lukas, this has been great. Is there anything you wanted to talk about that we didn't cover?
- LBLukas Biewald
No, this has been fun. I mean, I just... I think the message that I'm trying to tell the world is that we're really trying to make tools for this new LLM workflow that people are calling LLMOps, and so my, my advertisement for Weights & Biases is like, "Hey, if you knew us and liked us for our MLOps stuff, try our LLMOps stuff called Prompts." I think it's... I think it's not amazing yet, but I think it's kind of ahead of the market and it's about to get a lot better 'cause we are, like, investing every, every resource that we have into making it as good as possible. And we're really listening to feedback and iterating, so if people wanna, you know, email me directly and tell me some issue they had with Prompts, I, I really want to hear it.
- EGElad Gil
Is it, is it lukas@1b.com?
- LBLukas Biewald
Yeah. Lukas with a K, yeah, @1b.com.
- EGElad Gil
Okay. You're gonna get a flood. Um, well, I'm-
- LBLukas Biewald
I'll-
- EGElad Gil
... I'm optimistic you're such a pioneer here. Thanks so much for doing this, Lukas. It was great.
- LBLukas Biewald
Thanks so much.
- EGElad Gil
Yeah, thanks for joining.
- NANarrator
(instrumental music)
Episode duration: 43:44
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode mU15NfCqDr0
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome