EVERY SPOKEN WORD
65 min read · 12,949 words- 0:00 – 0:09
Intro
- AMAndrew Mayne
[upbeat music]
- 0:09 – 0:43
DevDay kick-off: Building AI products with real-world stakes
- AMAndrew Mayne
Welcome to the OpenAI Podcast, where we're live from OpenAI DevDay. Here sitting with me from SchoolAI is Caleb Hicks. Caleb, hello.
- SPSpeaker
Hi, thanks for having me. This will be fun.
- AMAndrew Mayne
So, Caleb, you're working on tools for helping educators and helping people, uh, basically in the classroom understand progress of students?
- SPSpeaker
That's right, yeah.
- AMAndrew Mayne
So first off, what was your reaction so far to DevDay?
- SPSpeaker
Uh, a ton of fun, I think makes it... Uh, a, a, a lot of things to be excited about that help us build, but also help students and teachers be more creative as well, so that'll be fun.
- 0:43 – 1:06
SchoolAI’s mission: Safe, managed AI tutors for students
- AMAndrew Mayne
So what have you been working on over the last year? What has changed with AI that's accelerated what you've been doing?
- SPSpeaker
Ooh, um, I think probably the biggest advancement over the last year for us- so we, we, uh, put AI in students' hands. That's the main, uh, the main thing that we focus on, is safe, managed, uh, AI that can act as kind of one-time personal tutors for students.
- AMAndrew Mayne
Mm-hmm.
- 1:06 – 2:35
What changed in the last year: Model intelligence + cost improvements
- SPSpeaker
And so probably the biggest change from OpenAI has been model progression.
- AMAndrew Mayne
Hmm.
- SPSpeaker
I think we get two advantages from that. One is, uh, significant leaps in intelligence.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Uh, and the other one is, uh, you know, improvements in cost. Because we are working with, uh, an industry that isn't known for paying big dollars for software, uh, it's been important for us to be able to manage students using this in a cost-effective way. So those have been the two areas that AI progression has helped. From our work, it has been a lot of, uh, orchestration, which I'm sure we'll talk about a little bit.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Just, just getting different AI agents and models to work together, uh, for the best outputs for students in particular.
- AMAndrew Mayne
So a couple of the releases we saw today, one was the Agents SDK.
- SPSpeaker
Yeah.
- AMAndrew Mayne
Now, and you've talked about that. How much have, one, tools changed the ability of, one, to work faster, and, two, the scope of what you find is capable now?
- SPSpeaker
Yeah, I think we're seeing teams across industries work way faster-
- AMAndrew Mayne
Mm
- SPSpeaker
... and building better software because they've got kind of this always on expert, hyper-senior, uh, engineer next to them that they're pair programming with, right? And, uh, so we see that with our teams as well, and that just allows us to, to build better software faster, uh, and get it in the hands of teachers and students, which is what we're here to do.
- 2:35 – 4:02
Educator adoption curve: From banning AI to teaching AI literacy
- AMAndrew Mayne
What has been the biggest shift you've seen in talking to educators or people like that in regards to AI in general?
- SPSpeaker
Yeah, great question. So every teacher, school, and district is on a very similar journey. It starts with permission, right?
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Two-and-a-half years ago, it was everyone under the sun was just banning AI altogether.
- AMAndrew Mayne
Right.
- SPSpeaker
Uh, we've, we've moved past that into productivity-
- AMAndrew Mayne
Mm
- SPSpeaker
... as teachers and, and school leaders realizing, "Hey, this helps me in my job."
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Uh, the leading schools are starting to get into that, that really important spot, which is recognizing that every student has to know how to use this stuff.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
If you're going into- if you're graduating from high school and you're competing for colleges or jobs and you are- don't know how to use AI yourself, um, you're at a severe disadvantage. So, um, most people are now orienting to, like, "Yeah, we have to teach this." Uh, we think there's a couple of special steps beyond those two, uh, where, where you get to better support students, the AI tutor in every kid's pocket, right? But we think that is... It's got to be classroom connected, it's got to know what you're doing in class, and it's got to know, um, you know, where you're trying to go and help you move that direction. Um, that last step we get really excited about is how you can put AI to work with teachers, uh, families at home, school leaders, kind of the system at large, uh, to really make school awesome for students.
- 4:02 – 6:45
SchoolAI product stack: Dot, teacher tools, and real-time student dashboards
- AMAndrew Mayne
Could you tell me a little bit about the kind of stack you're using from both the teacher-facing, student-facing back end, or what you're working on in that regard?
- SPSpeaker
Yeah. From, from the product side, we have kind of three different, uh, parts of the product that the, that students, teachers, and school leaders use.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
So there's, uh, kind of just a, a pretty basic, uh, AI assistant, right?
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Um, the, the, the GPT wrapper, as they say. [chuckles]
- AMAndrew Mayne
Right. Right.
- SPSpeaker
Right? Uh, but, but tuned to use cases in schools. I think a big thing that we felt like was really important is teachers should never have to become prompt engineers.
- AMAndrew Mayne
Right.
- SPSpeaker
Uh, so-
- AMAndrew Mayne
You know I was a prompt engineer.
- SPSpeaker
Yes. [chuckles] Yeah, so, so we do a lot of that, uh, kind of extra orchestration-
- AMAndrew Mayne
Mm
- SPSpeaker
... uh, to, we essentially enrich every prompt that, that a teacher writes to get better output for them, for their, what they're teaching, uh, their grade level, all that stuff. Um, so there's, there's that. Uh, we call that Dot. It's a fun, little, blue-
- AMAndrew Mayne
Okay
- SPSpeaker
... animated character. Uh, and then we have tools. It's a form, you fill it out, and it gives you an output: a lesson plan, adapted reading content, uh, things like that. Those, we l- we like to call those the 101, like the-
- AMAndrew Mayne
Mm
- SPSpeaker
... the checkers-level features, that kind of table stakes that you've got to give to, to teachers for them to move from, "Can I even allow this?" to, "This is useful for me." Uh, but the special part is when you start, uh, doing kind of these one-time, guardrailed, safe, managed AI tutors that the teacher can create-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... give to their students, students are interacting, and then the teacher gets a real-time dashboard of how the students are doing, what they're doing with the AI. And, uh, to make that very concrete, uh, the last five, 10 minutes of class, a teacher may give what's called an exit ticket to their students. Um, so they've got a recording of everything that they did during class that day, and it loads up and says, like, "Hey, how'd the content go today?"... and it does almost a, what's called a, a formative quiz. It, it asks them questions, and then it's coaching them on the ans- uh, on, on what they learned, what they want to learn, where they might go next, tease them up for whatever homework they might have. Um, and then it will do just kind of an emoti- a, a kind of a social, emotional check-in. It will say like-
- AMAndrew Mayne
Mm
- SPSpeaker
... "Hey, how, like, how was class? Um, what feedback do you have for the teacher? Uh, what are, what are you looking for out of using this content?" And rolling all of that up to the teacher so that they know how to better support those students in the future. Um, a thing a lot of us don't recognize about teachers is they might be working with 300 students at a time.
- AMAndrew Mayne
Mm-hmm.
- 6:45 – 8:41
Classroom reality and the “GPS for impact” idea
- SPSpeaker
I had 42 desks in my classroom when I was a teacher.
- AMAndrew Mayne
So you started off as a teacher before all this-
- SPSpeaker
I did
- AMAndrew Mayne
... that informed a lot of what you're doing.
- SPSpeaker
Yeah, yeah. So I had 42 desks in my classroom, and I was teaching seven or eight periods a day.
- AMAndrew Mayne
Mm.
- SPSpeaker
Every day, teachers have to make this impossible choice: Do I work with, uh, the top 10% of students who love this, they get it, they want more of it, they'd stay after school if I let them; the 10% of students that are really struggling, maybe that's because they don't understand it, maybe it's a learning disability, maybe it're a problem at home, maybe they got bullied in the last class, or everyone else?
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
And I care a lot about that everyone else, that middle 80%-
- AMAndrew Mayne
Yeah
- SPSpeaker
... because I was one of those students, and most of us were, definitionally. And we think about what we're building, uh, and what we've been able to build with, with OpenAI and some of the tools that were announced today, we'll be able to do even more of, is, uh, give teachers almost a GPS for impact.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Like, "These four students really need you today," and you can jump in and support those four students in a way that you maybe wouldn't have even known that they had a concern.
- AMAndrew Mayne
That- it brings up a very, I think, interesting point. When I talk to developers, often people confuse the tools for the product.
- SPSpeaker
Mm-hmm.
- AMAndrew Mayne
And what you have to do is you have to both understand the needs of the educator. That comes from you working in the classroom and your peers working at SchoolAI and understanding that.
- SPSpeaker
Yeah.
- AMAndrew Mayne
And I think that's the thing you're able to bring to it, where you look at this and go, "This is a platform to build on top of it," and that's something where you've identified all these areas that you can bring into to specialize it and make it very custom what you're doing.
- SPSpeaker
Yeah. I think that's something exciting we saw today-
- AMAndrew Mayne
Mm
- SPSpeaker
... in, uh, in the announcements, was, uh, opportunities for, uh, people like me with subject matter expertise-
- AMAndrew Mayne
Mm
- SPSpeaker
... with that, that really know a domain and were able to build kind of with you all, uh, with OpenAI, not just on top of OpenAI-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... which is, which is, a, a really cool unlock for a lot of people.
- 8:41 – 13:48
DevDay tooling for education: Agent Builder, MCP, and evals at scale
- AMAndrew Mayne
What are you most excited about that you saw today?
- SPSpeaker
The Agent Builder, for sure.
- AMAndrew Mayne
Agent Builder?
- SPSpeaker
Yeah.
- AMAndrew Mayne
Yeah. Yeah, that looks like, uh, it could be a lot of fun. And going back to when w- you know, you, you know how it was when you first started, you had to just wire up a lot of that yourself-
- SPSpeaker
Yep
- AMAndrew Mayne
... and code tools certainly make that easier, but just being able to drag and drop, think something like file search, the permission structure seemed really well thought out, and particularly what you're doing in a classroom situation-
- SPSpeaker
Yeah
- AMAndrew Mayne
... where you really have to have those safeguards in there.
- SPSpeaker
Yeah. Uh, we actually built our own-
- AMAndrew Mayne
Really?
- SPSpeaker
... and have for a number of years, and I think we're excited to go and get our hands on, on it and see what do we get to get rid of [chuckles] -
- AMAndrew Mayne
Yeah
- SPSpeaker
... what do we get to, uh, make even better now that we have access to it ourselves, um, o- o- or, or from you all? I think that's, uh... Either way, we're gonna learn a lot from working with it-
- AMAndrew Mayne
Yeah
- SPSpeaker
... and, and that makes it easier for us to use, but also, again, bring it into the hands of these people that aren't technical. They're not developers. They're not, they're not thinking about, you know, which model they're using, but they just want to get their stuff done.
- AMAndrew Mayne
Yeah, I think that's a great point in that the more you can have the people who will understand who your customer is, in this case, students and teachers, understand what that problem is in less time, they're spending time to wire things up and run, you know, fast API servers and stuff to do this-
- SPSpeaker
[chuckles]
- AMAndrew Mayne
... seems like it can make the produ- product get a lot better faster.
- SPSpeaker
Absolutely.
- AMAndrew Mayne
What else have you been excited about that you've seen?
- SPSpeaker
Um, I think, uh, we've, we've built some similar things, and in our product, we have this concept called power-ups, which are-
- AMAndrew Mayne
Mm
- SPSpeaker
... basically apps that you can use with Dot, our, our character.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
I think we got to see some good patterns and examples of that today with the new apps that were, that were announced, and I think f- one of the things we're really excited about is these partners that we've been developing, uh, that everyone is really solidified around MCP servers-
- AMAndrew Mayne
Mm
- SPSpeaker
... as a way to communicate with, uh, AI. And so, um, a thing that will definitely benefit us is OpenAI kind of drawing a line in the sand, saying, "We're doubling down on this."
- AMAndrew Mayne
Yeah.
- SPSpeaker
Uh, and so now when we go to a partner that's already building for integration with ChatGPT, they can also bring that directly into SchoolAI, again, for that safe, managed, guardrailed experience-
- 13:48 – 15:32
Jam.dev introduces “Please Fix”: editing websites like a doc, shipping PRs automatically
- AMAndrew Mayne
Caleb, hope you enjoy the rest of DevDay. So we're gonna be talking to a few other developers about some of their different experiences, and I'm always fascinated to see what it gets accelerated and how they're able to focus when a new tool comes out on really what is their core thing they're working on. So up next, we have Dani Grant, who's with Jam.dev, which is an in-browser tool for helping you evaluate your site and to figure out basically how to improve it. Is that correct?
- SPSpeaker
I'm so happy to be here.
- AMAndrew Mayne
Great.
- SPSpeaker
And actually, later today on stage, we're gonna announce a brand-new tool.
- AMAndrew Mayne
Oh, boy!
- SPSpeaker
It lets any PM, designer, marketer... Oh, I'm a little short, aren't I? [chuckles]
- AMAndrew Mayne
You're good. No, no, no. Microphone's too tall. It's all good.
- SPSpeaker
I agree. Um, well, just like you just did, it helps anyone fix what's broken instantly without writing code. Um, it's called Please Fix.
- AMAndrew Mayne
Okay. So I would use this, like my website, go into it and say, just, "Please fix," or?
- SPSpeaker
Yeah, I mean, like, yesterday, before [chuckles] this tool was announced-
- AMAndrew Mayne
Right
- SPSpeaker
... um, if you wanted to change, like, some copy on your site or, like, a, like, a button didn't look good or missed a hover state, you'd have to probably ask an engineer to do that for you.
- AMAndrew Mayne
Right.
- SPSpeaker
And the engineer would-
- AMAndrew Mayne
Nobody wants to do that. Nobody wants to do that.
- SPSpeaker
Yeah, they're like, "Okay."
- AMAndrew Mayne
No.
- SPSpeaker
And then they'd be like, "Can you make a ticket?" You'd be like-
- AMAndrew Mayne
Yeah
- SPSpeaker
... "Okay," you know? [chuckles] And then the pri- the ticket gets prioritized-
- AMAndrew Mayne
Yeah
- SPSpeaker
... and then, like, maybe they get to it.
- AMAndrew Mayne
I worked for a company, I won't name it, where we joked it was easier to release a model than put up a blog post.
- SPSpeaker
That's hilarious.
- AMAndrew Mayne
Yeah. Won't name it.
- SPSpeaker
Um-
- AMAndrew Mayne
Yeah
- SPSpeaker
... well, so, so now, as of today, what you can do is, while you're looking at your site, you click on the Please Fix browser extension-
- AMAndrew Mayne
Wow
- SPSpeaker
... and then it lets you edit your site right there, like it's a Google Doc, like it's a Figma. And then when you're ready and you like how it looks, you just click Submit, and it creates the PR for you.
- 15:32 – 18:35
A new web paradigm: apps inside ChatGPT and “read, write, think”
- AMAndrew Mayne
Very cool. What were you most excited about what you saw today?
- SPSpeaker
I mean, I think we just saw a new way to browse the web. Like, I think OpenAI maybe just changed what we mean by the web, what we mean by a browser. Like, if you think, like, Web1 is read, Web2 is read, write... Okay, there's Web3.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
And then maybe this is, like, Web4, like, read, write, think. I think we just saw a whole new way of experiencing the web, where it's a lot less mechanical, and it's a lot more stream of consciousness.
- AMAndrew Mayne
So yeah, talk about the apps inside ChatGPT, right?
- SPSpeaker
It's freaking cool! [chuckles]
- AMAndrew Mayne
Yeah, it is. It's... It makes you think a lot about when you're building something, what the core functionality is, like, what the purpose of it is, and the idea that if it's going to be presented inside ChatGPT. And so watching the demo where they interacted with, with the Zillow website and actually having to have data about that and be able to drill down into it, and it's, it's exciting to think about the possibilities there. And so I could see for you all, that seems like a pretty interesting area because people, as they focus on usability, they're gonna want to make sure those things work really well.
- SPSpeaker
Yeah, like, now when you build your inside-of-ChatGPT app-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... if you're the PM, if you're the designer, and you want to tweak some things and make, make the app look a little nicer, you can just use our browser extension, change it from the ChatGPT interface-
- AMAndrew Mayne
Mm
- SPSpeaker
... and make the PR in GitHub. It's so cool. And I think, you know, these tiny tweaks, they often don't get prioritized today, but they should.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Because the difference between a fine design product and a well-designed product is that the well-designed product changes the world. Like, the iPhone changes the world because it's usable, and there were actually attempts before that, that I won't name, that many people under the age of 30 have never heard of, um, because they just, they didn't have that attention to detail.
- AMAndrew Mayne
Mm-hmm. Have you seen your development process change with tools that have been coming out, particularly AI tools?
- SPSpeaker
Yeah, I mean, I... Like, I'm biased. We're- we've been using Please Fix as we've been building Please Fix.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
And it, like, even today, our pricing page changed dramatically because a PM could just go in and test a bunch of stuff without asking an engineer. I think the thing I'm most excited about is this idea that you can move as fast as your entire creative team altogether, like, no disruptions for engineers. It's like people can move without having to have bottlenecks on a few people.
- AMAndrew Mayne
So it seems kind of cool because what we're talking about is the idea that people who aren't specifically engineer background are able to make those changes, and has that been something you've seen basically within your company, the adoption of tools like Vibe Code and stuff like this for people to contribute and be able to come up with ideas?
- SPSpeaker
Yeah, it's really cool. Um-... when we, we talk to our users every day, and some of the stories we hear are awesome. Like, last week we talked to a user who's a firefighter, and building software for firefighters. That's cool.
- AMAndrew Mayne
Mm.
- SPSpeaker
Um, talked to a user a couple weeks ago who, um, grew up in the church system and is building software for churches.
- AMAndrew Mayne
Mm.
- SPSpeaker
Has no software experience, but is able to now make something that's very impactful to their community. I think that is awesome. I think we are about to see the Cambrian explosion of software, the same way that with the web, there's the Cambrian explosion of, like, news sources with, you know, Substack and Twitter. I, I think this is about to happen for software, and it's... I think it's one of the best things for humanity.
- 18:35 – 23:29
Jam.dev’s product philosophy: optimize for user “wow,” not just features
- AMAndrew Mayne
So how do things work at Jam.dev as far as ideation, testing things with customers, releasing, and then figuring out what's best features and whatnot?
- SPSpeaker
The only thing we care about is, does this deliver a wow for our users when they use it?
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
And so that's what we're focused on. There is an emotional response we want people to have when they use our product, because we work on the worst part of software development. Like, I don't know anyone who is like, "Today, I want to fix some bugs."
- AMAndrew Mayne
Right.
- SPSpeaker
And so our job is just to make that whole part of the software experience a lot better, and so we're laser focused on that. We're just constantly talking to users. Every user that signs up from Jam hears from a co-founder. Every user who uses Jam hears from a PM. We are just co- in constant contact.
- AMAndrew Mayne
When I worked at OpenAI, one of the most exciting things, and it sounds absolutely silly today, was when we watched GPT-3 spit out a React button. That was like, "Oh, my gosh, it could do..."
- SPSpeaker
I remember that.
- AMAndrew Mayne
Yeah.
- SPSpeaker
I, I don't remember it inside, but I remember from the outside.
- AMAndrew Mayne
Yeah.
- SPSpeaker
I remember seeing demos of that, being like, "The world just changed."
- AMAndrew Mayne
That was, that was the threshold for us being impressed, was literally, like, four lines of code, and to be able to, "Oh, that's really cool, it can do that." What has been a big aha moment for you?
- SPSpeaker
I think... Look, it's, it's, it's similar to what you're saying, but when you think of what you just described-
- AMAndrew Mayne
Mm
- SPSpeaker
... coupled with what was just announced with the apps SDK, you can imagine that in the future, yes, humans are gonna be building a lot of apps-
- AMAndrew Mayne
Mm
- SPSpeaker
... that are shown dynamically as they browse the web, but also, I think that agents will dynamically build apps-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... for you as you browse the web. And that opens up a whole world of possibilities, because you can have dynamic software right there. Like, imagine inside of an organization. Right now, if a PM wants to see how a product is doing, they have to build a dashboard, and it's a lot faster to do that today than even six months ago.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
But imagine the PM needs a dashboard, and they're in ChatGPT, and ChatGPT just gives it to them-
- AMAndrew Mayne
Right
- SPSpeaker
... and no human had to do that. It means that there are two types of software. There's sort of long-term software that humans are gonna work on. They're gonna, like, really fine-tune, make it great, like Zillow, Canva. And then there's going to be sort of disposable, one-time use software that an agent can just whip up, and that is just freaking cool.
- AMAndrew Mayne
Yeah, it's, it's a thing where, you know, as a developer, you're used to sometimes spinning up a tool one time to use that, and then you're done with it. But it's another thing to think about that could be a new modality. You know, we get kind of fixated in sort of, like, the App Store sort of idea, and I think that there's gonna be, like you said, long-term stuff that you deploy inside of ChatGPT, but then the ability for somebody just to spin up a thing they're only gonna use once, and that's the end of it.
- SPSpeaker
Yeah.
- AMAndrew Mayne
And have you seen, as far as things that sort of start off as hobby projects or things like that, or maybe inside Jam.dev, people have said, "Hey, I had this idea I wanted to solve for this thing," maybe even something from finance or comms or something, that's turned out to be something useful?
- SPSpeaker
Yesterday we were, um, at the rehearsal for our 10X codebase session. It's four startups. We're all demoing, and we're just sitting around, kind of backstage talking, and we're like, "Oh, how'd you start your startup?" And almost everyone, [chuckles] three out of four, I'm the only one, um, where their startup started as an internal tool at their company that they needed. And actually, these startups pivoted to the internal tool because they found it so tremendously valuable to how they build software.
- AMAndrew Mayne
Well, that's, you know, how Slack got its start, and other companies. And it's a very interesting thing, where the thing you spend the most time on is often the thing that's gonna be the best product. And, you know, one of the things they talked about OpenAI on here today was, like, how 70% of the code's coming from, you know, the PRs are coming from, you know, generated by Codex. And it's a very interesting thing to see. When a company's using their own tool to build the tool, it seems to iterate much faster.
- SPSpeaker
It's actually so funny, because if you listen to standard startup advice, like, what would Paul, Paul Graham tell you? It's, "Hey, don't optimize the internal processes."
- 23:29 – 25:59
Future direction: self-improving agents via automated evals and prompt optimization
- AMAndrew Mayne
What are you looking forward to? What kind of tools would you like to see that don't exist yet?
- SPSpeaker
Um, I... Well, I was, I was talking to the engineers sitting around me during the keynote.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
And they were really excited about the optimizer and the evals-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... as part of the Agent Kit. And, um, and what they wanted to see was, well, like, what if the evals could be written automatically for you using your own data, and then use that to automatically optimize your prompt?
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
So rather than people sort of sweating the details about these agents, what if the agents could improve themselves? And I think if we had that at Jam, like, we could move a lot faster, and we can make a lot more powerful software a lot sooner.
- AMAndrew Mayne
Mm. Uh, what advice do you have to founders or developers right now?
- SPSpeaker
It's, it's never been a more fun time to build.
- AMAndrew Mayne
Mm.
- SPSpeaker
I think we all just get to enjoy it.
- AMAndrew Mayne
... So how did you figure out that this was what your team wanted to work on, which Jam.dev? How did you decide this was the problem you wanted to solve?
- SPSpeaker
When, uh, my co-founder and I were product managers together at Cloudflare, we, we worked on the fastest moving team in the company. It's this sort of skunk works team that would try to do big, new things. It's the team that shipped Cloudflare Workers, um, their cloud compute platform.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
It shipped 1.1.1.1, which now fields a trillion DNS queries a month.
- AMAndrew Mayne
Right.
- SPSpeaker
Um, and it was just trying to move fast on this team that we realized a lot of the frustrations and bottlenecks come from just reproducing issues, and there's no tooling out there. And no user, no PM, no one in sales knows how to communicate to an engineer what the engineer needs to know. And we're like, "We can't believe that there's no way to help the engineer get things done faster." We're spending all this, like, great brain time on communicating bugs, and hopping in calls, and screen sharing, and, and not enough on fixing them, and we thought, "That, that, that we can solve."
- AMAndrew Mayne
It seems, uh, like a lot of things sort of in front of you as far as what you can do. How do you decide what you're gonna do next?
- SPSpeaker
Okay, I think, I think there's something that we t- under talk about in, like, startup world, which is if, if your startup works up- works out to your wildest dreams, you're gonna work on it for, like, 10 years.
- AMAndrew Mayne
Right.
- SPSpeaker
And that's a really long time, and so you better just love the problem because that makes it really, really fun. And so I think you work on the thing where if you get to wake up every day, work on it, and talk to users of the thing, you're gonna be pretty darn happy about that.
- AMAndrew Mayne
That's pretty awesome. Uh, how can people get started at Jam.dev?
- SPSpeaker
Jam.dev/pleasefix, and we'll please fix some code for you.
- AMAndrew Mayne
Awesome. Thank you so much.
- SPSpeaker
Thank you!
- 25:59 – 31:36
Abridge’s core value: ambient documentation that gives doctors time back
- AMAndrew Mayne
Uh, so it's been interesting to see what each kind of developer, what they're looking forward to, what kind of excites them as far as that. A lot of that's based upon what they've been working on. I know a lot of devs have to build a lot of internal tooling, and when you can solve for that with an SDK, it makes your life a lot easier 'cause you can focus on the thing you wanna work with. Up next, we have Zach Lipton, who is with Abridge. How you doing, Zach?
- SPSpeaker
Not too bad. How you doing?
- AMAndrew Mayne
Fantastic. So Abridge, you're working on tools for helping the medical community, people who are doing, like, transcription and that kind of area. Is that... How would you describe it?
- SPSpeaker
It's an AI platform for doctor-patient conversations.
- AMAndrew Mayne
Okay.
- SPSpeaker
Like, if we were to, like, rewind to, like, the state of affairs, you know, uh, post-adoption of electronic health records but pre-ambient listening, um, doctors were spending about two hours doing paperwork for every one hour of direct patient care. So there was this kind of, like, clerical, uh, burden crisis, um, and it was pulling- it was, it was a situation where technology was pulling doctors away from patients rather than bringing them closer.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Um, what we do is we provide this platform that helps- kind of gives doctors superpowers, helps them with their paperwork, does all the note-taking in the background, preps them everything, so all these kind of documentation artifacts are ready for them the moment the visit's over in exactly the form they need, so they can be fully present with the patient instead of spending all their time staring at the computer.
- AMAndrew Mayne
What kind of metrics do you have so far as far as, like, time saved for doctors?
- SPSpeaker
Sure. Interesting story, um, and a l- difficult to track down, in part because there's the time that doctors spend, uh, documenting during the day. But the reality of the status quo before was that most doctors actually didn't finish their notes during the workday. So they were home after hours, logging into the EHR and doing what we called, um, pajama time.
- AMAndrew Mayne
Right.
- SPSpeaker
They're basically sitting there, like, pulled away from dinner with their families, or logged in after hours, you know, sitting in [chuckles] sitting in bed, finishing up their notes. Um, our reports, so we, we, we've cobbled together this information from a bunch of different sources, but we see doctors saving as much as an hour or more a day. Um, we see, you know, doctors- some doctors seeing, like, 10, 15 patients in a day.
- AMAndrew Mayne
Wow!
- SPSpeaker
We're talking about, like, you know, five to 10 minutes often in note-taking, so it's a, it's a tremendous, um, kind of relief. But even beyond the, um, actual time saved, oftentimes there's an even larger sort of, like, perception of burden lift, and that's because the doctor's worrying about less and able to focus on their patient.
- AMAndrew Mayne
Yeah, I mean, that's great 'cause, like, you, you know, look at with... An hour a day is either an hour they can spend with patients or just not have burnout and, you know, have more focus on that. You know, that's, it's- We were talking before SchoolAI and looking at how they're basically able to help teachers spend more time in the classroom with students, and that's the same problem you're dealing with here. How can doctors be there for students or, or excuse me, patients and also-
- SPSpeaker
Yeah.
- AMAndrew Mayne
Yeah.
- SPSpeaker
Well, so we have a channel in Slack, and it's what we call Love Stories, and it's where we hear, like, uh, kind of, like, all the feedback from the field from doctors. And one of the, one of the wild indicators that, like, we had landed on something big, and this is relatively early, um, maybe like a couple months after we launched the first, like, enterprise pilot at a hospital system, was when we started getting stories coming in. And they weren't just talking about the clinical experience, but they were talking about, like: "Uh, I, I spent... Uh, I actually got to have dinner with my family every night this week for the first time in, like, 10 years," or, like, "Abridge is saving my marriage." And that was, that was not, like, where I was expecting-
- AMAndrew Mayne
Wow
- SPSpeaker
... things to land so quickly. But, uh, you know, the, the problem is that big.
- AMAndrew Mayne
What did you see today at DevDay that has you excited?
- SPSpeaker
... Oh, um, so many things, but maybe like two that are top of mind. Um, one, a lot of people have already talked about the agent developer kit.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
And I think that's just extremely exciting. There's this moment right now where I think everyone- it's sort of a proto-discipline.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
So everyone's been rolling their own tools in the hope of, like, kind of trying to figure out, like, what is the paradigm? How does this work? And there's so many things that have to work together. Um, there's the context engineering, there's the prototyping, there's the sanity checking, um, there's evaluation. There's also, down the road, you know, everything around production and monitoring.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
So I'm really excited to see where these tools, um, ultimately go. But seeing OpenAI, like, take a strong position and put a, a kind of comprehensive offering that brings together a lot of these things that, you know, people have been d- rolling their own orchestration tools, their own evaluation platforms, we certainly have. Um, and seeing how much, how much this is gonna create a common platform and allow us to sort of lift off and focus more on the content, um, is something I'm super excited about. Um, and then in general, I've just been extremely excited and drawn a lot of inspiration from all the work that's gone on in terms of software developer tooling.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
And so we are, we are developing AI-powered products for our customers, but we're also big consumers, as, uh, you know, productivity tools like Codex, um, um, play a big role for us, and just seeing how far we've come there. I mean, I remember, um... So my background is academia-
- 31:36 – 44:07
Healthcare-grade reliability: defining hallucinations, building eval pipelines, and earning trust
- AMAndrew Mayne
So when you work in medical, it's a very high-stakes area, and so that's got to be something you, you think a lot about, about how you deal with hallucination and also how you deal with customer concerns about these things.
- SPSpeaker
Hundred percent.
- AMAndrew Mayne
So what do you look forward to in tools? What have you seen the biggest help in that area?
- SPSpeaker
Um, that's an area where we've had to develop a lot of our own technology.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Um, you know, what is a hallucination? It's kind of like, like, back in the old days-
- AMAndrew Mayne
Mm
- SPSpeaker
... of, like, developing simple classifiers, we had, we had false negatives and false positives.
- AMAndrew Mayne
Yeah.
- SPSpeaker
And now, kind of like everything that's like... If, if it's there and you don't want it, it's a hallucination.
- AMAndrew Mayne
Yeah.
- SPSpeaker
And the question is, well, what is a hallucination? Sometimes it's completely confabulated information that is asserting facts about the world that are not real. But in the context of medical note-taking, medical documentation, order placement, um, it's... There, there's, there's a kind of particular situated notion of what we really mean. It's something that's sort of unlicensed, uh, you know, by the, the sort of surrounding context, the substantiating evidence, even if it might be true.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
You know, if, uh, if a doctor doesn't, um... If, if a kind of explanation of a disease, like, shows up in a generated patient-facing summary that the doctor never said-
- AMAndrew Mayne
Yeah
- SPSpeaker
... you know, that's, that's kind of, like-
- AMAndrew Mayne
Bad [chuckles]
- SPSpeaker
... even if some of the information might be, like, either correct or plausible, like, that's not within our- So we have a kind of bespoke notion of what constitutes a hallucination.
- AMAndrew Mayne
Mm.
- SPSpeaker
Um, but we are able to, like, kind of often draw a lot of inspiration from what can, like, the frontier models already do out of the box.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
Like, if we define our ontology of, like, these are the types of, of errors we're concerned with, then we can go and say: What is the ability of an out-of-the-box model, given, like, each, each documentation sort of sentence, à la carte, to correctly designate them as belonging to the right category? And we find, okay, we're already within the realm of, like, the model's able to judge, even if, even if it's not able to sort of never commit the crime in the first place, it's able to recognize when the crime's been committed. That gives us a sort of, like, proof of concept, and now what we need to do is make it better, more accurate, cheaper, faster. So ultimately, create our own special-purpose models that are able to take, in parallel, every single sentence in all the generated documentation and surrounding artifacts-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... and process for each one, like, sort of, does it contain an error of, of, of a unacceptable variety? Like, of, of, of what kind? And then a kind of pipeline downstream for remediating. And we're able to do that, um, with about ninety-seven percent recall at this point.
- AMAndrew Mayne
Cool. So do you have any advice for people who are trying to work on basically developing their own evals for hallucination, or just sort of a good starting point?
- SPSpeaker
I think it all starts with getting really crisp about what you really mean.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
And I think that's what we've seen before, is that what is a hallucination for us is a little bit different from what is a hallucination for, like, a general, um, sort of like open world QA system.
- AMAndrew Mayne
Mm-hmm. So is there kind of like a, uh, boundary which you keep expanding? For instance, we talk about medical, say, "Okay, we feel maybe working in scribing right now is, is an area that we can probably solve for and produce a pretty good product that's at or better than human level." Then do you look at, like, there's areas which you would expand out to as you feel more confident?
- SPSpeaker
Absolutely. So for us, um, we've always bristled a little bit when, you know, like, VCs put up this chart, and they're like, "These guys over here are the, the conversation agents, and these guys here are doing- are the coding startups, and these guys are the scribing companies."
- 44:07 – 46:22
Cursor’s evolution: from autocomplete to autonomous coding agents (and why “vibes” matter)
- AMAndrew Mayne
See you. It's interesting to see where you have companies that are dealing with very high-stakes stuff, like medical or education, and a lot of it is the trust building. It's not a thing where you just pop out with a product and you say, "Hey, we're ready! We're done." You can see from the examples here between Danny and Zach and how they have to basically... and, um, Caleb, of basically just trying to sort of show the customers one step at a time, iterate on the product, and improve it. And here to talk to us about tools for helping work on this is Lee Robinson from Cursor. How are you doing?
- SPSpeaker
I'm doing well. Thanks for having me.
- AMAndrew Mayne
Lee, it is great to have you here. Uh, so Cursor.
- SPSpeaker
Mm-hmm.
- AMAndrew Mayne
Um, I probably have about three Cursor windows open right now on my computer-
- SPSpeaker
[chuckles] Amazing
- AMAndrew Mayne
... that I'm thinking about right now. Uh, it has been an incredible product evolution for you all here. And just a little backstory: I was at OpenAI when we worked on the earliest version of Codex and code completions.
- SPSpeaker
Mm-hmm.
- AMAndrew Mayne
And I kind of naively thought that, "Oh, well, I just ask, you know, GPT-3, the 3.5 at this time, or the Codex model or Da Vinci Code, whatever, and say, 'Just complete it,' and it's done, and I got my code." I'm like, "That's it. Code is solved with AI- [laughing] ... and we're done."
- SPSpeaker
Yep.
- AMAndrew Mayne
Turned out that's not the case.
- SPSpeaker
Yeah, it turns out there's a lot that goes into it, especially from maybe simple text autocomplete, but to where we're at now with fully autonomous coding agents who can self-correct and fix their own errors and pull in information from the outside world, and it's wild how much better the coding with AI space has gotten just in the past year, I would say.
- AMAndrew Mayne
Mm-hmm. And it's a tool very much... What I appreciate is the fact that you guys are using Cursor to make Cursor better.
- SPSpeaker
Definitely. Yeah, one part of our culture that I think helps us produce a better product is not only, of course, dogfooding how we build Cursor by all of our engineering teams, all of our design teams, product management teams, using the product and using Cursor Agent to build the actual experience-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... and for every part of their daily work. But then it allows us to give not only hard evals and, you know, feels on how a model works, but just what it, what it feels like every day to use the product. You know, sometimes the feedback is, "This just didn't feel right. I don't know what it was with this model experience, but we, we need to tweak this a little bit here." And that is actually just as valuable as, you know, a, an eval that shows that maybe we need to tweak this number one little bit. So the vibes are, the vibes are important, too.
- 46:22 – 49:49
Shipping at Cursor: internal adoption as product-market fit + rapid model iteration
- AMAndrew Mayne
How do things work internally as far as adding feature? I imagine when you're a company filled with developers, building a tool for developers, there's a lot of suggestions on what you should be doing, but how do you prioritize?
- SPSpeaker
Yeah, I think across the entire company, anyone has the ability to contribute features-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... and we kind of measure if a feature gets product-market fit internally.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
So people make changes, they push them to main, and they see if it gets adoption. They kind of push it out to company Slack and say, "Hey, we built this feature. Um, here's what it's good for. We'd love for you to try it out." We've even seen people incentivize with bagels and other pastries- [laughing] ... to get people to try out the feature. And they get a lot of feedback internally on if it's good, whether they should change. We also have weekly demo sessions where people show off the cool things that they're working on. And some features don't work out.... and we end up removing them. And others, like, really take off. Like, people have been wanting this for a while. We build it internally. All of a sudden, there's-- we look at metrics, and we see how many internal devs are actually adopting the feature and what the churn rate is, and if it reaches a certain threshold, we have the conversation, "Maybe we should actually release this externally." So then we go through kind of a series of steps where it starts internal, then maybe rolls out to some of our ambassadors, rolls out to some of the people who opt in to our nightly channel, which is, like, you know, releases all the time, and then eventually, there's the path towards getting it out to everybody.
- AMAndrew Mayne
So one of the challenges in AI now is evaluating models, and we can put something into one of the, the evals, one of the benches or whatever, but say, "Oh, it performs this well," but it's entirely different when you put it inside of an IDE, and you have to basically put a harness around it.
- SPSpeaker
Right.
- AMAndrew Mayne
So how much time do you all have to spend really evaluating these things to really figure out if this is gonna be a good fit or not?
- SPSpeaker
Yeah, quite a bit. I mean, one of the things that, you know, we love working with OpenAI on is getting access to integrate the model early, work with your team on-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... tweaking the prompts, getting the harness updated as newer model comes out. I know we spent a lot of time on this with GPT 5 to-- actually, we were able to delete some of our system prompt, where-
- AMAndrew Mayne
Hmm
- SPSpeaker
... as the models get better, you don't have to be as precise with the instructions that you're giving it, and the tools are getting better, too. So we, we spend a lot of time not only dogfooding and just trying it for a bunch of tasks internally, but also having many of the engineers internally all try to build new features on the core product experience with new models as they come out. So then you get a good range of, you know, maybe smaller tasks, but also real tricky engineering problems, like real gnarly bugs in a very large code base.
- AMAndrew Mayne
Yeah, it's interesting in that, um, a couple years ago, people sort of wondered, like, what was gonna happen in the model space. Was everything gonna be using big foundational models, whatever? But you guys have sort of an all of the above.
- SPSpeaker
Mm-hmm.
- AMAndrew Mayne
You're using... You can have-- I can use-- If I want, I can choose a variety of different models in there. I can use Codex, whatever, inside of there, but also, you guys are, like, training your own models for the smaller, for, like, tab completions.
- SPSpeaker
Yeah, it's been an interesting journey to watch. The first versions of Cursor were really focused just on code autocompletion-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... and it was predicting the next line or maybe the next action.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
And now we kind of moved from a world of using an off-the-shelf model to training our own model for autocomplete, to now we do online reinforcement learning, where we can actually roll updates every thirty minutes to the model, which is pretty amazing, just based on the signals we get back from developers, whether they're accepting or rejecting changes that the autocomplete, um, suggests. And that's been really helpful, where you can have both these really focused and intentional models for very specific parts of tasks in coding, and then as you all make better models, these foundational models, we can integrate them and make that experience really great, too.
- 49:49 – 1:01:14
New users, new workflows: agent-first interfaces, context practices, and the future of ‘vibe coding’
- AMAndrew Mayne
I'm seeing more and more of my friends who never even thought about coding start off with writing something maybe in Canvas inside of ChatGPT, and then all of a sudden go, "Oh, how would I deploy this?" And then they go try Cursor and try to do this. And how much have you seen the demographic of the people using your tools change?
- SPSpeaker
Yeah, first and foremost, because we dogfood Cursor to build Cursor, like, we're always building for the professional software engineer-
- AMAndrew Mayne
Mm
- SPSpeaker
... and we're trying to make that experience really great. But as a side effect of making the product easier and more accessible for engineers, it kind of welcomes in this whole new generation of people who maybe were on the fringes of coding or had tried coding in the past, and now it's, it's much more accessible for them to do that. So a lot more product managers, a lot more designers. Uh, our support team uses Cursor quite a bit, and as more of these people have came in and started to use the product, it's allowed us to change actually some of the core features we're building. So one of the things that we're releasing right now is a new view of viewing the Cursor experience, where it's less like a traditional code editor-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... and it's more about working with the agents. Because for a lot of people who haven't coded before, you open up an IDE, and you're like: What am I looking at? Like-
- AMAndrew Mayne
It's a notepad, all this stuff. Yeah, I see.
- SPSpeaker
There's, like, all this file tree on the left-
- AMAndrew Mayne
Yeah
- SPSpeaker
... is very overwhelming for people who are just getting started, versus opening up something that looks closer to a ChatGPT. You're like, "Okay, I know this experience. I've got my agents on the left. I can type into my in-input text box," and so far, we've seen this really resonate with that type of persona, where they're graduating into becoming a developer.
- AMAndrew Mayne
I went from somebody who was first copy-pasting before we even had ChatGPT from the playground or whatever, building some simple little custom tools, to then using the IDE, and like in case of the case, using Cursor and using tab completions, and now really just sitting down and saying, "I just need to write a really good agents.md or a really good description." Have you seen those patterns shift, you know?
- SPSpeaker
Yeah, I think as people get more advanced with working with coding agents, they really start to realize that high-quality context in is gonna give you much higher quality responses from the models, and we can build a lot of this into the core product itself. But some of it require... Just, it depends on your use case of what you're trying to build, the system requirements, and a lot of this stuff gets put into an agents.md file-
- AMAndrew Mayne
Mm-hmm
- SPSpeaker
... or gets put into newer features we have, like planning.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
So you can actually work through a plan mode, go back and forth with the agent to do some research on what you're trying to build, and it can search your code base and kind of help paint a picture, and then when you're handing it off to the model to do coding, it's already got all of this amazing input context, and the quality of the code generated is just significantly better at that point. So we've seen that be very helpful, too.
- AMAndrew Mayne
What would you advise for best practices?
- SPSpeaker
[lips smack] For general just adopting Cursor?
- AMAndrew Mayne
Starting there, yeah.
- SPSpeaker
Yeah. I think that for people who are completely fresh to using Cursor, and we'll start with more of the professional engineer use case, then we can talk about others, but for professional engineers, I think what we see for most people is their on-ramp is they start using the editor just like they would traditional editors or IDEs, where mostly they're using tab, and they're doing their normal coding-
- AMAndrew Mayne
Mm
- SPSpeaker
... and they're still getting AI suggestions as they write along, and they're using agent to somewhat augment that experience, but they're not coding maybe agent first.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
But I think slowly over time, as the models get better and the tools get better, they're graduating then into, "What would it look like if I can hand off more gnarly tasks to the agent to maybe run in the background or go refactor some part of the code base?" So I usually recommend that progression for professional engineers. It's kind of the inverse-
- AMAndrew Mayne
Mm
- SPSpeaker
... for those who are not professional engineers because it's very hard for them to drop into code and then know, "Okay, this is JavaScript. Oh, actually, this is TypeScript, and this is what a type is, and this is what all these fancy words mean." It's much easier for them to take the other approach, where they start with the agent view.
- AMAndrew Mayne
Mm-hmm.
- SPSpeaker
They talk to the agent in natural language, and the agent outputs code, and then they can ask the agent, "What does that mean? What is a const versus a let?" You know, "What do these things mean?" And, you know... Or something in Python, for example. So generally, that's what I recommend.
- AMAndrew Mayne
... where do you think this is gonna be headed? We're, we're five years into GPT-3. You know, the first time I mentioned it earlier was that, you know, seeing somebody make a React button was exciting.
- SPSpeaker
Yeah.
Episode duration: 1:01:14
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode QIdUllqmuls
