Building AI for better healthcare — the OpenAI Podcast Ep. 14

1. 0:00 – 0:38

   Intro

   1. AMAndrew Mayne0:00

      Hello, I'm Andrew Main, and this is the OpenAI Podcast. Today, we're talking to Dr. Nate Gross, Head of Health, and Karan Singhal, who leads Health AI Research at OpenAI. We'll cover what went into training models to handle sensitive questions and how it's helping clinicians, patients, and healthcare systems.

   2. KSKaran Singhal0:17

      We actually worked really closely with a, with a group, uh, a cohort of around two hundred and fifty physicians across every stage of, of generation of this data.

   3. NGDr. Nate Gross0:25

      And we're starting to see medications that have been sitting on a shelf that all of a sudden AI has found ways for them to have direct value in, in, in patient lives.

2. 0:38 – 5:01

   Origins of Nate and Karan’s interest in AI and healthcare

   1. AMAndrew Mayne0:38

      How did you find your way into healthcare?

   2. NGDr. Nate Gross0:40

      So, uh, what, what drew me to healthcare initially was, uh, health policy.

   3. AMAndrew Mayne0:45

      Mm-hmm.

   4. NGDr. Nate Gross0:45

      Um, I was, was very interested. This was before the, the first Obama election. Uh, value-based care was first becoming a thing. Um, I, I started studying, uh, different ways to make healthcare more accessible to more people and then, um, e-eventually went to, uh, Emory-

   5. AMAndrew Mayne1:03

      Mm.

   6. NGDr. Nate Gross1:04

      for medical school, and, and what, what drew me to that was, uh, a large, uh, public hospital, Grady Hospital, you know, to make sure that you're, you're taking advantage of every clinical hour you have.

   7. AMAndrew Mayne1:13

      So what kind of things were you doing?

   8. NGDr. Nate Gross1:15

      So I was mostly pissing off the IT department.

   9. AMAndrew Mayne1:17

      [laughs]

   10. NGDr. Nate Gross1:18

       Um, uh, when, when I was in medical school, uh, the newsfeed came out, the iPhone came out, uh, Twitter came out, uh, the App Store came out, and, and so comparing the technology that we had as doctors, which was fax machine, clipboard, paper binder, the beginnings of electronic health records, to like what my friends had or what the patients had in the waiting room was pretty profound.

   11. AMAndrew Mayne1:40

       So you come at it from the point of view as an AI researcher. Where did your interest in applying this to healthcare come from?

   12. KSKaran Singhal1:47

       So I, I nerded out a lot when I was younger about things like philosophy of mind, and I thought a lot about, you know, intelligence and how far could intelligence go and could machines be intelligent. Um, and, um, a lot of those explorations took me towards, as I was learning about AI and starting to work on my first AI projects, thinking a lot about the ways in which AI could have a lot of impact on humanity in the future. And I thought something like-- I didn't, I didn't predict the future or how fast it would happen, but I thought something like AGI would happen within our lifetimes. So then once, once I had that conviction, I thought a lot about, you know, what are the ways in which I can have either positive impact and, and hopefully make that a really large upside for humanity, or think about the ways in which we could avoid downside. So since then, uh, in my career, I've been thinking a lot about both sides of that coin, thinking about that from the perspective as a safety researcher, which is part of my background, and then really some of that work on safety and privacy that I was working on previously, I started applying it in healthcare, and then I started being like, "Whoa, there is a really massive opportunity to think about the application of this technology, especially large language models in healthcare." And that's what took me to transitioning to it full-time, was just the size of that opportunity and the fact that I felt like the healthcare and clinical AI world was kind of not fully aware of that, of that gap. Um, and so I just thought it was kind of a really amazing opportunity and responsibility to, to bring us there.

   13. AMAndrew Mayne3:05

       I want to understand both the vision and actually how this is going to be implemented.

   14. NGDr. Nate Gross3:10

       So our mission at OpenAI is to ensure that AGI benefits all of humanity, and, and health is one of the places where I think that is not only most achievable, um, but is the clearest. So healthcare today, as everyone knows, is fragmented. Care is missed left and right. Patients are often left three hundred and sixty-four days per year without the opportunity to engage with the organizations that have the information centralized. And doctors, uh, have extremely limited time when they do get that chance to engage with the patient to actually have, uh, you know, a, a meaningful impact beyond a simple surgery or a simple reactive prescription. You know, uh, the, the system is more reactive than it is proactive today, and that leads to, uh, you know, tremendous challenges in the system. It leads to tremendous gaps in care. It leads to, you know, leaving people behind in situations when they could be thriving. And, uh, one of the reasons that I joined, uh, OpenAI is, is because access has always been a, a through line, uh, in, in my life. Access to knowledge, first in medicine, then in building a, a product for, uh, doctors to access the latest medical literature, and then in supporting entrepreneurs as they were building healthcare tools. But OpenAI has the type of technology that can, can do that at scale for the entire ecosystem all at once, help patients, help healthcare professionals, and, uh, help incredible entrepreneurs who are building for all of the corners and edge cases and tough challenges that exist in, in each area of the, the health market.

   15. AMAndrew Mayne4:59

       What is the strategy here?

3. 5:01 – 6:57

   Strategy for building AI tools for clinicians

   1. AMAndrew Mayne5:01

      We know that people use chatbots all the time now for medical questions, but it seems like you're building and working towards something bigger and more comprehensive, not just for the patient side, but the clinician side. Could you talk about, like, what your goals are?

   2. NGDr. Nate Gross5:14

      Patients are increasingly turning to tools like ChatGPT, uh, throughout the year. In fact, um, nine hundred million people now use ChatGPT per week, and if you look at how many are, are doing health-related queries, it's about one in four-

   3. AMAndrew Mayne5:30

      Wow.

   4. NGDr. Nate Gross5:30

      in a given week. So that's forty million people per day. And so our strategy in health is as much proactive as it is reactive in stepping up to the responsibility, uh, and, and the opportunity to do good that, that comes with that strong consumer demand. And so, uh, with ChatGPT Health, we have created a, a space to keep these conversations not just secure-

   5. AMAndrew Mayne5:56

      Mm.

   6. NGDr. Nate Gross5:57

      but empowered. Um, so when I say secure, of course, uh, en-encrypted with this essentially one-way valve protecting your conversations. So these extra security layers, these protections to make sure that we will never train on, on users' healthcare data, combined withEmpowerment, really. Um, you know, search engines that, that people have used to-- before to navigate health have amnesia.

   7. AMAndrew Mayne6:21

      Mm-hmm.

   8. NGDr. Nate Gross6:22

      Um, you know, they're, they're one size fits all, and, uh, I, I think context really matters in healthcare, and so, um, building a, a, a series of, of features and, and technology hooks to help patients bring in their own context that they choose to, so that each time they choose to engage with AI, it's grounded in their own context, uh, is, is a key reason why we've built this ChatGPT for Health foundation.

   9. AMAndrew Mayne6:47

      So I understand the safeguards you put in place to keep the data separate and to make sure that you don't get leak between there and to be able to, you know, undergo, uh, a very rigorous method of making sure that your data's secure.

4. 6:57 – 10:15

   How AI models are trained for health use cases

   1. AMAndrew Mayne6:57

      But when it comes to the model itself, what comes into training models that are capable of working with something like healthcare? It's kinda like the most important thing in the world.

   2. KSKaran Singhal7:06

      For sure. It's, it's a high-stakes domain, and because of the use of-- that, that people are doing, it's super important that we get it right. So we think a lot about a few things when we think about evaluation and training for healthcare, and this is actually the foundation for the work at Health at OpenAI. Um, when we were first starting to work on the health effort at OpenAI, we were thinking a lot about the safety and grounding motivation as an important part of what we were doing. And so part of the thesis actually for starting work on health at OpenAI was thinking, "This is an excellent way to ground our work in safety and alignment and provide kind of concrete incentives and feedback loop for researchers who think about this problem." Um, so the, z-- it's like the model improvements and the safety, safety thinking here is not just an afterthought. It's actually the, the beginning of our work here. And so where we started really was thinking about evaluation.

   3. AMAndrew Mayne7:51

      Mm-hmm.

   4. KSKaran Singhal7:51

      Um, so can we, can we think about the ways in which, you know, models were already starting to become useful to people then, and there's already starting to be this capability overhang between what the models could do and what people were using them for. And so we started to navigate that problem and think about, you know, where do the models still have gaps today? And, and so that's where our work on evaluation comes in. And so we've taken a pretty metho-methodological, um, methodologically interesting, um, approach to that. And a lot of that has reflected in our work in Health Bench, which is this kind of, uh, realistic evaluation of conversations between users, uh, who are either health professionals or consumers talking to models, um, and seeing, uh, measuring the performance and safety of the models in, in these situations, which are these kind of multi-turn conversations. And the way we worked on this is we, we actually worked really closely with a, with a group, uh, a cohort of around two hundred and fifty physicians-

   5. AMAndrew Mayne8:43

      Hmm

   6. KSKaran Singhal8:44

      ... that we work with to kind of, um, across every stage of, of generation of this data from, um, thinking about the ways in which, um, you know, the areas that we would focus in for the evaluation and the areas that we, um, thought about were, were gonna be clinically relevant or impactful to the specific, you know, what are the specific things that are being graded in this evaluation. So that's like a range of things from, you know, are you tailoring your response to a layperson versus a more technical, um, health professional? Are you, um, thinking about the ways in which, um, you should seek context first before providing an initial response? Um, the models used to be, um, significantly... are, are much better today at kind of seeking context when needed, uh, it-- because users are typing in, you know, much less than the models often need-

   7. AMAndrew Mayne9:29

      Mm-hmm

   8. KSKaran Singhal9:29

      ... to be able to provide, um, information that's most helpful to them.

   9. AMAndrew Mayne9:32

      It burns.

   10. KSKaran Singhal9:33

       Exactly. [chuckles] You know, if, if a user types in, it burns. You know, how, how, how do you think about the right way to provide information? You can provide some initial information potentially based on a, a, you know, impression you might have of what the user might be saying, but the most helpful thing to do in that situation and the safest thing to do in that situation is actually to ask for more context. And so that's, that's just one example of, of the many ways that we kind of measured performance in Health Bench, and Health Bench in particular actually measured around forty-nine thousand different dimensions of performance, and that's, that's just an example of one possible dimension of performance. So it's a very multifaceted evaluation that we built kind of in concert with this cohort of two hundred and fifty physicians over a long period of time, and it took us about a year, actually, end to end to, to work on that evaluation and then release it.

5. 10:15 – 14:21

   How OpenAI is able to score well on health evals

   1. AMAndrew Mayne10:15

      In the kind of the, the model development cycle, it seems like sometimes some company gets a bit ahead, and somebody comes up and catches up and whatnot. I've noticed a pattern with the OpenAI health models. They've consistently been far ahead in Health Bench and other evals that like by a big margin. Why is that?

   2. KSKaran Singhal10:33

      I, I think we have a pretty dedicated effort here and a pretty serious effort that is cross-functional and, and kind of, um, across the stack, um, for everything from kind of pre-deployment evals to, um, like, like Health Bench, to monitoring in production traffic and thinking about the ways in which we are ensuring safety in, in production traffic in, in a privacy-preserving way, and working with physicians across every step of that process. And so to my knowledge, um, OpenAI's models are the only major models where every phase of model training, from pre-training to, to mid-training, to post-training, and e-every step in between really integrates health into every major stage. Um, and I think the result is that our, our models are pretty good, not just on our own benchmarks, but also the benchmarks that people-- other people put together.

   3. NGDr. Nate Gross11:19

      I'd like to add a little to, uh, what Karan said about the, the, the model training because I, I think when we spend time with the healthcare ecosystem, that's one of the things that is most important to them. So not only were these models, uh, uh, trained in development with hundreds of physicians who created over five thousand conversations and forty-eight thousand five hundred rubric criteria through which to, uh, evaluate, uh, AI responses and, and score them and identify ways that we could improve the model, uh, do additional data acquisition, uh, do additional post-training, hone in on a particular subspecialty or particular area of the world where users were telling us we could improve, uh, health or healthcare in, in that specific topic. Um, but in addition, uh, I, I think that close proximity to physicians really leads to calling out the, the most important parts that should be focused on in, in model development. So, uh-You know, other places sometimes I see how a model fared on, uh, a medical school exam-

   4. AMAndrew Mayne12:29

      Mm-hmm

   5. NGDr. Nate Gross12:29

      ... or a board exam. And healthcare is, is not multiple choice. You know, patients are, are coming in with a tremendous amount of complexities and their own stories and, and nuance and context. Um, and that's presented in many different ways, and, and part of the job of, of working in healthcare is being able to draw from those disparate sources, uh, draw from experience, balance all that in your head. And so having a training mechanism that thinks about things like when to escalate and how to escalate and keep that always as the, the top priority or adaptive literacy, I mean, com- compare the, the one-size-fits-all handouts that, that people get when they visit the doctor today-

   6. AMAndrew Mayne13:11

      Mm-hmm

   7. NGDr. Nate Gross13:11

      ... to a model that can respond differently when it knows you're an oncologist versus a primary care doctor versus a pharmacist in Kenya versus a, a, a patient at the 12th grade literacy level or the third-grade literacy level, is extremely important for not only making sure that, um, accuracy and, and impact, uh, is, is maximized, but also just to make sure that everyone can maximally participate-

   8. AMAndrew Mayne13:36

      Mm-hmm

   9. NGDr. Nate Gross13:36

      ... in their own care on the patient side. And then finally, uh, uncertainty. You know, if you go back a, a year and a half ago, many of the, the mistakes people would call out about AI models were overconfident hallucinations.

   10. AMAndrew Mayne13:50

       Mm.

   11. NGDr. Nate Gross13:50

       And I think in, in such a high-stakes field like healthcare, one of the most im- important things is that the model can be trained to better know when it doesn't know and say that, and in, in addition, suggest follow-up that can be dug into either, uh, by the patient in a referral to the healthcare system or by the doctor if the doctor is using the model, a test that they might run, additional, uh, pathways they may go down to make sure that, uh, the, the patient can be led to the, the best

6. 14:21 – 21:05

   Key challenges deploying AI in healthcare

   1. NGDr. Nate Gross14:21

      possible outcome.

   2. AMAndrew Mayne14:22

      We've seen the cost of intelligence drop every year, and it's exciting because every year you're able to get better answers, medicine, everything, healthcare across the board. But what are the challenges? What are gonna be the blockers, or what are you looking at ahead to say that, "Okay, we have to solve for this"?

   3. KSKaran Singhal14:39

      The, the drop in cost intelligence has been super exciting here because so much of what we think about and care about here is actually about access. Um, and so the more people have access, um, to technology, the more, the more people will benefit, and that's why we're, um, uh, working on rolling out ChatGPT Health more widely to all free users.

   4. AMAndrew Mayne14:56

      Mm-hmm.

   5. KSKaran Singhal14:57

      Um, and so that, that's incredibly exciting. Another thing that we think about as researchers is like, where will the marginal gains intelligence compound the most, right? And so I, I think Nate mentioned this exciting thing, which is like there is more and more data that is being collected that is across different modalities. How do you think about integrating that data across all the different ways that people use ChatGPT and all the different, um, modalities and wearables and things like this that people are collecting, lab tests, things like this? And that's one place where I think a lot of the intelligence will compound, and we'll start to see kind of new zero or one capabilities, like a model looks at my entire history over a decade and tells me a prediction that even a human couldn't have because-

   6. AMAndrew Mayne15:34

      Mm

   7. KSKaran Singhal15:34

      ... it's just the model has a higher context size. Um, so thinking about those zero to one capabilities I think are, are gonna be really cool. The other thing we keep in mind is just like, how are people thinking about and using ChatGPT today? Um, can we, can we measure that? Can we improve that? Um, and, and I think we're kind of at this interesting point right now. I, I, I call this to our team the transition where, you know, for context I, I bike to work, um, and I bike to work, I wear my helmet, I, I, I worry about cars and things like this next to me. I just reached the point here in SF, you know, S- in SF we have a, a bunch of self-driving cars, i- including Waymos. I just reached the point where, you know, when I'm, when I'm biking next to a Waymo, I actually feel safer than if I was biking next to a human driver, right? I don't worry about whether I'm in their blind spot or, or not or anything like this. So I feel this protective effect by being next to this Waymo, and I want everybody to have this protective effect, right? I, I want everybody to have this protective effect with, with health AI. There are these studies showing that, you know, if you have a doctor in your family, that adds a protective effect to your health as well. Um, and I want everybody, whether they're a patient or a health professional, to think about the ways in which, um, the, the, like y- as a patient, you wanna feel safer, um, having this. As a health professional, you want this to be a safety net for, for the decisions that you're making. So that, that's another frontier that I think we're gonna cross in the next six months or so, which is really exciting, um, this kind of inflection point. Um, another thing that we're thinking about is kind of like the right ways to think around, um, post-deployment monitoring of certain workflows. And, and I think a good example here, um, that I would love to talk about is our AI clinical co-pilot, um, study-

   8. AMAndrew Mayne17:10

      Mm

   9. KSKaran Singhal17:10

      ... that we did, um, with Penda Health. Um, this was a study where we worked with these 20 or so clinics in Nairobi and actually thought about the ways in which we can deploy a safety net for clinicians in that context, which is basically, um, monitoring, uh, things that they type into their electronic health record and only interrupting their flow when, um, th- there's something, um, potentially concerning that's going on or a potential error or things like this. And what we found is that when we deployed this to, to clinicians, um, in the setting, that there was actually a statistically significant reduction in, um, diagnostic and treatment errors-

   10. AMAndrew Mayne17:45

       Hmm

   11. KSKaran Singhal17:45

       ... for the clinicians who are using this tool versus not. And I think this is a step in the direction of moving beyond kind of model evaluations and even, um, monitoring of the ways in which people are thinking about using ChatGPT today to actually, like, thinking about workflows in which th- these technologies can be deployed and, and the right ways to evaluate those workflows after deployment. I think that's another frontier that we are, are really excited about and would love to see more from our partners.

   12. AMAndrew Mayne18:08

       Nate, what do you think the challenges are gonna be?

   13. NGDr. Nate Gross18:11

       I'll start with talking through some of the, the challenges that exist, uh, on the professional side.

   14. AMAndrew Mayne18:16

       Mm-hmm.

   15. NGDr. Nate Gross18:17

       So, uh, each day, uh, when healthcare professionals use AI, they're looking for-The ability to trust what they're seeing in the answer. And so a lot of our recent work has been making sure that answers that the AI is providing are not just grounded in what the model was training on, but is ground-- are grounded in the latest medical literature, the latest guidelines, and sometimes the latest guidance from their own institution or their own region.

   16. AMAndrew Mayne18:49

       Mm.

   17. NGDr. Nate Gross18:50

       Uh, some conditions are treated differently in areas of-- different areas of the country. Uh, other times, uh, different care settings have different levels of, of resources, different levels of, uh, specialists and additional services on hand, and, um, it can be helpful as a, a healthcare professional to be able to quickly navigate that and come up with completely personalized care plans. And so building connectivity within ChatGPT to not only be, uh, HIPAA aligned-

   18. AMAndrew Mayne19:19

       Mm.

   19. NGDr. Nate Gross19:19

       -and be used in these secure environments, but also be able to combine sensitive information with the latest medical knowledge, uh, I think is, uh, a great path that we've started down and, and something that will, you know, continue to, uh, keep trust as the top priority between how healthcare professionals engage with AI. So I think one of the other challenges, um, is that, uh, the systems themselves in healthcare are quite siloed, both at an organization level, but also at the tools that have to be used within each organization. AI thus far has been deployed on a, a really a point solution basis, uh, in the technology industry. But, uh, increasingly, the, uh, connectivity is becoming available to connect the dots between the hundreds of different systems, some analog, some digital, some structured, some unstructured, um, many decentralized, many not on the cloud. Uh, being able to connect all of those through unified AI layers to actually make sure that, um, patients and information isn't, you know, falling through the cracks, and that the connectivity can be maximized to, to actually bring the greatest amount of, of impact. That's hard in healthcare, and it's, it's certainly not something that we can say is solved. But with many of our, our, our recent products ranging from ChatGPT for healthcare and its, uh, connectivity to, to apps and connectors, to the OpenAI API for healthcare, to our frontier foundation for models and agents, uh, w-we think increasingly there's gonna be an opportunity to, um, really accelerate what is possible within the healthcare system and what agents can achieve.

7. 21:05 – 23:05

   Collaboration with hospitals and healthcare systems

   1. AMAndrew Mayne21:05

      Part of this seems like it's very collaborative working with the healthcare industry, and I noticed when using the ChatGPT Health app, the first thing I did was able to put in my records and get all of that, and it looked like there was a lot of just cooperation working across the sort of ecosystem to do this. How has that come to be? Where is this headed?

   2. NGDr. Nate Gross21:25

      It's extremely important that, um, all of the healthcare system has an equal chance to, uh, contribute and engage, um, nationally, uh, and internationally, uh, with providing the context that will help empower, uh, patients to receive the best possible answers from, from ChatGPT. And so, uh, on the electronic health record side, this means working with the government, um, and Centers for Medicare and Medicaid services, uh, adopting national standards for electronic health record syncing so that patients in, in just a, a few taps are able to, um, bring in their context in consented ways. Um, it's being able to tap into, um, existing standards like, like mobile phones and the most popular consumer health products and, uh, the most popular biosensors and wearables to make sure, again, in, in just one or two taps, patients are able to not only bring in that information but, but leverage it in, in thoughtful ways, in ways that may not have been possible without the combined, uh, set of data that, that can exist in this sort of ecosystem. So for instance, um, being able to, uh, reference your recent exercise activity when making a, uh, plan of how to spend your evening, um, or being able to even do things as, uh, as simple as, you know, reference your overnight sleep and, and stress-

   3. AMAndrew Mayne22:56

      Mm.

   4. NGDr. Nate Gross22:56

      -when your agent is helping you set your calendar for the next day and what tasks you may take on first.

   5. AMAndrew Mayne23:02

      It's, it's very exciting. Uh, you know, I have,

8. 23:05 – 26:43

   Practical everyday uses of AI health assistants

   1. AMAndrew Mayne23:05

      you know, wear a smart ring, a watch, whatever, but I get this data, and all I kind of have in my apps are rings to look at and go like, "Okay, I guess it's doing something." Being able to plug into ChatGPT has been fantastic 'cause now I'm able to ask those kinds of questions. But that's very exciting what you talk about too, is if you get a plan from your doctor or suggestions, is literally say, "Hey, uh, I didn't walk enough yesterday. What should I do today?" I've had it be really good at menu planning and literally go on this menu, tell, tell me what to order and whatnot. And so you're saying we're just gonna get more of that and much better.

   2. NGDr. Nate Gross23:34

      Yeah, and that, that's why our partnerships, I believe, are, are so important because i-in these instances, ChatGPT doesn't replace, uh, the incredible technology that our partners are building-

   3. AMAndrew Mayne23:45

      Mm-hmm.

   4. NGDr. Nate Gross23:45

      -to go deep on, uh, health insights for a, a particular wearable. But, but our, our surface area, our opportunity to bring in that health information can now extend to the many different ways people use, uh, ChatGPT, such as what they're gonna cook for dinner-

   5. AMAndrew Mayne24:03

      Mm-hmm.

   6. NGDr. Nate Gross24:03

      -or how they're gonna plan their, their afternoon. Uh, you know, sometimes I think of two patients and, and one patient has to navigate the healthcare system by themself, and the other patient maybe has a spouse come with them.

   7. AMAndrew Mayne24:19

      Mm-hmm.

   8. NGDr. Nate Gross24:19

      And that spouse has a clipboard and used to work as, as a healthcare professional and is very attentive, if not neurotic, and can follow up on-

   9. AMAndrew Mayne24:28

      [chuckles]

   10. NGDr. Nate Gross24:28

       -on details and is connected to your personal calendar. And, and, you know, the best aspects of, of that, uh, with consent for the patients that, that want to, I think represents a, a future where we can make it easier and easier for patients toFollow care plans to play active captain-like roles in, in their own health in, in partnership with their, their care teams and their physicians. And I think if we, we can remove a lot of the, the friction that historically exists between those processes, whether it's, uh, just information not following, or there's a lot to keep track of, or a lot of old information to parse and bring in, uh, we can do a, a tremendous amount of, of good, or, or we can help, uh, patients themselves be empowered to do a tremendous amount of, of good in their own care plans.

   11. AMAndrew Mayne25:18

       And you know, as a physician that it's hard to give as much time as you would like because you're gonna always have more patients you have to deal with than you have hours in the day, and it's interesting to see a kind of a technology that has infinite time, infinite patience to be able to do that as a complement to that.

   12. NGDr. Nate Gross25:33

       I mean, if there's one thing that healthcare professionals are short on, it's time.

   13. AMAndrew Mayne25:37

       Mm-hmm.

   14. NGDr. Nate Gross25:37

       Uh, so when we, um, uh, think about our, our role internally at OpenAI, we o- we often break down, uh, the work that we're doing into three buckets. Uh, raise the floor, so make sure that AI and the benefits of AI are accessible to everyone, um, and that could be patients, that could be healthcare professionals, and, and others working in health-related industries. Sweep the floor, uh, which means help doctors, uh, and help other health professionals save time from the tremendous administration, uh, and bureaucratic burdens that they have every day so that they can spend more time with their patients. Uh, and then thirdly, raise the ceiling. You know, the, the, uh, impact that AI can have in healthcare, I, I think will, uh, you know, allow us to look back on this, this space in a few years and say, "Wow, we, we have all accelerated together in a way that medicine is still in the, in the driver's seat, but, uh, is also far more empowered than ever before."

   15. AMAndrew Mayne26:37

       Yeah. I don't think anybody feels like their doctor spent too much time with them.

   16. NGDr. Nate Gross26:41

       [chuckles]

   17. AMAndrew Mayne26:42

       So it looks like this is gonna be

9. 26:43 – 28:46

   Biggest “wow” moment during development

   1. AMAndrew Mayne26:43

      helpful to solve for that. What has been your favorite aha or wow or this is a really cool moment in the intersection of AI and healthcare?

   2. KSKaran Singhal26:52

      I'll, I'll answer your question in a non-standard way, which is I, I, I think the most amazing thing to see for me in the last year has been the rate of adoption of health, actually even, even beyond the, the ChatGPT Health product, before we, we announced the ChatGPT Health product. Um, it's been one of our fastest-growing use cases is kind of health and wellness questions, and we, we shared that hundreds of millions of people a week are starting to use ChatGPT f- for health and wellness. I think seeing that rapid growth, especially, you know, coming from a background of, of being motivated to work on this problem because it felt like healthcare and clinical AI world were not super aware of the potential of LLMs in healthcare, and seeing how far we had come I, I think has been a really special moment for me.

   3. NGDr. Nate Gross27:29

      There's no doubt that the adoption of this technology and the, and the fact that, uh, it is increasingly collaborative with, with the healthcare system and in- is increasingly driving feedback loops back to us to improve the models is the most meaningful thing and the most mission-aligned thing. But what I also get excited about is, is what our research team is increasingly be able, uh, able to give back to them using that feedback. And not only is it the capabilities of the models, but it's what can be unlocked once those models are allowed to run longer and have more context. And we're starting to see discoveries of medications that have been sitting on a shelf that all of a sudden AI has found ways for them to have meaningful and, uh, direct value in, in, in patient lives. It is starting to scale experiments that, uh, you know, we as individuals wouldn't have been able to juggle on our own. Um, and, and that, that partnership combined with that increased capability to finally move from being interesting to being useful and increasingly to being transformative is, I think, what is the most exciting thing for us heading into this year.

   4. AMAndrew Mayne28:45

      Now that you've been working on this for

10. 28:46 – 30:54

    Feedback from clinicians and early users

    1. AMAndrew Mayne28:46

       some time, you've been engaging with clinicians and talking to people helping deploy this, what has been some of the feedback you've seen?

    2. KSKaran Singhal28:52

       I think, I think the, the experience of flying to Nairobi and seeing the clinicians using the tool and the ways in which, um, we did this thing which we call active change management, where we worked really closely with these clinicians and, uh, flew to Kenya a couple times to think about the ways that we could deepen their workflows using the AI tool, um, and make it something that not only made sense to them, but actually became a kind of something that was, um, indispensable for them. And so as we were concluding the study, um, the team was actually thinking about, uh, the, the team at Penda Health was thinking about, uh, potentially running another study. Um, and they actually had a lot of hesitance around running another study because that would have involved having some group of clinicians using AI and some group of clinicians not using AI. They actually felt that it was dangerous to have a group of clinicians not using the AI. Um, and so that's the point at which I was like, "Wow, we, we have done something major here."

    3. NGDr. Nate Gross29:44

       I think the, the stories that we get back from our members every day are, are one of the most meaningful parts of the job, and these are from caregivers that are, uh, you know, increasingly under strain, taking care of, of family members, trying to navigate their own health at the same time. Uh, this is from, uh, doctors and nurses who are truly overloaded every day, and, um, we can help them extend their expertise, uh, and, you know, compress the, the tough parts of their, their day a little bit more. And then sometimes, and, and, and this is more rare but increasing, it's the miracle cases.

    4. AMAndrew Mayne30:23

       Mm.

    5. NGDr. Nate Gross30:23

       It's the, the patient who had been bouncing around the system for years, the, the unsolved diagnosis, the, uh, the emergency where information wasn't present, and suddenly being able to step in and assist and accelerate and bring people into the care that could really help, uh, is, uh, truly a privilege.

    6. AMAndrew Mayne30:44

       It's exciting. It, it's, it's an amplifier, and every doctor I know wants to be able to do more for their patients. Thank you very much. This has been very interesting, guys.

    7. KSKaran Singhal30:53

       Thank you.

Episode duration: 30:54