Skip to content
David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI | Lex Fridman Podcast #44
This video isn’t embeddableWatch on YouTube →
Lex Fridman PodcastLex Fridman Podcast

David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI | Lex Fridman Podcast #44

Lex Fridman and David Ferrucci on from Watson’s Jeopardy Triumph to Truly Understanding Human-Like Intelligence.

Lex FridmanhostDavid Ferrucciguest
Oct 11, 20192h 24mWatch on YouTube ↗

EVERY SPOKEN WORD

  1. 0:00 – 4:58

    From biology to AI: are brains and computers fundamentally different?

    1. LF

      The following is a conversation with David Ferrucci. He led the team that built Watson, the IBM question-and-answering system that beat the top humans in the world at the game of Jeopardy. From spending a couple of hours with David, I saw a genuine passion, not only for abstract understanding of intelligence, but for engineering it to solve real-world problems under real-world deadlines and resource constraints. Where science meets engineering is where brilliant, simple ingenuity emerges. People who work at joining the two have a lot of wisdom earned through failures and eventual success. David is also the founder, CEO, and chief scientist of Elemental Cognition, a company working to engineer AI systems that understand the world the way people do. This is the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, give it five stars on iTunes, support it on Patreon, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D-M-A-N. And now, here's my conversation with David Ferrucci. Your undergrad was in biology with a- with an eye toward medical school before you went on for the PhD in computer science. So let me ask you an easy question. What is the difference between biological systems and computer systems? In your... when you sit back, look at the stars, and think philosophically.

    2. DF

      I often wonder, I often wonder whether or not there is a- a substantive difference. I mean, I think the thing that got me into computer science and into artificial intelligence was exactly this presupposition that, uh, if we can get machines to think, or I should say this question, this philosophical question, if we can get machines to think, to understand, to process information the way do- we do, so if we can describe a procedure or describe a process, even if that process were the intelligence process itself, then what would be the difference? So, um, from a philosophical standpoint, I'm not sure I'm convinced that there- there- there is. I mean, you can go in the direction of spirituality or you can go in the direction of a soul, but in terms of, you know, what we can- what we can experience, uh, from an intellectual and physical perspective, I'm not sure there is. Clearly, there implement- there- there are different implementations. But if you were to say, as a biological information, processing system fundamentally more capable than one we might be able to build out of silicon or- or some other, uh, substrate, uh, I don't- I don't know that there is.

    3. LF

      How distant do you think is the biological implementation? So fundamentally, they may have the same capabilities, but is it, um, really a far mystery where a huge number of breakthroughs are needed to be able to understand it? Or is it something that, for the most part, in the important aspects, echoes of the same kind of characteristics?

    4. DF

      Yeah, that's interesting. I mean, uh, so, you know, your question presupposes that there's this goal to recreate, you know, what we perceive as biological intelligence. I'm not- I'm not sure that's the- I'm not sure that- that's how I would state the goal. I mean, I think that studying-

    5. LF

      What is the goal?

    6. DF

      Good. So I think there are a few goals. I think that understanding the human brain and how it works is important for us to be able to diagnose and treat issues, for us to understand our own strengths and weaknesses, um, both intellectual, psychological, and physical. So neuroscience and understanding the brain from that perspective has a ne- there's a clear, clear goal there. From the perspective of saying I want to m- I want to- I want to mimic human intelligence, that one's a little bit more interesting. Human intelligence certainly has, um, a lot of things we envy. It's also got a lot of problems too. So I think we're capable of sort of stepping back and saying, "What do we want out of it? Uh, what do we want out of an intelligence? Uh, how do we want to communicate with that intelligence? How do we want it to behave? How do we want it to perform?" Now, of course, it's- it's- it's somewhat of an interesting argument because I'm sitting here as a human with a biological brain and I'm critiquing the strengths and weaknesses of human intelligence and saying that we have the capacity to s- the capacity to step back and say, "Gee, what do- what is intelligence and what do we really want out of it?" And that even- in and of itself suggests that human intelligence is something quite enviable, that it could- it- you know, it can- it can- it can, um, introspect that- it can introspect that way.

  2. 4:58 – 8:25

    What is intelligence, and what’s wrong with human reasoning?

    1. LF

      And the flaws, you mentioned the flaws. That humans have flaws.

    2. DF

      Yeah. But I think- I think that flaws that human intelligence has is ex- extremely, um, prejudicial and biased in the way it draws many inferences.

    3. LF

      Do you think those are... Sorry to interrupt. Do you think those are features or are those bugs? Do you think the- the prejudice, the forgetfulness, the fear... What other flaws? List them all. What? Love? Maybe that's a flaw. Do you think those are all things that can be get- gotten- get in the way of intelligence or the essential components of intelligence?

    4. DF

      Well, again, it's- i- if you go back and you define intelligence as being able to sort of accuracy- accurately, precisely, rigorously reason, develop answers, and justify those answers in an objective way, yeah, then human intelligence has these flaws in that it tends to be more influenced by some of the things you said.

    5. LF

      Mm-hmm.

    6. DF

      Uh, and it's- and it's largely an inductive process, meaning it takes past data, uses that to predict the future.... very advantageous in some cases, but fundamentally biased and prejudicial in other cases, 'cause it's gonna be strongly influenced by its priors, whether they're f- whether they're right or wrong for some, you know, objective reasoning perspective. You're gonna favor them because that's, those are the decisions or those are the paths that succeeded in the past. And I think that mode of intelligence makes a lot of sense for, um, when your primary goal is to act quickly and s- and, and survive and make fast decisions. And I think those create problems, uh, when you wanna think more deeply and make more objective and reasoned decisions. Of course, humans capable of doing both.

    7. LF

      Right.

    8. DF

      They do sort of one more naturally than they do the other, but they're capable of doing both.

    9. LF

      You's saying they do the one that responds quickly and it more naturally?

    10. DF

      Right.

    11. LF

      'Cause that's the thing we kinda need to not be eaten by, uh, the p- the predators-

    12. DF

      Well-

    13. LF

      ... in the world.

    14. DF

      ... for example, but I mean, but, uh, then we, we've, we've learned to reason, uh, through logic. We've developed science. We've trained people to do that. I think that's harder for the individual to do. Uh, I think it requires training and, you know, and, and, and teaching. I think we are... human mind is cer- certainly is capable of it, but we find it more difficult. And then there are other weaknesses, if you will, as you mentioned earlier, just memory capacity and, and, um, how many chains of inference can you actually, um, go through without, like, losing your way, so just focus and...

    15. LF

      S- so the way you think about intelligence, and we're really sort of floating in this philosophical s- slightly space, but I think you're, like, the perfect person to talk about this because, uh, we'll get to Jeopardy! and beyond, th- that's like an incredible, one of the most incredible accomplishments in AI, in the history of AI, but hence, the philosophical discussion. So let me ask, you've kind of alluded to it, but let me ask again, what is intelligence underlying the discussions we'll have with, with Jeopardy! and beyond, how do you think about intelligence? Is it a sufficiently complicated problem, being able to reason your way through solving that problem? Is that kinda how you think about what it means to be intelligent?

  3. 8:25 – 21:52

    Prediction vs understanding: why explanation is a social requirement

    1. DF

      So I, I think of intelligence two, primarily two ways. One is the ability to predict. So in other words, if I have a problem, what's gonna... can I predict what's gonna happen next? Whether it's to, you know, predict the answer of a question or to say, "Look, I'm looking at all the market dynamics and I'm gonna tell you what's gonna happen next," or you're in a, in a room and somebody walks in and you're gonna predict what they're gonna do next or what they're gonna say next.

    2. LF

      So in a, in a highly dynamic environment full of uncertainty, be able to-

    3. DF

      Lots of, lot-

    4. LF

      ... predict.

    5. DF

      ... you know, the more-

    6. LF

      Yeah.

    7. DF

      ... the more variables, the more complex, the more possibilities, the more complex. But can I take a small amount of prior data and learn the pattern and then predict what's gonna happen next accurately and consistently? That's a f- that's certainly a form of intelligence.

    8. LF

      W- what do you need for that, by the way? You need to have an understanding of the way the world works in order to be able to unroll it into the future, right? Like, w- what do you think-

    9. DF

      Well-

    10. LF

      ... is needed to predict?

    11. DF

      ... depends what you mean by understanding. I, I, I, I, I need to be able to find that function. This is very much like what-

    12. LF

      It's a function.

    13. DF

      ... deep learning does, machine learning does, is if you give me enough prior data and you tell me what the output variable is that matters, I'm gonna sit there and be able to predict it. And if I can predicu- predict it accurately so that I can get it right more often than not, I'm smart. If I can do that with less data (laughs) and less training time, I'm even smarter. If I can figure out what's even worth predicting, (laughs) um, I'm smarter, meaning I'm at... I'm, I'm figuring out what path is gonna get me toward a goal.

    14. LF

      What about picking a goal? Sorry to interrupt again.

    15. DF

      That's... Well, that's the interesting about picking a goal, sort of an interesting thing, and I think that's where you bring in what are you pre-programmed to do? We talk about humans and, well, humans are pre-programmed to survive. So sort of their primary, you know, driving goal, what do they have to do to do that? And that, that could be very complex, right? So it's not just, it's not just figuring out that you need to run away from the ferocious tiger, but we survive in s- a social context, as an example. So understanding the subtleties of social dynamics becomes something that's important for surviving, finding a mate, reproducing, right? So we're continually challenged with complex sets of variables, complex constraints, rules if you will, that we, we... or patterns, and we learn how to find the functions and predict the things, in other words, represent those patterns efficiently, and be able to predict what's gonna happen, and that's a form of intelligence. That doesn't really recoi- that doesn't really require anything specific other than the (laughs) ability to find that function and, and predict that right answer. It's certainly a form of intelligence. But then when we, we, when we say, "Well, do we understand each other?" In other words, um, do... would you perceive me as, as intelligent beyond that ability to predict? So now I can predict, but I can't really articulate h- how I'm going through that process, what my underlying theory is for predicting, and I can't get you to understand what I'm doing so that you can follow... you can figure out how to do this yourself if you hadn't r- if you did not have, for example, the right pattern-matching machinery that I did. And now we ha- potentially have this breakdown where, in effect, I'm intelligent, but I'm sort of an alien intelligence relative to, to you.

    16. LF

      (laughs) You're intelligent, but nobody knows about it.... uh, or the-

    17. DF

      Well, I can see-

    18. LF

      ... or I can't-

    19. DF

      ... I can see the, I can see the output, like-

    20. LF

      So, so you're saying, let's sort of separate the two things. One is you explaining why you were able to predict the future and, and, uh, and the second is me being able to, like, impressing me that you're intelligent, me being able to know that you successfully predicted the future. Do you think that's...

    21. DF

      Well, it's not impressing you that I'm intelligent. In other words, you may be convinced that I'm intelligent in some form.

    22. LF

      So how, what would convince-

    23. DF

      Because of my ability to predict.

    24. LF

      So I would look at the metrics.

    25. DF

      When you can, I just say, "Wow."

    26. LF

      "Wow."

    27. DF

      "You're right all, you're, you're, you're right more times than I am. You're doing something interesting." That's a form of, that's a form of intelligence. But then what happens is, if I say, "How are you doing that?" and you can't communicate with me, and you can't describe that to me, now I may lab- label you a savant. I may, I may say, "Well, you're doing something weird and it's, and it's just not very interesting to me, because you and I can't really communicate." And, and so now the, the, so this is interesting, right? Because now this is, you're in this weird place where for you to be recognized as intelligent the way I'm intelligent-

    28. LF

      Right.

    29. DF

      ... then you and I sort of have to be able to communicate. And then my, we start to understand each other, and then my respect and my, my appreciation, my ability to relate to you starts to change. So now you're not an alien intelligence anymore. You're, you're a human intelligence now, because you c- you and I can communicate. And so I think when we look at, when we look at a- when we look at animals, for example, animals can do things we can't quite comprehend. We don't quite know how they do them, but they can't really communicate with us. They can't put what they're going through in our terms, and so we think of them as sort of, "Well, they're these alien intelligences and they're not really worth necessarily what we're worth." We don't treat them the same way as a result of that. But it's, it's hard because who knows what, what, you know, what's going on.

    30. LF

      So just a, a quick elaboration on that. The, explaining that you're intelligent, the, explaining the, the reasoning that went into the prediction is not some kind of mathematical proof. Uh, if we look at humans, look at political debates and discourse on Twitter, uh, it's mostly just telling stories. So you, your-

  4. 21:52 – 31:50

    Persuasion, recommender systems, and the meaning problem

    1. LF

      How would you begin to try to solve that? And maybe just a quick pause, because there's an optimistic notion in the things you're describing, which is r- uh, being able to explain something through reason. But if you look at algorithms that recommend things that we'll look at next, whether it's Facebook, Google, advertisement-based companies, you know, their goal is to convince you to buy things based on anything. Uh, so that could be reason 'cause the best of advertisement is showing you things that you really do need and explain why you need it. But it could also be through emotional manipulation. The algorithm that describes why a certain reason, a certain decision was, was made, how hard is it to do it w- through emotional manipulation? And why is that a good or a bad thing? So you've kind of focused on reason, logic-

    2. DF

      Mm-hmm.

    3. LF

      ... really showing in a, in a clear way why something is good. One, is that even a thing that us humans do? (laughs) And, uh, and, and two, how do you think is the difference between, um, the reasoning aspect and the emotional manipulation?

    4. DF

      Well, you know, so you call it emotional manipulation but more, more objectively is essentially saying, you know, thing, you know, there are certain features of things that seem to attract your attention. I mean, they kinda give you more of that stuff.

    5. LF

      Manipulation is a bad word.

    6. DF

      Yeah. I mean, I'm not-

    7. LF

      There might be more.

    8. DF

      ... saying it's good, right or wrong, I mean, is, uh, uh, it works to get your attention and it works to get you to buy stuff, and when you think about algorithms that look at the patterns of the, you know, patterns of features that you seem to be spending your money on and say, "I'm gonna give you something with a similar pattern," so I'm gonna learn that function because the objective is to get you to click on it or get you to buy it or whatever it is. I don't know, I mean, that, it is, like, it is what it is. I mean, that's what the algorithm does. You can argue whether it's good or bad. It depends what your, you know, what your, what your goal is.

    9. LF

      I guess this seems to be-

    10. DF

      It is g-

    11. LF

      ... very useful for convincing or telling a story.

    12. DF

      It, for convincing, for convincing humans-

    13. LF

      Yeah.

    14. DF

      ... it's good because you, because again, this goes back to how does a hu- you know, what, what is the human behavior like? How does, what does, how, w- how does the human, you know, brain respond to things? Um, I think there's a more optimistic view of that too, which is...... that if you're searching for a certain kinds of things, you've already reasoned that you need them and these, these algorithms are saying, "Look, that's up to you. The reason whether you need something or not, that's your job." You know, you, you may, you may have an unhealthy addiction to this stuff, or you may have a reasoned and thoughtful explanation for why it's important to you. And the algorithms are saying, "Hey, that's, like, whatever. Like, that's your problem. All I know is you're buying stuff like that, you're interested in stuff like that. Could be a bad reason, could be a good reason. That's up to you. I'm gonna show you more of that stuff." And so... And I, and I, and I think that that's... It's not good or bad. It, it's not reasoned or not reasoned. The algorithm is doing what it does, which is saying, "You seem to be interested in this. I'm gonna show you more of that stuff." And I think we're seeing this not just in buying stuff, but even in social media. You're reading this kind of stuff. I'm not judging on whether it's good or bad. I'm not reasoning at all. I'm just saying, "I'm gonna show you other stuff with similar features." And, you know, and, like, and that's it. And I wash my hands from it and I say, "That's all," you know, "that's all that's going on."

    15. LF

      You know, there is... Uh, you know, people are so harsh on AI systems. So one, the bar of performance is extremely high, and yet we also ask them to, uh, in the case of social media, to, uh, help find the better angels of our nature and help make a better society. What do you think about the role of AI there?

    16. DF

      So that's, so that's... I agree with you. That's, that's the interesting dichotomy, right? Because on one hand, we're sitting there and we're sort of doing the easy part, which is finding the patterns. We're not building a... The system's not building a, a theory-

    17. LF

      Right.

    18. DF

      ... that is consumable and understandable by other humans that can be explained and justified. And, and so on one hand to say, "Oh, you know, AI is doing this. Why isn't it doing this other thing?" Well, this other thing is a lot harder. And it's interesting to think about why, why, why it's harder. And bec- because you're interpreting, you're interpreting the data in the context of prior models. In other words, understandings of what's important in the world, what's not important. What are all the other abstract features that drive, um, our decision-making? What's sensible? What's not sensible? What's good? What's bad? What's moral? What's valuable? What isn't? Where is that stuff? No one's applying the interpretation. So when I, when I see you clicking on a bunch of stuff and I look at these simple features, the raw features, the features that are there in the data, like what words are being used or how long the material is, um, or other very superficial features. What colors are being used in the material? Like, I don't know why you're clicking on the stuff you're clicking. Or if it's products, what the pri- what the price is or what... the categories and stuff like that. And I just feed you more of the same stuff. That's very different than kind of getting in there and saying, "What does this mean?" What, what... The stuff you're reading, like, why are you reading it? What assumptions are you bringing to the table? Are those assumptions sensible? Uh, is the mat- does the material make any sense? Does it, does it lead you to thoughtful, good conclusions? Again, there's judgment, there's interpretation and judgment involved in that process that isn't really happening in, in, in the AI today. That's harder-

    19. LF

      Right.

    20. DF

      ... because you have to start getting at the meaning of this... of the, of the stuff, of the content. You have to get at how humans interpret the content-

    21. LF

      Right.

    22. DF

      ... relative to their value system and deeper thought processes.

    23. LF

      So that's what meaning means, is not just some kind of deep, timeless, semantic thing that the statement represents, but also how a large number of people are likely to interpret. So it's, again, even meaning is a social construct. So you have to try to predict how most people would understand this kind of statement.

    24. DF

      Yeah. Meaning is often rel- relative, but, uh, m- meaning implies that the connections go beneath the surface of the artifacts. If I show you a painting, it's a bunch of colors on a canvas. What does it mean to you?

    25. LF

      Yeah.

    26. DF

      And it may mean different things to different people because of their different experiences. It may mean something even different to the artist who, who painted it. As we try to get more rigorous with our communication, we try to really nail down that meaning. So we go from abstract art to precise mathematics, precise engineering drawings, and things like that. We're really trying to say, "I want to narrow that, that space of possible interpretations," because the precision of the communication ends up becoming more and more important. And so that means that I have to specify, and I think that's why this becomes really hard. Because if I'm just showing you an artifact and you're looking at it superficially, whether it's a bunch of words on a page or whether it's, um, you know, brushstrokes on a canvas or pixels in a photograph, you can sit there and you can interpret lots of different ways at many, many different levels. Um, but when I wanna, when I want to, uh, align our understanding of that, I have to specify a lot more stuff that's actually not in, uh, not directly in the artifact. Now I have to say, "Well, how do you wer- how are you interpreting this image and that image? And what about the colors and what do they mean to you? What's, what perspective are you bringing to the table? What are your prior experiences with those artifacts? What are your fundamental assumptions and values? What, what is your ability to kind of reason, to chain together, um, logical implications as you're sitting there and saying, 'Well, if this is the case, then I would conclude this. And if that's the case, then I would conclude that'?" And, uh... So, your reasoning processes and how they work, your prior models and, uh, what they are, your values and your assumptions. All those things now come together into the interpretation. Getting in sync on that is, is hard.

    27. LF

      And yet, humans are able to intuit some of that without any pre-

    28. DF

      Because they have the shared experience.

    29. LF

      ... mea- and we're not talking about shared, two people having a shared experience.

    30. DF

      No.

  5. 31:50 – 41:02

    Frameworks and shared priors: the missing ingredient for human-level communication

    1. LF

      Do you think that shared knowledge... If, if, if we can maybe escape the hardware question, how much is encoded in the hardware? Just the shared knowledge and the software, the, the history, the many centuries of wars and so on that, that came to today. That shared knowledge. Uh, how hard is it to encode? And did you have a hope? Can you speak to how hard is it to encode that knowledge systematically in a way that could be used by a computer?

    2. DF

      So I think it is possible to learn to, for a machine, to program a machine to acquire that knowledge with a similar foundation. In other words, an inter- a similar interpretative, interpretative foundation for processing that knowledge.

    3. LF

      Uh, what do you mean by that? How-

    4. DF

      So in other, in other words-

    5. LF

      ... foundation?

    6. DF

      ... we view the world in a particular way. And so, in other words, we, we have a, if you will, as humans, we have a framework for interpreting the world around us.

    7. LF

      Mm-hmm.

    8. DF

      So we have multiple frameworks for interpreting the world around us. But, uh, if you're interpreting, for example, socio-political interactions, you're thinking about, well, there's people, there's collections and groups of people. They have goals. Goals largely built around survival and quality of life.

    9. LF

      Mm-hmm.

    10. DF

      There are e- there are fundamental economics around scarcity of resources. And when, when humans come and start interpreting a situation like that, because you brought, you brought up, like, historical events. They start interpreting situations like that. They apply a lot of this, a lot of this, this fundamental framework for interpreting that. Well, who are the people? What were their goals? What resources did they have? How much power or influence did they have over the other... Like, just fundamental-

    11. LF

      Yeah.

    12. DF

      ... substrate, if you will, for interpreting and reasoning about that. So I think it is possible to imbue a computer with that, that stuff that humans, like, take for granted when they go and, and, and sit down and try to interpret things. And then, and then with that, with that foundation, they acquire, they start acquiring the details, the specifics in any given situation, are then able to interpret it with regard to that framework. And then given that interpretation, they can do what? They can predict. But not only can they predict. They can predict now with an explanation that can be given in those terms, in the terms of that underlying framework that most humans share.

    13. LF

      Mm-hmm.

    14. DF

      Now, you could find humans that come and interpret events very differently than other humans, because they're, like, using a, a different s- different framework. You know, the movie Matrix comes to mind, where, you know, they decided that humans were really just batteries, and that's how they (laughs) interpreted the value of humans-

    15. LF

      Mm-hmm.

    16. DF

      ... um, as a source of electrical energy. So but, um, but I think that, you know, for the most part, we, we, we have a way of, of interpreting the events or at least social events around us, because we have this shared framework. It comes from, again, the fact that we're, we're similar beings that have similar goals, similar emotions, and we as... We can make sense out of these. These frameworks make sense to us.

    17. LF

      So how much knowledge is there, do you think? So it's... You said it's possible.

    18. DF

      Well, there's always a tremendous amount of detailed knowledge in the world. There are, you know. You can imagine, you know, effectively infinite number of unique situations and unique, unique configurations of these things. But the, the knowledge that you need, w- what I refer to as, like, the frameworks, for... You need for interpreting them, I don't think. I think that's, those are finite. Um-

    19. LF

      You think the frameworks are more important than the bulk of the knowl- so, like, framing

    20. DF

      Yeah. ... describes the- Because what the frameworks do is they give you now the ability to interpret and reason, and to interpret and reason and to interpret and reason over the specifics in ways that other humans would understand.

    21. LF

      What about the specifics? You know-

    22. DF

      Well, you acquire the specifics by reading and by talking to other people.

    23. LF

      So I'm mostly actually just even... If we can focus on even the beginning, the common sense stuff, the stuff that doesn't even require reading or it almost s- requires playing around with the world or something. Just being able to sort of manipulate objects, drink w- water and so on.

    24. DF

      Right.

    25. LF

      All of that. Every time we try to do that kind of thing in robotics or AI, it seems to be like an onion. (laughs) You seem to realize how much knowledge is really required to perform even some of these basic tasks. Do you have that sense as well? And if so, how do we get all those details? Are they written down somewhere? Do they have to be learned through experience?

    26. DF

      So I think when, like if you're talking about sort of the physics, the basic physics around us, for example, acquiring information about... Acquiring how that works, um, yeah, man, I think that, I think there's a combination of things going... I think there's a combination of things going on. I think there is like fundamental pattern matching, like what we were talking about before, where you see enough examples, enough data about something, you just start assuming that. And with similar input, I'm gonna predict similar outputs. You don't, can't necessarily explain it at all. Um, you may learn very quickly that when you let something go, it falls to the ground.

    27. LF

      That's a, that's a, such a-

    28. DF

      But you can't necessarily explain that.

    29. LF

      But that's such a deep idea, that if you let something go, like the idea of gravity.

    30. DF

      Is it? I mean-

  6. 41:02 – 52:02

    Hybrid AI architectures: combining learning with knowledge representations

    1. LF

      Do you think... In terms of encoding architectures like that, do you think systems that are able to do this will look like neural networks or representing... If you look back to the '80s and '90s with the expert systems, so more like graphs, uh, um, systems that are based in logic, able to contain a large amount of knowledge, where the challenge was the automated acquisition of that knowledge. So I guess the question is when you collect both the frameworks and the knowledge from the data, what do you think that thing will look like?

    2. DF

      Yeah. So I mean, I think the thing... Asking the question they look like neural networks is a bit of a red herring. I mean, I think that they, they will, they will certainly do inductive or pattern match based reasoning and I've already experimented with architectures that combine both, that use machine learning and neural networks to learn certain classes of knowledge. In other words, to find repeated patterns in order, or in order for it to make good inductive guesses, but then ultimately to try to take those learnings and, and marry them. In other words, um, connect them to frameworks so that it can then reason over that, in terms other humans understand. So for example, at Elemental Cognition we do both. We have architectures that, that do both. But... Both those things, but also have a learning method for acquiring the frameworks themselves and saying, "Look, ultimately I need to take this data. I need to interpret it in the form of these frameworks so they can reason over it." So there is a fundamental knowledge representation like what you were saying, like these graphs of logic, if you will. There are also neural networks that acquire certain class of information. Uh, they then, they then align them with these frameworks-... but there's also a mechanism to acquire the frameworks themselves.

    3. LF

      Yeah, so it seems like the idea of frameworks requires some kind of collaboration with humans.

    4. DF

      Absolutely.

    5. LF

      So, do you think of that collaboration as-

    6. DF

      Well, and let's to be clear, let's be clear. Only for the, for the, the expressed purpose that you're designing, you're, you, you're designing machi... You're designing an intelligence that can ultimately communicate with humans-

    7. LF

      Mm.

    8. DF

      ... in ter- in the terms of frameworks that help them understand things, right? So, so now to be really clear, you can create... You can independently create an a, a, a machine learning system and an intel... an intelligence that I might call an alien intelligence that does a better job than you with some things, but can't explain the framework to you. That doesn't mean it isn't... it might be better than you at the thing. It might be that you cannot comprehend the framework that it may have created for itself that is inexplicable to you. That's a reality.

    9. LF

      But you're, you're more interested in, in a case where you can.

    10. DF

      I, I am. I am, yeah. Uh, I p- My, my sort of approach to AI is because I've set the goal for myself, I want machines to be able to ultimately communicate, um, understanding with human. I want them to be able to acquire and communicate, acquire knowledge from humans and communicate knowledge to humans. They should be using what, you know, inductive, uh, machine learning techniques are good at, which is to observe patterns of data, whether it be in language or whether it be in images or mo- videos or whatever, to acquire these patterns, to induce the generalizations from those patterns, but then ultimately work with humans to connect them to frameworks, interpretations if you will, that ultimately make sense to humans. Of course, the machine is gonna have the strength that it has, the richer and longer memory, but that, you know, it has the more rigorous re- reasoning abilities, the deeper reasoning abilities. So it'd be an interesting, you know, complementary relationship between the human and the machine.

    11. LF

      Do you think that ultimately needs explainability like a machine? So if you look, uh, you study, for example, Tesla autopilot a lot, or humans, I don't know if you've driven the vehicle or, are, are, are aware of what-

    12. DF

      Yeah.

    13. LF

      ... the, the... So you basically, the human and machine are working together there, and the human is responsible for their own life to monitor the system. And, you know, the system fails every few miles. And so the- there's, there's hundreds of... there's millions of those failures a day, and so that's like a moment of interaction. Do you see?

    14. DF

      Yeah. That- that- that- no, that's exactly right. That's a moment of interaction, um, where, you know, the, the, the machine has learned some stuff, uh, it l- has a failure. Somehow the failure is communicated. The human is now filling in the, the mistake, if you will, or maybe correcting it or doing something that is more successful. In that case, the computer takes that learning. So I believe that the collaboration between human and machine, I mean, that's sort of a primitive example and sort of a more, um... Another example is where the machine is literally talking to you and saying, "Look, I'm, I'm reading this thing. I know, I know that, like, the next word might be this or that, but I don't really understand why. I have my guess. Can you help me understand the framework that supports this?" And then can kind of t- acquire that, take that, and reason about it and reuse it the next time it's reading to try to understand something. Not un- not unlike a human, uh, student might do. I mean, I remember like, uh, when my, my daughter was in first grade and she was... had a, um, re- reading assignment about electricity and, you know, somewhere in, in, in the text it says, "And electricity is produced by water flowing over turbines," or something like that. And then there's a question that says, "Well, how is electricity created?" And so my daughter comes to me and says, "I mean, I could..." You know, created and produced are kind of synonyms in this case. "So I can go back to the text and I can copy by water flowing over turbines, but I have no idea what that means."

    15. LF

      Mm-hmm.

    16. DF

      "Like, I don't know how to interpret water flowing over turbines and what electricity even is. I mean, I can get the answer right by matching the text, but I don't have any framework for understanding what this means at all."

    17. LF

      And framework really is... I mean, it's a set of, not to be mathematical, but axioms of ideas that you bring to the table in interpreting stuff and then you build those up somehow.

    18. DF

      You, you, you build them up with the expectation that there's a shared understanding of what they are.

    19. LF

      Share... Yeah, yeah. It's the social-

    20. DF

      Yeah, right.

    21. LF

      ... that, that us humans... Do you have a sense that humans on Earth, in general, share a set of f... Like, how many frameworks are there?

    22. DF

      I mean, it depends on how you bound them, right? So, in other words, how big or small, like their, their individual scope. Um, but there's lots and there are new ones. I think they're... I, I think the way I think about is kind of in a layer. I think of the architecture as being layered in that there's, there's a small set of primitives that allow you the foundation to build frameworks and then there may be, you know, many frameworks, but you have the ability to acquire them, and then you have the ability to reuse them. I mean, one of the most compelling ways of thinking about this is little reasoning by analogy where I can say, "Oh, wow, I've learned something very similar." Um, you know, I never heard of this, I never heard of this game, uh, soccer, but, um, if it's like basketball in the sense that the goal is like the hoop and I have to get the ball in the hoop and I have guards and I have this and I have that, like where, where does the... where, where are the similarities and where are the differences? And I have a foundation now for interpreting this new information.

    23. LF

      And then, uh, different groups, like the millennials will have a framework and then, and then, and then-

    24. DF

      Well, that, you know that-

    25. LF

      ... and then ever-

    26. DF

      Yeah.

    27. LF

      You know?

    28. DF

      Well, like that, that-

    29. LF

      The Democrats and Republicans.

    30. DF

      Well-

  7. 52:02 – 57:40

    Why Jeopardy! is hard: witty clues, buzzer timing, and confidence under pressure

    1. LF

      So, one of the greatest accomplishments in the history of AI is, um, Watson competing against Je- uh, in- in a game of Jeopardy against humans, and you were a lead in that, a crit- a critical part of that. So, let's start at the very basics. What is the game of Jeopardy? The game for us humans, human versus human.

    2. DF

      Right. So it's to take a question and answer it. (laughs)

    3. LF

      (laughs)

    4. DF

      Um, the game of Jeopardy! Well-

    5. LF

      It's just the opposite.

    6. DF

      ... actually, it's- actually-

    7. LF

      (laughs) It's the opposite.

    8. DF

      Well, well, no, but it's not, right?

    9. LF

      Right.

    10. DF

      (laughs) It's like- it's really not. It's really-

    11. LF

      Yeah.

    12. DF

      ... it's really to get a question and answer, but it's- it's what we call a factoid question. So this notion of like it's- it really relates to some fact that every- few people would argue whether the facts are true or not. In fact, most people would. And Jeopardy! kind of counts on the idea that these- these statements have factual answers. And, um, and the idea is to, first of all, determine whether or not you know the answer, which is sort of an interesting twist.

    13. LF

      So, first of all, understand the question, right?

    14. DF

      You have to understand the question. What is it asking? And that's a good point because the questions are not asked directly, right? They're-

    15. LF

      They're all like... The way the questions are asked is non-linear. It's like, uh, it's a little bit witty. It's a little bit playful sometimes. It's, uh, it's a little bit tricky.

    16. DF

      Yeah, they're asked in- in exactly numerous witty, tricky ways.

    17. LF

      Yeah.

    18. DF

      Uh, exactly what they're asking is not obvious. It takes- it takes inexperienced humans a while to go, "What is it even asking?"

    19. LF

      Right.

    20. DF

      And it's sort of an interesting realization that you have when somebody says, "Oh, what's the... Jeopardy! is a question answering show," and then he's like, "Oh, like I know a lot," and then you read it and you're- you're still trying to process the question, and the champions have answered and moved on. They're three like-

    21. LF

      (laughs) Yeah.

    22. DF

      ... they're three questions ahead by the (laughs) by the time you figured out what the question even meant. So, there's- there's definitely an ability there to just parse out what the question even is.

    23. LF

      Yeah.

    24. DF

      So, that was certainly challenging. It's interesting historically though, if you look back at the Jeopardy games much earlier, you know-

    25. LF

      Like 60s, 70s, that kind of thing?

    26. DF

      ... early games, the questions were much more direct. They weren't quite like that. They got sort of more and more interesting. The way they asked them that sort of got more and more interesting and subtle and nuanced and humorous and witty over time which really required the human to kind of make the right connections in figuring out what the question was even asking. So yeah, you have to figure out what the question's even asking, then you have to determine whether or not you think you know the answer, and because you have to buzz in really quickly, you still have to make that determination, uh, as quickly as you possibly can. In other words, y- you lose the opportunity to buzz in. You may-

    27. LF

      Even before you really know if you know the answer.

    28. DF

      I think a lo- I think a lot of humans will- will assume. They'll- they'll- they'll look at- they'll look at it- they're processed very superficially. In other words, what's the topic? What are some keywords? And just say, "Do I know this area or not?" before they actually know the answer.... then they'll buzz in and, then they'll buzz in and think about it. So it's interesting what humans do. Now, some people who know all things, like Ken Jennings or something, or the more recent big Jeopardy! player, um, they, I mean they'll just buzz in. They'll just assume they know all of Jeopardy! and they'll just buzz in.

    29. LF

      Hmm.

    30. DF

      You know, Watson, interestingly, didn't even come close to knowing all of Jeopardy!, right? Wa- Watson really-

  8. 57:40 – 1:07:17

    The origin story: why IBM took the Jeopardy! moonshot

    1. LF

      So you stepped in. So there's this, there's these three humans playing a game, and you stepped in with the idea that IBM Watson would be one of, replace one of the humans and compete against, uh, two. Can you tell the story of Watson taking on this game?

    2. DF

      Sure.

    3. LF

      This seems exceptionally difficult.

    4. DF

      Yeah. So the story, uh, was that, um, it was a- it was coming up, I think, the 10-year anniversary of, uh, of Big Blue. Uh, not Big Blue.

    5. LF

      Deep Blue.

    6. DF

      Deep Blue. IBM wanted to do sort of another kind of really, you know, fun challenge, public challenge, that can bring attention to IBM Research and the kind of the cool stuff that we were doing. I had been working in, in AI at IBM for some time. Uh, I had a team doing, uh, what's called open domain factoid question answering, which is, you know, we're not gonna tell you what the questions are. We're not even gonna tell you what they're about. Uh, can you go off and, and get accurate answers to these questions? And, um, it was an area of AI research that I was involved in. And so it was a big pa- it was a very specific passion of mine. Language understanding had always, always been a passion of mine. One sort of narrow slice on whether or not you could do anything with language was this notion of open domain, meaning I could ask anything about anything. Factoid, meaning it essentially had an answer.

    7. LF

      Mm-hmm.

    8. DF

      And, and, you know, being able to do that accurately and quickly. So that was a research area that my team had already been in. And so completely independently, several, you know, IBM executives were like, "What are, what are we going to do? What's the next cool thing to do?" And Ken Jennings was on his winning, uh, streak. This was like, whatever it was, 2004, I think, was on his win- winning streak. And someone thought, "Hey, that would be really cool if, um-"

    9. LF

      (laughs)

    10. DF

      "... if the computer can play Jeopardy!" And so this was-

    11. LF

      Yeah.

    12. DF

      ... like in 2004, they were shopping this thing around. And everyone was telling the ex- the, the research execs, um, "No way. Like, this is crazy." And we have some pretty, you know, senior people in the field, and they're saying, "Yeah, this is crazy." And it'll come across my desk, and I was like, "But that's kind of what, what I'm really interested in doing." And, um, but there was such this prevailing sense of, "This is nuts. We're not going to risk IBM's reputation on this. We're just not doing it." And this happened in 2004. It happened in 2005. At the end of 2000, um, 6, um, it was coming around again. And I was coming off of a, um... I was doing the, the open domain question answering stuff, but I was coming off a co- couple other projects. I had a lot more time to put into this. And I argued that it could be done, and I argued it would be crazy not to do this.

    13. LF

      Can I, um... You could be honest at this point. So even though you argued for it, uh, what's the confidence that you had yourself, uh, privately, uh, that this could be done? What was... (laughs) We just told, told the story, how you tell stories to convince others.

    14. DF

      Mm-hmm.

    15. LF

      How confident were you? What was your estimation of the problem at that time?

    16. DF

      So I thought it was possible, and a lot of people thought it was impossible. I thought it was possible.

    17. LF

      Okay.

    18. DF

      The reason why I thought it was possible is because I did some brief experimentation. I knew a lot about how we were approaching o- open domain factoid question answering. We had, we had been doing it for some years. I looked at the Jeopardy! stuff. I said, "This is gonna be hard," for a lot of the, uh, points that you, we mentioned earlier. Hard to interpret the question, um, hard to do it quickly enough, hard to compute an accurate confidence. None of this stuff had been done well enough before. But a lot of the technologies we're building were the kinds of technologies that should work. But more to the point, what was driving me was, I was in IBM Research-I was a senior leader in IBM Research, and this is the kind of stuff we were supposed to do.

    19. LF

      Yeah, yeah.

    20. DF

      In other words, we were basically supposed to-

    21. LF

      This is the moon shot. This is the-

    22. DF

      I mean, we were supposed to take things and say, "This is an active research area. It's our obligation to kind of, if we have the opportunity, to push it to the limits, and if it doesn't work, to understand more deeply why we can't do it." And so, I was very committed to that notion, saying, "Folks, this is what we do. It's crazy not, not to do it."

    23. LF

      Yeah.

    24. DF

      "This is an active research area. We've been in this for years. Why wouldn't we take this grand challenge and, and push it as hard as we can?" At the very least, we'd be able to come out and say, "Here's why this problem is, is way hard."

    25. LF

      Yeah.

    26. DF

      "Here's what we tried and here's how we failed." So, I was very driven, um, as a scientist from that perspective, and then I also argued, based on what we did a feasibility study, uh, why I thought it was hard but possible, and I showed examples of, you know, where it succeeded, where it failed, why it failed, and sort of a high level architectural approach for why we should do it. But for the most part, that... at that point, the execs really were just looking for someone crazy enough to say yes, because for s- several years at that point, everyone had said no.

    27. LF

      Yeah.

    28. DF

      "Um, I'm not willing to risk my reputation and my (laughs) career, you know, on this thing."

    29. LF

      Clearly, you did not have such fears. Okay.

    30. DF

      I, I did not.

  9. 1:07:17 – 1:10:05

    Engineering constraints and the limits of naive search

    1. LF

      So, yeah, and backtracking to, uh, search. So if you were to do... What's the brute force solution? What, what would you search over? So you have a question, how would you search the possible space of answers?

    2. DF

      Look, web search has come a long way even since then. Um, but at the time, like, you- you- you know, you... First of all, I mean, there are a couple other constraints around the problem, which is interesting. So you couldn't go out to the web. You couldn't search the internet. In other words, the AI experiment was we want a self-contained device. A device, if the device is as big as a room, fine, it's as big as a room, but we want a self-contained advice, uh, contained device. You're not going out to the internet. You don't have a lifeti- lifeline to anything. So it had to kind of fit in a shoebox, if you will, or at least a size of a few refrigerators, whatever it might be. So you... But also, you couldn't just get out there. You couldn't go off network, right, to, to kind of go. So there was that limitation. But then we did ex-... But the basic thing was go, go do, go do a web search. The problem was, even when we went and did a web search, I th- I don't remember exactly the numbers, but somewhere on the order of 65% of the time, the answer would be somewhere, you know, in the top 10 or 20 documents. So first of all, that's not even good enough to play Jeopardy!. Um, you, in other words, even if you could pull the, even if you could perfectly pull the answer out of the top 20 documents, top 10 documents, whatever it was, which we didn't know how to do, but even if you could th- do that, you're, you'd be... And, and you knew it was right and you had enough confidence in it, right? So you'd have to pull out the right answer. You have, you'd have to have confidence it was the right answer. And, and then you'd have to do that fast enough to now go buzz in, and you'd still only get 65% of them right-

    3. LF

      Right.

    4. DF

      ... which doesn't even put you in the winner's circle.

    5. LF

      Right.

    6. DF

      Winner's circle, you have to be up over 70, and you have to do it really qui- and you have to do it really quickly. But then now the problem is, well, even if I had somewhere in the top 10 documents, how do I figure out where in the top 10 documents that answer is and how do I compute a confidence of all the possible candidates? So it's not like I go in knowing the right answer-

    7. LF

      Right.

    8. DF

      ... and I have to pick it. I don't know the right answer. I have a bunch of documents. Somewhere in there is the right answer. How do I, as a machine, go out and figure out which one's right? And then how do I score it? So, and now how do I deal with the fact that I can't actually go out to the web?

    9. LF

      First of all, if you pause on that, just think about it. If you could go to the web, do you think that problem is solvable? If you just pause on it, just thinking even beyond Jeopardy! Do you think the problem of reading text to find where the answer is?

    10. DF

      Well, we solved, we solved that in some definition of solved, given the Jeopardy! challenge.

    11. LF

      How did you do it for Jeopardy!? So how did you take a s- a body of work in a particular topic and extract the key pieces of information?

  10. 1:10:05 – 1:17:59

    Watson’s pipeline: pre-processing, parallel search, candidates, and hundreds of scorers

    1. DF

      So, what... So, now forgetting about the, the, the huge volumes that are on the web, right? So now we, we have to figure out... We did a lot of source research. In other words, what body of knowledge is gonna be small enough but broad enough to answer Jeopardy!?

    2. LF

      Yeah.

    3. DF

      And we ultimately did find the body of knowledge that did that. I mean, it included Wikipedia and a bunch of other stuff.

    4. LF

      So like encyclopedia type of stuff. I don't know if you can speak to-

    5. DF

      Encyclopedia, dictionaries-

    6. LF

      ... the different types of-

    7. DF

      Different types of semantic resources, um, like WordNet and other types of semantic resources like that, as well as like some web crawls. In other words, where we went out and took that content and then expanded it based on producing statistical seed... you know, statistically producing seeds, using those seeds for other searchers, searches, and then expanding that. So using these, like, expansion techniques, we went out and had... found enough content that we're like, "Okay, this is good." And even up until the end, you know, we had a thread of researchers always trying to figure out what content could we efficiently include.

    8. LF

      I mean, there's a lot of popular qu-... Like, "What is the church lady?" I think was one of the an-... Like-

    9. DF

      Yeah.

    10. LF

      ... what (laughs) w- where do you get... I guess you... That's probably an encyclopedia, so I guess-

    11. DF

      So as-

    12. LF

      ... the Beatles and some-

    13. DF

      ... encyclopedia, but then, but then we would, but then we would take that stuff and we would go out and we would expand. In other words, we'd go find other content-

    14. LF

      Right.

    15. DF

      ... that wasn't in the core resources and expand it. You know, the amount of content, grew it by an order of magnitude. But still, so again, from a web scale perspective, this is very small amount of content.

    16. LF

      It's very select. ?

    17. NA

      Very, very-

    18. DF

      We then, we then took all that content. We, we pre-analyzed the crap out of it, meaning we, we, we parsed it, you know, broke it down into all those individual words, and we did semantic, uh, syntactic and semantic parses on it, you know, had computer algorithms that annotated it, and we inde- we indexed that in a very rich and very fast index. So we have a relatively huge amount of, you know, let's say the equivalent of, for the sake of argument, two to five million bucks. We've now analyzed all that, blowing up its size even more because now we have all this metadata, and we then we richly indexed all of that, and, and by the way, in a giant in-memory cache. So Watson did not go to disk.

    19. LF

      So the infrastructure component there, if you could just speak to it, how tough, uh... I mean, I know 2000, maybe this is 2008, '9, you know, that, that's kind of a long time ago.

    20. DF

      Right.

    21. LF

      Uh, how hard is it to use multiple machines? Like, I mean, h- how hard is the infrastructure component, the hardware component?

    22. DF

      So we used IBM, so we used IBM hardware. We had something like, uh, I forgot exactly, but 2,000 or close to 3,000 cores completely connected. So we had a switch where, you know, every CPU was connected to every other CPU.

    23. LF

      And they were sharing memory in some kind of way.

    24. DF

      Large-

    25. LF

      Some kind of clever-

    26. DF

      ... shared memory, right? And all this data was pre-analyzed and put into a very fast indexing structure that was-

    27. LF

      That's awesome.

    28. DF

      ... all, all, all in, all in memory. And then, um, we took that question... We would analyze the question. So all the content was now pre-analyzed. So if I... So if I went and tried to find a piece of content, it would come back with all the metadata that we had pre-computed.

    29. LF

      How do you shove that question-How do you connect the, the big stuff with the meta, uh, the- the big knowledge base or the metadata and the- that's indexed to the- the simple little witty confusing question?

    30. DF

      Right. So therein lies, you know, the Watson architecture, right? So we would take the question, we would analyze the question, so which means that we would parse it and interpret it a bunch of different ways. We'd try to figure out what is it asking about, so we would come ... we had multiple strategies to kind of determine what was it asking for that might be represented as a simple string, a character string, um, or something we would connect back to different semantic types that were from existing resources. So anyway, the bottom line is we would do a bunch of analysis on the question and question analysis had to finish and had to finish fast. Um, so we'd do the question analysis because then from the question analysis, we would now produce searches. So we would, um, and we had built, um, using open source search engines, we modified them, but we had a number of different search engines we would use that had different characteristics. We went in there and engineered and modified those search engines ultimately to now take our question analysis, produce multiple queries based on different interpretations, um, of the question and fire out a whole bunch of searches in parallel.

  11. 1:17:59 – 1:27:52

    How to run a landmark AI project: end-to-end metrics, modular teams, ML as the integrator

    1. DF

      We- we actually looked at the raw scores as well as standardized scores because humans are not involved in this. Humans are not involved.

    2. LF

      Sorry, so I'm- I'm misunderstanding the- the- the process here. This- this is passages. Where is the ground truth coming from?

    3. DF

      Ground truth is only the answers to the questions.

    4. LF

      So it's end-to-end.

    5. DF

      It's end-to-end. So we al- ... so I was always driving end-to-end perform- ... it was a very interesting-

    6. LF

      Wow.

    7. DF

      ... a very interesting, you know, engineering, um, approach and ult- ultimately scientific and research approach were always driving end-to-end. Now that's not to say we- we- we wouldn't make hypotheses that individual component performance was related in some way to end-to-end performance.

    8. LF

      Right.

    9. DF

      Of course we would because people would have to build individual components. But ultimately to get your component integrated into the system, you had to show impact on end-to-end performance, question answering performance.

    10. LF

      So there's- there's many very smart people working on this and they're basically trying to- to, uh, sell their ideas as a component that should be part of the system.

    11. DF

      That's right. And- and they would do research on their component and they would say things like-

    12. LF

      So cool.

    13. DF

      ... you know, "I'm gonna improve this as a candidate generator," or, "I'm gonna improve this as a question scorer or as a passage scorer, I'm gonna improve this, or as a parser and I can improve it by 2%."... on its component metric, like a better parse or a better candidate, or a better type estimation, or whatever it is. And then I would say, "I need to understand how the improvement on that component metric is going to affect the end-to-end performance." If you can't estimate that and can't do experiments that demonstrate that, it doesn't get in.

    14. LF

      That's like the best run AI project I've ever heard. That's awesome. Okay. Uh, what breakthrough would you say... Like, I'm sure there's a lot of day-to-day breakthroughs, but was there like a breakthrough that really helped improve performance? Like where, where people began to believe? (laughs) Or is it just a gradual process

    15. DF

      Well, I think it was a gradual process. But one of the things that I think gave people confidence that we can get there was that as we follow this, as we follow this procedure of different ideas, build different components, plug them into the architecture, run the system, see how we do, do the error analysis, start off new research projects to improve things. And the, and, and, and the very important idea that the individual component, um, work did not have to deeply understand everything that was going on with every other component. And this is where we, we leverage machine learning in a very important way. So while individual components could be s- statistically driven machine learning components, some of them were heuristic, some of them were machine learning components, the system as a whole combined all the scores using machine learning. This was critical because that way you can divide and conquer. So you can say, "Okay, you work on your candidate generator," or, "You work on this approach to answer scoring. You work on this approach to type scoring. You work on this approach to pa- uh, passage search or to passage selection," and so forth. Um, but when we j- just plug it in and we had enough training data to say, "Now we can, we can train and figure out how do we weigh all the scores relative to each other based on the, uh, predicting the outcome, which is right, right or wrong on Jeopardy." And we had enough training data to do that. So this enabled people to work independently and to let the machine learning do the integration.

Episode duration: 2:24:31

Install uListen for AI-powered chat & search across the full episode — Get Full Transcript

Transcript of episode Whtt2H5_isM

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

iOSAndroidClaudeChrome