Rajat Monga: TensorFlow | Lex Fridman Podcast #22

1. 0:00 – 1:10:16

   Intro

   1. LFLex Fridman0:00

      The following is a conversation with Rajat Monga. He's an engineering director at Google, leading the TensorFlow team. TensorFlow is an open source library at the center of much of the work going on in the world in deep learning, both the cutting edge research and the large-scale application of learning-based approaches. But it's quickly becoming much more than a software library. It's now an ecosystem of tools for the deployment of machine learning in the cloud, on the phone, in the browser, on both generic and specialized hardware, TPU, GPU, and so on. Plus, there's a big emphasis on growing a passionate community of developers. Rajat, Jeff Dean, and a large team of engineers at Google Brain are working to define the future of machine learning with TensorFlow 2.0, which is now in alpha. I think the decision to open source TensorFlow was a definitive moment in the tech industry. It showed that open innovation could be successful, and inspired many companies to open source their code, to publish, and in general, engage in the open exchange of ideas. This conversation is part of the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter, @LexFridman, spelled F-R-I-D. And now, here's my conversation with Rajat Monga. You were involved with Google Brain since its start in 2011 with, uh, Jeff Dean. It started with this belief the proprietary machine learning library and turned into TensorFlow in 2014, the open source library. So, what were the early days of Google Brain like? What were the goals, the missions? How do you even proceed forward once there's so much possibilities before you?

   2. RMRajat Monga1:47

      It was interesting back then, you know, when I started out, when you, you were even just talking about it. The idea of deep learning was interesting and intriguing in some ways. It hadn't yet taken off, but it held some promise and it showed some very promising and early results. I think the, the idea where Andrew and Jeff had started was, what if we can take this, what people are doing in research, and scale it to what Google has in terms of the compute power? And, uh, also put that kind of data together, what does it mean? And so far, the results have been if you scale the compute, scale the data, it does better, and would that work. And so that, that was the first year or two, "Can we prove that out right?" And with this belief, when we started the first year, we got some early wins, which, which is always great.

   3. LFLex Fridman2:40

      What were the wins like? What was the wins where you were, "There's some promise to this, this is gonna be good"?

   4. RMRajat Monga2:46

      I think the two early wins were, one was speech that we collaborated very closely with the speech research team who was also getting interested in this, and the other one was on images where we, you know, the cat paper as we call it-

   5. LFLex Fridman2:59

      Mm-hmm.

   6. RMRajat Monga2:59

      ... that was covered by-

   7. LFLex Fridman3:00

      Yeah.

   8. RMRajat Monga3:00

      ... (laughs) uh, a lot of folks.

   9. LFLex Fridman3:03

      And, uh, the birth of Google Brain was a- around neural networks. That was ... So, it was deep learning from the very beginning.

   10. RMRajat Monga3:09

       That's right.

   11. LFLex Fridman3:09

       That was the whole mission.

   12. RMRajat Monga3:10

       Yeah.

   13. LFLex Fridman3:10

       So, what, what, uh, in terms of scale, what was the sort of, uh, dream of what this could become? Like, what, were there echoes of this open source TensorFlow community that might be brought in? Was there a sense of TPUs? Was there a sense of like, machine learning is now gonna be at the core of the entire company, is g- going to grow into that direction?

   14. RMRajat Monga3:36

       Yeah, I, I think ... So, so that was interesting, and like, if I think back to 2012 or 2011-

   15. LFLex Fridman3:41

       Right.

   16. RMRajat Monga3:41

       ... and first was, can we scale it? And in the year or so, we had started scaling it to hundreds and thousands of machines. In fact, we had some runs even going to 10,000 machines, and all of those shows great promise. Uh, in terms of machine learning at Google, the good thing was Google's been doing machine learning for a long time. Deep learning was new, but as we scaled this up, we showed that, yes, that was possible, and it was gonna impact lots of things, like we started seeing real products wanting to use this. Again, speech was the first. There were image things that photos came out of, and, and then many other products as well. So, so that was exciting. Um, as we went into that a couple of years, externally also, academia started to, you know, there was lots of push on, "Okay, deep learning's interesting. We should be doing more," and so on. And so, by 2014, we were looking at, "Okay, this is a big thing. It's gonna grow," and, uh, not just internally, externally as well. Yes, maybe Google's ahead of where everybody is, but there's a lot to do, so a lot of this start to make sense and come together.

   17. LFLex Fridman4:46

       So, the decision to open source ... I was just chatting with, uh, with Chris Lattner about this. Uh, the decision to go open source for TensorFlow, I w- I would say is that for me personally seems to be one of the big seminal moments in all of software engineering ever.

   18. RMRajat Monga5:01

       (laughs) .

   19. LFLex Fridman5:01

       I think that's a ... When a large company like Google decides to take a large project that many lawyers might argue has a lot of IP, just decide to go open source with it, and in so doing, lead the entire world in saying, "You know what? Open innovation is, is, is a pretty powerful thing, and it's okay to do." (laughs) Uh, that, that was ... I mean, that's an, uh, that's an incredible, credible moment in time. So, do you remember those discussions happening?

   20. RMRajat Monga5:29

       Yeah.

   21. LFLex Fridman5:29

       Whether open source should be happening? What was that like?

   22. RMRajat Monga5:32

       I would say, I think I ... So the, the initial idea came from Jeff, who was a big proponent of this. I think it came off of two big things. Uh, one was, research-wise, we were a research group. We were putting all our research out there, if you wanted to ... We were building on others' research, and we wanted to push the state of the art forward, and part of that was to share the research. That's how I think deep learning and machine learning has really grown so fast.So the next step was, okay, now would software help with that? And it seemed like there were existing a few libraries out there, Theano being one, Torch being another, and a few others, but they were all done by academia, and so the level was, was significantly different. The other one was, from a software perspective, Google had done lots of software or that we used internally, you know, and we published papers. Often, there was an open source project that came out of that, that somebody else picked up that paper and implemented, and they were very successful. Back then, it was like, "Okay, there's Hadoop, which has come off of tech that we built." We know the tech we've built is way better for a number of different reasons. We've, you know, invested a lot of effort in that. And turns out, we have Google Cloud and we are now not really providing our tech, but we are saying, "Okay, we have Bigtable, which is the original thing. We are gonna now provide HBase APIs on top of that, which isn't as good, but that's what everybody's used to." So there's, there's like, can we make something that is better and really just provide... Helps the community in lots of ways, but also helps push the right... a good standard forward.

   23. LFLex Fridman7:18

       So how does cloud fit into that? There's a TensorFlow open source-

   24. RMRajat Monga7:21

       Right.

   25. LFLex Fridman7:22

       ... library. And how does the fact that you can, uh, use so many of the resources that Google provides in the cloud fit into that strategy?

   26. RMRajat Monga7:31

       So, so TensorFlow itself is open and you can use it anywhere, right? And we wanna make sure that continues to be the case. On Google Cloud, we do make sure that there's lots of integrations with everything else, and we wanna make sure that it works really, really well there, so...

   27. LFLex Fridman7:47

       You're leading the TensorFlow effort. Can you tell me the history and the timeline of TensorFlow project in terms of major design decisions? So like the open source decision, but really, uh, you know, what to include and not. There's this incredible ecosystem that I'd like to talk about.

   28. RMRajat Monga8:04

       Yeah.

   29. LFLex Fridman8:04

       There's all these parts, but what, uh, if you just... Some sample moments that, uh, de- defined what TensorFlow eventually became through its... I don't know if you're allowed to say history when it's just...

   30. RMRajat Monga8:19

       (laughs)
2. 1:10:16 – 1:10:42

   How to get started with TensorFlow

   1. LFLex Fridman1:10:16

      so if I'm a complete beginner interested in machine learning and TensorFlow, what should I do?

   2. RMRajat Monga1:10:21

      Probably start with going to our website and playing there. There's-

   3. LFLex Fridman1:10:24

      So just go to tensorflow.org and start clicking on things?

   4. RMRajat Monga1:10:26

      Yep. Check our tutorials and guides. There's stuff you can just click there and go to a Colab and do things. No installation needed. You can get started right there.

   5. LFLex Fridman1:10:34

      Okay. Awesome. Rajit, thank you so much for talking today.

   6. RMRajat Monga1:10:36

      Thank you, Lex.

   7. LFLex Fridman1:10:37

      It was fun.

   8. RMRajat Monga1:10:37

      It was great.

Episode duration: 1:10:57