Stanford CS230 | Autumn 2025 | Lecture 3: Full Cycle of a DL project

1. KKKian Katanforoosh0:05

   So, um, what I'd like to do today is chat with you about the full cycle of a deep learning project. And as I promised in the first lecture, rather than me talking at you for an hour and twenty minutes or whatever, um, would love for this to be much more interactive. And so what I'm gonna do is illustrate this with an example, but also ask you a bunch of questions along the way about what you would do if you are the one working on the project that I'm gonna use as an illustrative example. Okay? So plan for this to be quite interactive, and please interrupt any time and ask questions since I think that's why, uh, you know, we-- I want to do this in person here at Stanford. Um, so one of the reasons why developing machine learning or deep learning or other types of AI projects, including AI projects using large language models or agent AI workflows or whatever, is that AI projects are different than traditional software engineering projects. So one of the biggest difference between AI projects of the many different flavors, supervised learning, LLM base, um, you know, generative AI base, is that in traditional software projects, you write code and you control your code, right? Write whatever code you want, compile it, your code does what you tell it to. But AI projects, you know, involve both code as well as data that you train your algorithm on, and you almost never know what strange and wonderful things there are in your data. For example, if you're working on a face recognition application, that's a running example I'm gonna use today, then as you're just getting started on the project, you actually-- it's really difficult to know in advance, um, what you find in the data. Is the lighting of the faces good? Do you struggle with people with very long hair or people with short hair? Um, do people making weird facial expressions make your system struggle? Do people wearing glasses make your system struggle? So data is so rich that a lot of the time you can't predict in advance what your AI system is going to do, not because you don't control the code. You can write whatever neural network or whatever code you want, but you don't know what's in the data. And this is why unlike traditional software engineering, um, machine learning development is a much more iterative process where you just have to build something, see how it works, and then through a process discover, almost discover what is in the data and therefore what you should be doing to change your code to make your overall system perform. And this is also true not just for-- this is true not just for deep learning based systems. This is also true for modern large language models. If you've-- There's been a lot of hype, buzz about how LLMs, large language models, are hard to control. I think there's a lot of excessive hype about that, you know, kind of fear-mongering. There's a little bit of that. But one of the reasons why none of us know in advance what LLMs do is because it was trained on, on a lot of data, more data than any human could possibly look at. And we just don't know, you know, with precision what is in all that data that the LLM was trained on. And so because we don't know the data, we can't really look at all the massive tens of trillions of tokens of data it was trained on. It's hard to know exactly how a large language model will perform. Which is why building agent AI applications or building large language models based applications is also very empirical or, or very experimental, meaning you just have to build something, then see where it goes well, see where it goes poorly, and then use that to fix problems, and that's how you drive progress. That make sense? So because you control your code but you don't really know, it's hard to control the data. And this is true both for the data you have stored in your hard disk. You know, kind of, I don't know, terabytes of data stored in a hard disk. I don't really know what's in my hard disk. Um, and the thing you really don't control is what data the world will give you in the future. So we deploy a system in the world, uh, say, face recognition, which I-- we should talk about. Will people wear a thick, heavy scarf that covers part of their face when it's winter? You just don't know. There are all these things in data that will surprise you. And, and if you don't even control your past data that's already in your hard disk, you certainly can't control your future data. So, um, it turns out that a lot of machine learning classes talk about building models, right? And you learn a lot from the, um, online videos, uh, about how to build powerful deep learning models. But it turns out that building the overall machine learning system or a deep learning system has a lot more work than just training models. But, um, if you look at a lot of courses, there's actually a very strong focus on modeling because I think that's what academia has focused on, right? We can train models, evaluate models, publish papers on models, and so you find that a lot of courses focus on, you know, training a good deep learning model. And that is absolutely important. Um, but because we know how to evaluate models, different research groups can benchmark different models. There's a lot of academic research work on that that's reflected in a lot of courses. But this is just a small part of what you need to do if you want to build an effective deep learning system. And what I want to do today is, um, go outside the small box of models to give you a broader view of what it feels like to develop a deep learning or AI or machine learning system. Okay? And this is what it often, uh... So this is, this is what, um, building a deep learning system would be like. Which is first, you know, right? Specify the problem, figure out what we're actually working on.The one example I want to use today will be to build a face recognition system or a face rec system for, um, security, for deciding when to unlock a door. And then, you know, Kian talked about some face recognition, which I believe in face rec-- face rec architectures last week as well. But, um, specific application I wanna talk about is something I've worked on. Some-- Built-- Actually, I built one of the commercial systems that, um, uh, uh, that, uh... For, for this is, um, if this is a door, um, and, you know, this is... Well, you or a friend or, or maybe a, a, a someone that you don't want to let in approaching. You have a little camera... Sorry, bad drawing. Take a picture of whoever's approaching the door and decide whether or not to unlock the door, right? So face recognition to decide who's authorized to enter, like, a restricted location, like a, you know, corporate office building or your house or whatever. Um, and actually one, one, one common use case that, um, one of my teams built was, uh, key card-- swipe key cards. So sometimes key cards get stolen, um, and so one of the systems we deployed, you know, fairly large office complexes, was, uh, um, if you swipe a key card, we also just take a picture, the zoom soon discard it to make sure that the key card is held by the person, you know, whose face is shown on the key card. So it makes it harder for someone to steal a key card to gain unauthorized access to, to, like, an office complex, right? So, so, so I'm gonna use this as, uh, the motivating example for today. And after, um, specifying a problem, typical process is then, right, um, [coughs] sometimes the open source models you can download, but let's say for today that the open source models aren't good enough, you wanna train your own model. The typical process, we get data, you know, design a model, right, train a model, um, and then we will iterate through these steps a bunch of times until the model looks like it's performing well enough. Um, and then after that, we have to deploy it. Um... And, uh, monitor and maintain the model. Okay? So I'm actually gonna talk a bunch about, uh, multiple of these steps today because I want you to come away with a feel for when you're working on a real machine learning application, what the, what, what the important steps you will face are. Right? Um, and so as I alluded, machine learning development is very iterative process. So for these three steps, we often drive a, um, rapid development loop where we, you know... Oops, sorry. Design the model, train it, analyze the results, and then maybe design or update the model or the data or something and iterate around this loop many times before you, um, are satisfied. And, um... All right, just one more detail. Um, it turns out that for face recognition, the very common architecture, which you learn in detail about, uh, later in the online videos, is a neural network called a Siamese network. And what that does is a neural network that takes as input two pictures, [clears throat] and two pictures get fed to a neural network or a deep learning algorithm, and it's the job of the deep learning algorithm to tell us, are these two pictures the same person or different persons? Right? Because, um, if you're trying to set this system, say, for your house, and maybe you have, I don't know, a few family members or roommates you want to let in, then it's, I don't know, it'd be quite annoying if you had to retrain a neural network for every single home. So the most common way to do face recognition is have a neural network that inputs two pictures and the job of the neural network is says, "Tell me, are these two pictures of the same person?" And then the way you set up the system is to input a few registration pictures. So take a picture of yourself, take a picture of your roommate, take a picture of, you know, any family members you want to have access. So then when someone comes, it can quickly check if the person that just showed up is one of the people that's authorized and then let them in. And then the corporate key card swipe example is someone swipes a card and, you know, my card says, "This is Andrew's card," then it'll quickly pull up my registration picture to double-check if I am-- if I seem to be the same person as the, you know, Andrew Ng that was registered. Right? So this is how, um, this is a typical neural network architecture. All right. So I have a question for you. One thing, one, one, one thing I'd like to do today is, um, walk you through a number of scenarios and invite you to think about what decision you would make if you are the CTO of a startup, uh, building these technologies. Right? So, so my question for you is, um, if you are the CTO of a startup building the next face recognition system, um, and if your lawyers have said you aren't allowed to download data from the internet, right? So let's not download data from the internet for this application. How would you go about getting data to train the system? So what you need is a bunch of pictures of people, right, to train a neural network to say, you know, are they the same person or not?So maybe take, take a few minutes to think about it, and then I'll see if people raise hands and give some answers. And one, one specific question I have as well is, um, how long would you take to collect data before you start training a model, right? So I think, um, uh, well, you need to get data. We've designed the model, I guess, and then you need to train model. So what does the timeline look for you? We specify the problem, you know, we design the model, how many hours or days or weeks would you spend to get data and how before you start running, you know, gradient descent? Right. Maybe-- Uh, I, I see a hand up. Oh, sure, go ahead.

2. SPSpeaker12:48

   Uh, are we doing this everyone around the world, or are we, like, specifically looking at, like, Bay Area or America scenario of this, or is, like, this wide range of different-

3. KKKian Katanforoosh13:03

   Let me, let me leave it a bit more open-ended. Say you just graduated from Stanford, and you're a CTO of a three-person startup, um, building this thing, uh, that eventually, hopefully, you sell all around the world, but your goal is to just get started with, with the three of you working out of, you know, Palo Alto, California. Right. And, and do it as your real self, right, with the resources that you have. All right. Anyone want to venture an answer? So how would you, how would you get data to train a neural network? Go for it.

4. SPSpeaker13:39

   Um, how about using some video streaming service and, uh, collect the video data? Or like, uh, uh, maybe people are asking some company to, uh, if I can get the data from them, or we can even create a small video streaming service and then we collect the data people upload. So then we need to share the videos.

5. KKKian Katanforoosh14:02

   Yeah. Cool. Video streaming service. By video streaming service, are you thinking like, you know, Netflix and Hulu? Or are you thinking like, um, security videos?

6. SPSpeaker14:11

   Like Zoom.

7. KKKian Katanforoosh14:11

   Oh, like Zoom. Oh, I see. I see. Cool.

8. SPSpeaker14:12

   Like video-- people talking video.

9. KKKian Katanforoosh14:15

   I see. Cool, cool. Right. Cool. Right. Video streaming service. How, how long do you think it'll take to, to do that?

10. SPSpeaker14:21

    Uh, a couple of months. Uh, you know, the reason, the reason is the activity machines that-- I may be able to collect it from the accounts of some people like that. People create their own different versions for your research.

11. KKKian Katanforoosh14:33

    Yeah. Cool. Right. Cool beans. Cool. Thank you. Right. Creative idea. Any other ideas? How would you go about and get data? Yes, go ahead.

12. SPSpeaker14:40

    This one might work for a three-person startup, but at a larger company, I would put a camera in the plan to have the camera at full swing up and swipe and just take-- asking employees if they would like to participate in data collection, and you can take pictures of the employees at least three times a day. So the employees that are actually working.

13. KKKian Katanforoosh15:00

    Yeah.

14. SPSpeaker15:00

    You might have to generalize it with the physical presence policy.

15. KKKian Katanforoosh15:05

    Yeah.

16. SPSpeaker15:06

    Yes, that works.

17. KKKian Katanforoosh15:06

    Yeah. Cool. Awesome. Right. Stick a camera there. Uh, let, let people, you know, opt in, right, to, to get their picture taken. Cool. I think I saw a hand up. Go for it.

18. SPSpeaker15:18

    Yeah. We subject the users.

19. KKKian Katanforoosh15:19

    Oh, sorry. It's, it-

20. SPSpeaker15:19

    Taking reference from all the services with the camera.

21. KKKian Katanforoosh15:25

    I see. Cool.

22. SPSpeaker15:27

    Users.

23. KKKian Katanforoosh15:27

    How, how would you get users?

24. SPSpeaker15:29

    Well, we review each user. So we need more people.

25. KKKian Katanforoosh15:38

    I see. Cool. Right. Yeah. By your own people, you would, like, grab some friends and also they'll, you know, give you that LinkedIn photo, give you some pictures from your camera roll. Yeah. Cool. I like that. That's fast. I, I, I like that. Any other ideas? Oh, go ahead.

26. SPSpeaker15:54

    Ask Stanford students, uh, survey Stanford students.

27. KKKian Katanforoosh16:02

    I see. Interesting. Yeah. Ask Stanford to send an email. Cool. But I love Stanford. Stanford's a wonderful institution. Ask-- [laughs] that's gonna take a while, I think. [laughs] Very creative idea. I li-- I like the creative ideas. So let me share with you one guiding principle for how I would encourage you to approach this problem of data collection. I appreciate all the creative ideas, actually. Um, one of the frameworks I often use to decide how I collect data is speed, um, because I find that, um... Actually, especially if you're building a startup, right, uh, one of the-- in my opinion, one of the strongest predictors for whether a startup will succeed and also whether a small innovative project in a large corporation, you know, it could be a giant corporation, but if a team of three, you know, work on a small innovative project in a big, big company, I find that one of the biggest predictors for the chance of success is just the sp-she-- speed of execution. It's just speed of getting stuff done. And so when I'm, um, sitting with a team and brainstorming different tactics, I will gravitate toward the tactics that let me get a dataset very quickly. And quickly usually means, you know, like one or two days, um, even if it ends up with a inferior, smaller, lower quality datasets or whatever. Um, because I don't really know what problems I'll see in my data, and the quicker I can, you know, get a dataset, train a model, see where it goes wrong, the quicker I can then discover what's wrong with my data and fix it. Um, true story. Chatting with a CEO that told me he once-- actually once, um, he, he actually had spent, um, uh, I think it was over a hundred million dollars, de-definitely more than tens of million dollars. Spent a lot of money buying a company for his data. Um, and then he actually said, "Hey, Andrew, I spent all this money to get all this data, you know, can you help me figure out how to monetize this? How to make money off of this?" Right? And I kind of looked at him and goes, "Boy, I kind of wish I hadn't done that." Right? Uh, um, and what I find is that, um, the value of data, it, it's just so difficult to know in advance, right? What's important and what's not important about data. So for a lot of these tactics, um-Uh, for example, I think student ID is an interesting one. Uh, but, you know, are student ID photos weird in some way, right? Or are they too expressionless or, or people smiling too much in student IDs? I actually have no idea. Um, actually, my, my Stanford ID, I look really weird in my Stanford ID. Um, [laughs] uh, or, um, uh... And, and I think you-- and, and, or, or, or, uh, and I actually like the idea of sticking a camera and just letting people, you know, come up and, and opt in to take a picture, if you can do it quickly. Um, and I think in a big company or even in a small startup, if, you know, it's important to respect user privacy or individual privacy. But if you can stick a camera in some place, you know, that doesn't, um, uh, kind of, uh, what's the word, um, invade people's privacy that don't want to be any part of this, but let people opt in, take the pictures with permission. If you can do that in days, I find that to be valuable. One of the things I found for quite a few of my projects, found that, you know, Stanford students, our community here is pretty cosmopolitan. We're people from all around the world. We're not fully representative of the world's, you know, distribution of people, but we actually do have people from all around the world. A lot of Stanford people are actually very nice. So one thing I've done multiple times is when I need to collect data, uh, we'll go to places on campus with high foot traffic. It turns out cafeterias are very high foot traffic. And we just ask people, "Hey, we're working on a project. Can I get a sample of your voice? Can I take a picture of you?" And kind of with, you know, really informed consent. Tell people, "Is it okay if I do this?" I've been delighted at how collaborative, right, Stanford students are. Uh, uh, um, but-- And, and I find that one thing I've often done is gone to my teams and said, um, "We have two days to collect data." It's like, whatever. "It's eleven fifty-two AM now. Let's figure out what we can do by, uh..." What day is it? Tuesday. "Let's figure out what we can do by Thursday, eleven fifty-two AM. So let's give us forty-eight hours, and let's brainstorm. How can we collect data?" And it's fine that the data is, isn't all there, fine that the data is low quality, but that velocity lets us more quickly train a model, figure out what's wrong with the data, and then jigger or retweet how we collect data. Right? So I find that there's some teams that will ask, um, "How can we collect the data we need, and how long?" And then ask, "How long will that take?" That usually leads to much slower execution. I instead tend to go to my teams and say, "We have two days or one day or maybe a week," right? Some short time span like that, and say, "What's the most creative, you know, respectful, responsible, but creative way you can use to collect data in this short time span?" And one of the ways to think about that too is training a model takes, I don't know, let's, let's say it takes two days. Maybe it actually takes more like one day, right? If you can train a model in a couple of days, then I would not spend, you know, like, whatever, right, two months to find data to train a model, because then this becomes a huge bottleneck. Because you can train a model relatively quickly, let's take a commensurate amount of time to design a model or, or design the data as to train a model. And depending on how long it takes to train a model, sometimes training a model, you know, needs to run overnight. Um, I actually see teams sometimes take one-day iteration loops around this. The fast-moving teams I work with, we often go around this loop once per day, uh, for the smaller models. If we're training a large AI foundation model, sometimes training a model takes weeks or even months, then the process can be different. If, if your, if your model run is gonna be, you know, two months, then yeah, maybe it makes sense to spend, you know, a couple of months to get the data really right. But for face recognition, you could train that overnight or actually in a couple of hours quite easily. So it makes sense not to spend massive amounts of time before you go in to train the model. Does that make sense? And because, um... Oh, uh, the, the word empirical means experimental, right? Sorry, I've used that word a few times today. And, um, I, I think, uh, uh, we, we, we say that machine learning is a very empirical process, meaning it's a very experimental process. You have to do it, see what happens, and decide what to do next. I know that sometimes others have criticized our field as like we never know what we're doing. We just try stuff and see what works. And then, you know, there's a little bit of truth to that. Um, uh, I think understanding neural networks, hyperparameters, architecture, that is really valuable, so we don't just try stuff at random. But because we don't know what's in the data, we do try a lot of things and then drive a disciplined process to understand what works and what doesn't, and then use that to navigate forward. Does that make sense? And then I'll just say there's one exception to the advice I'm giving here, which is if you're working on a project that you have worked on many times before. So, you know, whatever. I've built a bunch of face recognition systems. So I kinda have a sense from previous experience and from reading research papers that certain things I know are just not gonna work if I have, you know, a hundred images, right? So there's some face recognition systems I know I probably need at least fifty thousand images before it'll have any hope of working. So because of that prior experience, having gone through that loop a lot, I now have a basis to say, "Okay, I do need fifty thousand images." Then I might invest upfront and, and put more effort upfront to, to get those fifty thousand images. But I think for most applications you work on for the first time, if you don't have a academic literature to justify certain larger data investments, or if you don't have the prior experience yourself, then, um, I would focus on the speed of iterating around this loop. Make sense? Um, all right. And to relate this to, uh, large language models-based applications as well-A lot of us, a lot of you are probably-- Well, you may be building applications like prompting LLMs and calling an API, right? Like a, like a OpenAI or Anthropic or Gemini or Meta Llama or whatever API to get things back from an LLM. One of the reasons that too is a very experimental, a very iterative empirical process is because when you write an LLM prompt, you don't really know in advance how well it's going to do because it was trained on data that none of us have really looked at. And that too is why instead of, um, theorizing for a long time about what prompt to use an LLM, you know, just try it out. And then it's by doing that, you then see the problems and, um, then your focus can be on fixing the problems. Make sense? And in fact, there's a lot of discussion on responsible AI, how do you make sure AI systems are safe or they don't have kind of unforeseen circumstances. And because a lot of the theorizing, you know, you can only theorize so much. I find that if you want to build safe, responsible AI systems, one of the best ways to do that is to just build it and then experiment with it in a sandbox environment. Just don't let it out in the world until you've tested rigorously. But just go build something and then, you know, test it, probe it in the safety of your own laptop, right? Don't let it-- into innocent users and, and have some weird impact on, on innocent third parties. But it's only by building and then probing it that you can figure out where can it go wrong, where it will say inappropriate things, where would, you know, respond inappropriately to certain user queries, and then that tells you where the problems are that you may work, work on addressing. Okay. All right. Yeah. Question?

28. SPSpeaker26:09

    How do you use, like, analyzing kind of like quality results to back that up, like data from changing validation?

29. KKKian Katanforoosh26:18

    Oh, sorry. Say that again.

30. SPSpeaker26:19

    So, like, when you use models to come up-- like, when you do analyzing of the model, right, and you come up-- come to a conclusion that the model is not good, like, how do you use that to kind of figure out what's wrong with the data?

Episode duration: 1:07:04