The Most Important Skill for AI Engineers: Evals Explained

1. 0:00 – 4:55

   Why AI products can't be tested like normal software

   1. ARAishwarya Reganti0:00

      Building an agent is honestly the easiest part of building AI systems, and these days it looks like anyone can get something working in less than a day. So the hard part is making sure that it actually holds up in production. So if you wanna build AI products that survive the real world, the number one skill you should be mastering is AI Evals. It's also one of the most common topics that you'll see in AI interviews, and the most misunderstood, too. So let's change that today. By the end of this video, you'll understand what AI Evals are, why they're important. We'll also learn the differences between model evals and product evals. We'll learn about the three most popular ways to build evals for AI systems and when to use each of them. And finally, we're gonna do a walkthrough where you build your first eval so that you see the process end to end. And then we'll wrap up with how evals work in production so that your system can keep improving while it's live. Here's everything we'll be covering today. You can take a screenshot of it or find the HD version on my GitHub repository. So let's start with the very first question. What are AI evals and why do you even need them? Because this is where I think a lot of the confusion lies for most beginners. So evaluations, or for short, evals in AI, are essentially checks you use to find out whether a system is doing what it was designed to do. And if you're coming here from a software background, this seems pretty much like testing. Like you would write unit tests and integration tests and regression tests for your applications. So that intuition is somewhat correct, but the way we test AI products is a little different than the way we test typical software products, and this is why, right? Maybe let's take the example of a website that most people use, right, something like a booking.com. Now, when you go there to book a hotel, you're picking the dates off a calendar, you're clicking through a couple of dropdowns, you're selecting a room, and you're hitting a button, and all of this. Now, if you closely observe how these typical web apps are built, the action space is heavily controlled by humans, that is, by us, the people who are operating these things. And you'll also notice that the kind of decisions or actions you take, they're very limited and predictable in the sense that the product has already decided what you're allowed to do. For instance, clicking buttons, dropdowns, and all of this. The output is equally well-defined. You click, the page either loads or the transaction goes through or you get an error. And the process which is making all of this happen is code, which is also deterministic in nature. So when you build something like that, you could have unit tests, integration tests, and all of that because of one basic assumption, which is if you've tested a particular input X and the system reliably gives you an output Y, it's going to keep doing that every time a new user shows up. Of course, there can be bugs or errors or sometimes issues which need corrections, but usually if something works before going into production, it should continue to work in production as well. Or in general, the number of times it messes up is pretty less, so you can pretty much simulate things well before production. Now, that assumption changes a little when you think about AI products because most of the interfaces for AI products are a little more free-form in nature. For instance, somebody might be typing into a chat box in their natural language and you don't know how people communicate. And people have very different styles of communication, and there's no dropdown or button that's constraining them. So in a sense, the range of things that people might send to an AI system is essentially unbounded, and you genuinely can't sit down beforehand and write out everything that the user might come up with. And the system that's making decisions is also not code. It's an AI model, which is probabilistic in nature. So for the same input or a slightly reworded one, you may not get the same output. And another annoying thing about AI products is that when traditional software breaks, it usually breaks loudly. For instance, you get an exception, you get a stack trace. But when an AI system gets something wrong, it often just returns a confident, well-written, and completely incorrect answer. There's nothing like an error report, so mostly it should be inferred by how your customer reacted to it. So to pretty much summarize, in AI, the input from the user side is far more free-form than in traditional software. And also the process, that's the model generating the output, is also probabilistic in nature. So if you really think of it, there are two surface areas that make an AI product much less deterministic than traditional software, and that is why AI evals look so different from traditional software testing that people are probably aware of. Another side effect of this is that evals are not a build once and forget kind of a thing. So you build them before you ship against the behavior you simulate, and then you keep updating them once your real users arrive because that's when you start learning how people are actually behaving with your product as opposed to how you anticipated they would. So you simulate behavior before production, you build evals based on that behavior, and then you observe real production behavior and keep improving your evals until you actually trust them. That's the whole evaluation loop.

2. 4:55 – 7:56

   Model evals vs. product evals (and why benchmarks like MMLU Pro aren't enough)

   1. ARAishwarya Reganti4:55

      So now that you understand why we need evals, the second question most people end up asking as soon as they learn about evals is this. If model providers like OpenAI, Anthropic, or Google already publish a bunch of evals and benchmarks and scores as to how their models are performing on tasks like coding, question answering, general ability, reasoning, et cetera, why should users be doing their own evals? So why should product builders be repeating all of this? Can we just not trust what these model providers tell us? Can we not trust all of those benchmarks? Now, that brings us to the two different kinds of evals which exist, and you really need to understand the difference between them, and those are model evals and product evals. And they're actually doing two completely different jobs. So when a company like OpenAI builds a new model, they need some way to tell people what the model is capable of and how well it does that, right? For instance, like can it write code? Can it reason? Can it answer general knowledge questions? Can it do math, and all of this. So they run their model against a set of standard benchmarks which test for all of these abilities, and they get back some scores. Those are usually model evals. And a benchmark essentially in this context is a data set that contains a bunch of inputs paired with the expected output, and the score essentially tells you how often the model matched that expected output. And some popular model eval benchmarks include MMLU Pro, which tests reasoning across a range of academic subjects. There's also another popular benchmark called SWE-bench, which tests models' coding abilities. And there are also independent organizations doing this. One example is Arena AI, and this is a crowdsourced benchmark where people can go and give their scores to different models based on different categories of tasks. Now, all of this is super useful, but they're only tested on generic tasks, right? They pretty much tell you how capable one model is compared to the other one, but all of that is measured in standardized conditions. And when you start building your own AI product, your entire ecosystem starts changing, and those are not standardized conditions, right? You're probably building a support agent which has to work on your company's policies, internal tools, knowledge bases, your own users, and all of that, right? And Model evals do not simulate all of that behavior for you because that's very unique to your ecosystem. So while model evals are kind of useful to say what are the most capable models available, they're not super useful for you to judge whether that model will be able to perform in the context of your product or your environment, and that's why you need to be using your model evals just as a filter in order to make choices, but not entirely rely on them when you're building your products. But after you've narrowed down on a set of models that you want to use for your task, you need to run your own product evals, which is you need to test your system in the environment that you're working with. Which brings us to the next section, which is how do you actually build these evals? There are three ways to build evals. Let's go through each one of them. So the first way is the most unscalable way, but the most straightforward way,

3. 7:56 – 10:53

   The three ways to build evals: human, code-based, and LLM judges

   1. ARAishwarya Reganti7:56

      which is human evaluation, right? And it's exactly what it sounds like. A person looks at what the system produced and makes a judgment about whether the response was good. And human judgments are also usually called as ground truth because they help set the bar of what is good. And the other two methods we'll talk about are actually what are built in production, and they're approximations of human judgment so that we can scale it in production because you can't really ask humans to judge just every response in production, right? And that's exactly why we have the other two kinds of evals, the simplest one being code-based evals or code-based checks. Now, these are deterministic rules that you can implement through code, and there are two types of them, and these are the ones that are very similar to unit test, right? The first kind is pass or fail checks. For instance, schema validation or validating a particular JSON, identifying if all the fields you require are present, or things like does the document it cited actually exist? Did it pick the category your label data says it should have picked? And those kind of things, right? Stuff that you can build using code. The answer most of the times is a yes or a no, and it's mostly objective in nature. Now, under code-based checks, you can have a second kind, which are similarity-based evals. So for instance, BLEU score is a traditional metric that looks at how much phrase overlap there is between the text that your AI system generated and the reference text that was expected. And in these kind of checks, you get a number usually between zero to one instead of just a pass or fail check. But still, this is completely deterministic in the sense that the same two pieces of text will always give you the same score. Now, while these code-based checks are useful for measuring objective criteria, they don't really understand language or nuance. And remember that for most AI systems, some understanding of context, of language, and subjective understanding is really required in order to make judgment as to whether your AI system is producing the right thing or not, and that is where we go to the next kind of eval, which is LLM judges or AI judges. Now, these are the ones that are used for more subjective evaluations. Now, an AI judge is just an AI model that is given the task of judging the output of another AI model or a system by writing down a set of rubrics or criteria and having the AI apply them in order to provide a score. And the way you'd write down these rubrics is to think of the criteria that humans would use in order to judge a piece of text or a piece of output that is generated by an AI system. Let's maybe take an example, right? So let's say you wanna check whether a support system that you built replies in a polite way to your customers. A code-based check can look for a ban word or count how many exclamation marks there were in the response. What it can't do is tell you that, "I've refunded it," reads as curt when someone is already upset and reads perfectly fine when they aren't upset. It's the same five words, right? But depending on the context, depending on how frustrated the user is, you need to understand whether your agent gave the right response and it was polite, and these are the kind of scenarios where you'd need AI judges or LLM-based judges. And I know

4. 10:53 – 13:31

   Live walkthrough: building your first eval in Arize

   1. ARAishwarya Reganti10:53

      that all of this is pretty new if you're a beginner and can be kind of confusing to understand what AI judges are. In order to solidify this idea and make sure that you can wrap your head around it, let's go ahead and build our first AI judge eval so that you see how the process works. Let's say we're building a customer support agent for an online shopping app. I'm gonna call it QuickCart just for fun. Now, let's say people keep messaging this agent about their orders, their accounts, their payments, and all the things you would have messaged a support chat yourself, right, at some point. Now, you wanna understand as the builder of this app if it's doing a good job and if it's ready to go into production. And so you wanna start building evals around it, right? Because that's what the best builders do. I'm gonna walk you through five steps in order to do this. Now, the very first step to building evals is for you to put together a small reference data set. A reference data set is essentially something that looks like this. So I'm using an app called Arize, which is an observability tool, and it makes it much more easier to build evals. You can either use this or use a tool of your choice as long as you understand the concepts very well. I have a small reference data set that I've built just for the purpose of this demo. Let me quickly walk you through that. I'm gonna click on Data Sets and upload my reference data set, and I will walk you through how the reference data set looks like. Okay, you'll see that this is how my reference data set looks like. A reference data set at its simplest should at least have three columns, which is what was your user's request, what did the AI answer for that user request. So this is essentially if you built a prototype for this agent, you can run the prototype and get the answers that is... your AI is generating. And the third column is a reference answer, which is what is the right answer that it should have generated, and you need this so that you know what kind of evals you have to build, right? So let's go on Create data set here. Now, how do you come up with a reference data set is essentially you try to understand your product deeply and identify and anticipate what kind of queries your users might come up with. Let's take this example, right? You'll see that the user request says, "I was charged twice for the same order." The reference answer says, "I'm escalating this to our billing team." That's, that was the action that was supposed to happen for the agent. But it looks like my AI system just says, "You can request a refund from your orders page." That means it's giving out the wrong answer. It looks like escalation is not working well here. Let's go and see a few more data points. Here the customer says, "Someone ch- changed my account email and it wasn't me." And the reference answer should be, "It should be escalated to a human," but the AI says something like, "Try resetting your password and updating your email," right? Looks like from the data patterns that I'm seeing, one of the common issues that my AI agent is having is it's not able to escalate as required, which means

5. 13:31 – 15:30

   Finding the real problem through error analysis

   1. ARAishwarya Reganti13:31

      that I need to be evaluating on this particular dimension. And the way I can do it on Arize is just go to my data set and click on Add Evaluator, and I can create one from scratch and choose LLM as a judge. And I can just call this escalation check, and I can choose any model of choice. Um, let's just do a GPT 5.5. And I'm gonna type a bunch of rubrics and criteria here depending on my use case, and I'll walk you through why I wrote that. All right. So this is kind of what it says. It says you're checking whether a support assistant made the right call about handing a conversation to a human. There's a customer request here, there's the assistant answer here. And I also put a bunch of rubrics, which are essentially what a human would have judged for each of these, right? And I say, "These are the criteria where escalation should have been done. If under this criteria my AI system is not doing it, flag that for me," because that's kind of what my eval is checking for. Then it says, "If the customer is angry, you need to be escalating it. If they're swearing, if it involves fraud, security, and all of that," right? Now, how do I get all of this criteria is because of my domain expertise and because of how well I understand the product itself. You know, say whether it's correct or not. I say answer incorrect if the assistant failed, and answer correct if it's right, right? And then I choose two classes for this, correct and incorrect, so that the AI judge can give me one of these responses. After I set up my evaluator, all I need to do is just run evaluator, which runs my evaluator on all of these use cases. Now you can see that the evaluator has run, and you can see the prediction that it's made for each of the escalation. So you already, for the rest of the data set, you know that the escalation decision was correct or incorrect. So that's your first LLM judge eval, right? You wanna go through all of these data points that you have and make sure

6. 15:30 – 17:43

   Writing rubrics and running your first LLM judge

   1. ARAishwarya Reganti15:30

      that you improve your LLM judge and build on top of it. Now, this is the simplest way to build an eval, but as the architectures get more complex, eval building as a process also gets more sophisticated, but you can kind of see how easy it is to build your very first eval. So we pretty much depicted five steps in order to run your eval, which is you generate a small reference data set, you run error analysis on it, which is identifying where your AI system is making errors, and then you decide what is the eval that you're going to be building. In my use case, we built an escalation eval which identified whether the escalation decision made by my agent was correct. And then we wrote a bunch of rubrics, which is essentially how would a human make this decision? What criteria would they use? And finally, we ran the judge, right? Now, that's pretty much the five steps that go into building your first eval. So instead of a human going and reviewing each of these requests, you can now hand the criteria to an AI judge that can scale the judgment of the human, which is pretty much what we did, right? But remember that the judge itself might need more calibration, and you have to make sure that what you've built is actually replacing human judgment without it introducing more errors, right? Now, remember that this is only the first half of building evals. The second half actually happens in the calibration phase, which is when you start seeing real users. Your data set increases from just being a reference data set to a production data set, so you might have to continuously calibrate the system, which is why initially we spoke about this idea of evals being a loop and not really a build once and forget kind of a system. So that's pretty much it for this video. Hope you really had fun, and you also understood how simple it is to build your very first eval. I want you to think of a use case that you've been trying to build. Run your agent, get responses, build your first reference data set, and then build your eval, right? You don't have to do it with the same tools that I did. You can build your own AI judge using any API that you want. And in this video, we pretty much scratched the surface. If you wanna go much deeper on this concept, there's a lot more to understand in evals. We have an entire free course on GitHub that you can take and also get certified. It comes with 10 chapters and also hands-on components. I highly recommend it. There are about fifteen thousand people who have taken the course and gotten certified. Do check it out, and hopefully you have fun. All the very best.

Episode duration: 17:44