Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody

(instrumental music) The wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities.

We're entering the era of evals.

We started working with all of the top AI labs. What the labs need is a labor marketplace. They actually need extraordinary professionals that can measure model capabilities.

They found this pocket, maybe the biggest business opportunity in history.

We grew from one to 400 million in revenue run rate in 16 months. Fastest ascent in history.

Why is this so valuable?

The market is bound by the amount of things where humans can do something that models can't. The lab's primary bottleneck to improved models is how they can effectively have some way of measuring what success looks like for the model.

There's this tweet that you retweeted, "If you really think about it, we were put on Earth to create reinforcement learning training data for labs."

It's highly likely that the entire economy will become an RL environment machine, building out all of these worlds and contexts. And I think the narrative in AI over the last three years has almost entirely been one of job displacement, but very few companies and people have talked about this new category of jobs that's being created.

I talked to a lot of people about, "What should I be studying? Where should I be getting better?"

How can they leverage this technology to do so much more? We'll give people interviews where we say, "Use whatever tools are available to build a website, and let's see what product you're able to build in an hour."

Today my guest is Brendan Foudy, CEO and co-founder of Merkle. Merkle is the fastest-growing company in history to go from one to $500 million in revenue. They did this in 17 months, less than a year and a half. Brendan is also the youngest unicorn founder ever. They just raised $100 million at $2 billion valuation. Merkle, if you haven't heard of them, helps AI labs and AI companies hire experts to help them train their models using AI. They've never had a customer churn. Their net retention is over 1,600%, and they're on a nine-figure revenue run rate. In our conversation, we talk about the increasing value and importance of evals, the landscape of AI training companies like Merkle and why they've become so important and valuable, how Brendan discovered this opportunity, his insights on what product market fit looks like, the core tenets he's instilled within his organization that have allowed him to build the fastest-growing company in history, what people writing evals for labs are actually doing day-to-day, which skills and jobs are gonna last the longest with the rise of AI, why he doesn't think we'll see AGI or super intelligence anytime soon, and so much more. This episode is incredible. You need to hear this. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. It helps tremendously. Also, if you become an annual subscriber of my newsletter, you get 15 incredible products for free for one year, including Lovable, Replit, Bolt, Innate and Linear, Superhuman, Descript, Whisperflow, Gamma, Perplexity, Warp, Granola, Magic Patterns, Raycast, ChatBRD, and Mobbing. Check it out at lennysnewsletter.com and click Product Pass. With that, I bring you Brendan Foudy. This episode is brought to you by WorkOS. If you're building a SaaS app, at some point your customers will start asking for enterprise features like SAML authentication and SCIM provisioning. That's where WorkOS comes in, making it fast and painless to add enterprise features to your app. Their APIs are easy to understand so that you can ship quickly and get back to building other features. Today, hundreds of companies are already powered by WorkOS, including ones you probably know, like Vercel, Webflow, and Loom. WorkOS also recently acquired Warrant, the fine-grained authorization service. Warrant's product is based on a groundbreaking authorization system called Zanzibar, which was originally designed for Google to power Google Docs and YouTube. This enables fast authorization checks at enormous scale while maintaining a flexible model that can be adapted to even the most complex use cases. If you're currently looking to build role-based access control or other enterprise features like single sign-on, SCIM, or user management, you should consider WorkOS. It's a drop-in replacement for Auth0 and supports up to one million monthly active users for free. Check it out at workos.com to learn more. That's workos.com. You fell in love with building products for a reason, but sometimes the day-to-day reality is a little different than you imagined. Instead of dreaming up big ideas, talking to customers, and crafting a strategy, you're drowning in spreadsheets and roadmap updates and you're spending your days basically putting out fires. A better way is possible. Introducing Jira Product Discovery, the new prioritization and roadmapping tool built for product teams by Atlassian. With Jira Product Discovery, you can gather all your product ideas and insights in one place and prioritize confidently, finally replacing those endless spreadsheets. Create and share custom product roadmaps with any stakeholder in seconds, and it's all built on Jira, where your engineering teams are already working, so true collaboration is finally possible. Great products are built by great teams, not just engineers. Sales, support, leadership, even Greg from finance. Anyone that you want can contribute ideas, feedback, and insights in Jira Product Discovery for free. No catch. And it's only $10 a month for you. Say goodbye to your spreadsheets and the never-ending alignment efforts. The old way of doing product management is over. Rediscover what's possible with Jira Product Discovery. Try it for free at atlassian.com/lenny. That's atlassian.com/lenny. Brendan, thank you so much for being here. Welcome to the podcast.