Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

How I AIHow I AI

Token fraud is here and it’s BAD. Here’s what you need to know.

Today I’m walking you through a problem that has cost me tens of thousands of dollars over the past year and eaten almost all of my engineering energy: token fraud.

Claire Vohost
Nov 16, 202639mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

Transcript

0:00 – 5:25

Token fraud and what it cost me

Claire Vo 0:00 If you think AI slop is bad, let me introduce you to my new, very unwelcome best friend, token fraud. In addition to being a lovely AI podcast host, I am the solo founder of ChatPRD, an AI platform for product managers that I've been running as a labor of love and a beautiful little SaaS product since, I think, the early GPT-3.5 days. And I have to tell you, the last year I have been spending more and more of my time, and almost all of our engineering energy, focused on blocking one thing: token fraud. We are gonna talk today all about it, what it is, how to spot it, how you can protect your business, and what you should be doing across technology, agents, and your operations to do your best to protect yourself from folks that are trying to steal your inference. Let's get to it. This episode is brought to you by WorkOS. AI has already changed how we work. Tools are helping teams write better code, analyze customer data, and even handle support tickets automatically. But there's a catch. These tools only work well when they have deep access to company systems. Your copilot needs to see your entire code base. Your chatbot needs to search across internal docs. And for enterprise buyers, that raises serious security concerns. That's why these apps face intense IT scrutiny from day one. To pass, they need secure authentication, access controls, audit logs, the whole suite of enterprise features. Building all that from scratch, it's a massive lift. That's where WorkOS comes in. WorkOS gives you drop-in APIs for enterprise features so your app can become enterprise-ready and scale up market faster. Think of it like Stripe for enterprise features. OpenAI, Perplexity, and Cursor are already using WorkOS to move faster and meet enterprise demands. Join them and hundreds of other industry leaders at workos.com. Start building today. Okay. Before I get into the full episode, let's just take a minute and establish what I mean by token fraud. So ChatPRD is a product, some would lovingly call it a AI wrapper, but I think we do a lot more than that. But part of our product is built on these LLMs and APIs that access the intelligence in LLMs sold to us at a token level by companies like Anthropic and OpenAI. So when you use a product like ChatPRD, some part of that product is making a call to an LLM and returning the result. Now, we at ChatPRD do not do usage-based pricing. We do seats-based pricing, and we have built our pricing based on average use and cost and routing of models, and it works out generally across our customers and the margin for our business. But if you are not very careful, people who are not your customers can abuse things like free trials, promotion codes, early access to really use those API calls to the LLMs for whatever purpose they want, and in really bad and honestly quite creative instances, really rack up your AI bill. This has cost me tens of thousands of dollars over, I would say, the past year. It has been incredibly painful. In fact, in this X post that I posted about a couple days ago, I said it really made me rethink running the business. It is not fun. I am not venture funded, y'all. This is a bootstrapped, solo-run business, and so every time somebody tries to scam me or take something from me outside of the bounds of our terms of service and outside of the bounds of our business, it is a real bummer, and it comes directly from my pocket. And I shared this on X, and it resonated across the timeline. I felt so much better, honestly, because some of the biggest companies and the best startups with some of the most awesome engineers fell into my replies, fell into my DMs, and said, "Oh, my God, Claire, we are dealing with this every day, too." In fact, I have a friend [lips smack] who's a neighbor and runs an amazing company, and he and I were chatting about it, and he said, "Look, I'm about to turn off signups. It's so bad." The abuse is rampant, and the misuse of this compute and the misuse of these LLMs is really hurting businesses big and small. And I will tell you, mine is a small business, and it is hurting me quite badly. But we're gonna take these fraudulent lemons and turn it into sweet, sweet educational lemonade. And I'm gonna tell you all about token fraud, how we identify it, what you can do, and hopefully you will leave this episode, um, with a couple more tools in your tool belt and a couple more ideas that you can take into your company and think about when you're trying to roll out AI products and avoid the abuse that I have unfortunately experienced and a lot of big companies have unfortunately experienced and are-- continue to battle on a regular basis. So

5:25 – 11:24

Six abuse patterns

Claire Vo 5:25 let's get to it. All right. So let's talk about a couple ways I see people abuse AI products. There's six ways that I have currently seen people abuse, um, our AI product, and I think these are pretty common across the ecosystem. So first is free tier or trial tier farming. Essentially, a lot of companies, mine included, and very often VC f- funded companies, offer some sort of free access to their users. And it's relatively generous free access because you want people to see the value out of your product and ultimately convert. It's that beautiful PLG motion. And what we see is a lot of customers, now I'm gonna call them fraudsters, a lot of fraudsters signing up for free tiers and then really using the most expensive models to do non kinda, like, product align- Work. And so we've seen a lot of automations around free tier farming. This is actually a thing, like the biggest lever we could pull is we just turned off free trials. The problem with this, the thing that sucks so much about this, is every time I do something that protects us from fraud, it has a negative impact on our real business and our real prospects. So free trials were a great source of PLG and enterprise leads for us, and I just had to make the decision it was not worth supporting a free tier because I could not come up with a cost-effective free tier that wouldn't be abused. And it's just like an extreme bummer, but I've heard from so many companies, they have just like simply turned off free trials. The second thing we see is a lot of promo code abuse. So again, ChatPRD is like my labor of love. I do it mostly because I think it's an awesome product. We've had, you know, like a million users. We have enterprise customers. We love this, but I'm not VC funded and I, you know, wanna get it into the hands of people that need, need it and can use it. And so we've offered through partners a lot of promo codes, and then what we see is so much fraud around promo cu- codes, reselling of promo codes, stealing of promo codes, mass use of promo codes for like spinning up other AI businesses. Promo codes have been such an unfortunate attack vector for us and our business, and again, I'm never gonna do these again. Like, I love you all. I wish I could give you and the partners and people that have supported us along the way free access to my product. I want to support you for a year because I think you would see amazing value out of it, and ultimately, like many of our customers, convert. But I just can't afford it because half of you sell them to Reddit resellers. Like it's, it really stinks, and so promo codes have become a huge point of abuse. Lots of bad card stuff. So we can talk a little bit about how to protect and, um, monitor for bad credit cards, but you're gonna see like a lot of bad cards, prepaid cards, card chargebacks. It's just like a total nightmare. But this is one that like luckily I had a little bit more around because I was just like very prepared for this, having done a lot of online business before, and there's a lot more controls about this. So this was almost like the least part of the problem. Resell and relays, the just crazy ways people will resell the AI and LLMs that you're using for your business. So like one wild example, I don't even like really understand why they were doing this, is they were like authenticated as ChatPRD in a web browser, and then would like take requests from their AI product that they were selling and like send them to ChatPRD and then send them back. And we already had technical controls around these requests, but it was really bizarre to see like the relays that were being put on top of these products, like hyper inefficient, probably not performant, but very, very strange and a lot of like reverse proxies here. Using and abusing MCPs to bypass like message level controls, and so, um, this one was like fascinating. I just like wasn't expecting it. But somebody built like a monster MCP and then like installed that as a custom MCP on ChatPRD because we let you install custom MCPs because we want you to have your business data connected. And then it became this like 24/7 worker that was prompting and saying like, "I'm never sleeping." It very, very strange. And then, you know, I s- I don't know at this distillation, I don't know what this is, but we had a lot of like model probing from international accounts. And so like maybe it was farming, maybe it was something else, but we had a lot of like, "What model are you? What's your latest like training cutoff date?" kind of probing. Um, we even had Simon's like pelican writer. Simon, if that was you making SVGs on ChatPRDs, and I'm very mad, but I don't think it was. Um, but we had like pelicans riding bikes, all sorts of stuff. Really strange. And it's just really challenging, right? Because as ChatPRD, as a enterprise business, this is a business, we are, we don't retain, we don't look at your data, we don't do any of this. And so it's like very hard to parse through the signal and the noise to figure out what of this is fraud, what of it is use. And only when you have extreme examples can you then go and investigate, and often, um, they're, these users are like verily, very sophisticated, and so they're using a combination of these techniques to stay under your overall threshold. Very, very interesting, but

11:24 – 13:45

Four kinds of abusers

Claire Vo 11:24 all of this stuff is happening. Let me just tell you a couple stories of, of token abuse that, that [chuckles] that we saw. Um, and I have to laugh because if I don't laugh, I cry. Um, but I am funding a international Roblox and Fortnite map studio, and so they'll like run a loop of free trials and promo codes and then tunnel to my agent and use it to build Roblox maps, I guess, Fortnite maps. So really kind of like obnoxious, um, use and abuse of coding capacity, which we also allow coding because we do prototypes and stuff in ChatPRD. So that one was really interesting. We have a reseller that wrapped our app in a proxy and then resold it, um, to their own customers by the hour. It was, that one was really interesting. Um, we see a lot of like sign-up loops. Those are fairly easy to catch, so people that are signing up, like first name, last name, one, two, three, four, five, first name, last name, one, two, three, four, six. So we see a lot of like sign-up farms. And then a lot of just, like, general trial abusers. As I say, people have been using ChatPRD as a, like a Temu ChatGPT, and it just sucks. Like, our free trial is not for you to, like, write your essays, um, or do work reports. It's for product managers to do product managery things. So, you know, it's just, like, very interesting. We get a lot of abusers. I would say the worst ones are the top two. The worst ones are ones that are commercial entities. Um, I'm calling them com- uh, the criminal enterprises, um, that are, like, really very nefariously trying to take advantage of our business. In trial abuse, we see a lot of this from countries where the cost of AI really outstrips the, the s- the, you know, average income. And so, like, these are a lot more sympathetic, too. These are just kinda like regular people trying to get access to this AI. Unfortunately, Claire Vo cannot fund these, um, [chuckles] these, uh, uses of AI, just simply, like, as a business, don't have the capacity. As an individual solo founder bootstrapped, I do not have the capacity. But those bother me the least because they feel like the least nefarious. Okay. So

13:45 – 21:27

Business controls

Claire Vo 13:45 I want to sa- thank Gareth at Atlassian, who messaged me and was like, "Yeah, we see it too." And Atlassian's a big company. So at first I was like, am I an idiot? Like, am I... Uh, you all, I'm, I'm a real grown-up software engineer. I've been doing this for 20 years. I have run many startups. I've been executive at many companies. And I just, like, did not, I did not think of the creative and honestly mean and horrible ways people would try to steal. I was just too naive, and I thought I had all these protections. It's not like this app is not secure. I've been pen tested multiple times, SOC 2 compliant. I run regular cone scans, regular vulnerability scans. And so it was really [chuckles] honestly nice to hear from my friends in big companies and my friends in, like, very, very technical startups say, "We deal with this all the time as well." And Gareth at Atlassian posted this amazing list of things that they did. We do a lot of them. And a lot of them are just business things that you have to do that unfortunately stink. So some of these are his. We'll link to his tweet, um, that he posted. Super useful. But I kinda like summarize some things you can do in your business that you might have to do if you're selling AI products. So the first thing is, I think free trial's over. I'm calling it right now. It's just not worth it. Man, I am gonna shut the door. You're gonna talk to sales. Um, if you got a PLG account with ChatPRD, that is a vintage account never to be seen again. Trials are over. I'm done. We had trials require a credit card, and they would credit card cycle, use bad credit cards, all, you know, do chargebacks. And so, like, actually credit card backed trials and even paid trials don't work. I just think trials are over. So, uh, you might have to shut the door on trials, and I'm hearing that a lot from, um, friends in the industry in both enterprise and PLG. I'm treating promo codes-- I'm getting rid of promo codes, but you have to be really paranoid about promo codes. Gareth said, uh, you have to bill sooner and be completely inflexible on billing terms, so you have to charge up front, no grace period on failed payments. Nothing, it's like pay to play, baby, because otherwise it's just open for abuse. You need hard limits on accounts and organizations, and unfortunately, this was not something that we needed last year. We had, like, a really high trust relationship with our customers, and we had very high caps because occasionally they would do stuff that we thought was interesting and we wanted to learn from their AI usage, and we didn't wanna over manage them. And now we're like have a really, really hard cap on usage, um, per accounts per org. And it's harder for us because, again, as I said, I'm bootstrapped. I'm not VC funded. I can't actually afford to lose money on every account like some other companies could. And so it's a different kind of, like, economics for me to build a, uh, margin accretive, cash flow positive business here. Um, but yeah, hard limits you definitely have to do. Restricting trials to business domains, so, um, email domain management is a technical part of this that I will talk about. Know how to turn things on and off. I use a lot of feature flags, so we can just, like, turn off, uh, trials if we see something strange. Our terms of service allow us to ban accounts for abuse and suspected abuse of the service, so make sure your terms of service get you a lot of controls. I, number eight, I am, and I'm sorry about this, but I am being really thoughtful about whether or not we'll do international business. We've had some experiences at ChatPRD where we have turned off whole countries via firewall and seen fraud drop by a significant amount. And so I do think this really will impact at least small businesses' international strategy, and I'm really rethinking if we move to kinda like trusted country only based on user behavior and fraud signals. It might just be a blunt instrument that's the most effective for us. And then I found out that you can report cybercrime to the FBI, um, and you know, I don't know if people are doing this, but I did it because, uh, there were a couple, um, what seemed like coordinated international enterprises specifically trying to steal and defraud and bypass, um, security mechanisms. And so, um, I submitted cybercrime reports. Codex helped me author those. And I also posted publicly, and I think this is one of the benefits of being a solo founder. I can do what I want. You know, I think a lot of Venture funded companies, public companies, larger companies cannot come out and say, "We are being straight up freaking defrauded by these token scammers," because it could expose a lot of risk. It could attract more scammers. Uh, most people don't like to talk about the bad things happening in their business. But you know what? I can. And I actually, in digging in some of the abusive chats and, and finding what they were doing, I found that they were running reverse, similar reverse proxies on several companies, several companies that you have heard of. Think of like the top 10 AI products. The people that were scamming little Claire were also ro- rotating these proxies and these API calls, and they're, um, reselling or they're coding loops through these other tools. I could tell because of the way they prompted my tool and, and discussed routing to these other proxies. And so we gotta talk about this openly. I don't know what this is. Startup friends, if you have seen this, I will start a group chat. Um, we can keep it in the house, but we've got to start talking to each other because this is happening. And my husband made a joke. He was like, "How many VC funded startups do you think are presenting all this token fraud as actual growth?" 'Cause, [laughs] because they can, they can afford, uh, to, to lose a lot more than you can lose. Um, but I do think we need to start talking a lot more about these issues, sharing patterns, um, with other founders, and then hopefully we can lock arms and take control of this. Have you ever asked your favorite AI partner whether what you're building is good or safe or trustworthy? Did you believe what it told you? Auros connects your team to a network of over 7.6 million verified human experts, the specialists who bring the taste, judgment, and nuance your evals can't capture, so you can be sure that what you're building is actually good and good for people. Auros, the human network behind AI. Get started today at aurosglobal.com/howiai. Okay. So that's what it is. That's all we've seen. These are some of the things you can do for business. And you know what sucks about this is all these business things have a negative business cost, right? Like, if I stop doing business in country A, sorry to great, amazing users, product leaders, engineers in that country that wanna use my product. You can't use it. That stinks, but the upside is not worth it. And so turning off the trial, it's a pipeline for us. It's pipeline for sales. We have to rethink our go to market. And so again, it's a real bummer, but let's get to what

21:27 – 30:25

Technical defenses

Claire Vo 21:27 you can do about it. I'm gonna talk about all the code things that you can do or some of the code things you can do. I'm gonna talk about all the kind of like agent and AI things you can do and that I'm doing. And I just wanna pause and thank everybody who reached out to me, contributed ideas, and just offered support. I mean, the entire Stripe team has been amazing. Thank you, Jeff, um, for getting me in a Slack channel and just helping me out with some like advanced radar stuff. Michael at WorkOS offered to help. They're sponsoring the podcast today, um, just by chance. Thank you. Ben at Raindrop offered to help. Gareth posted from Atlassian. Rick at Persona, they've sponsored the podcast before, reached out and said, "What can I do?" I mean, there's just been so many people that have reached out and said, "There's stuff we can do," and also, you know, "Here are some things that we've tried and helped." So it just... I'm so grateful to everybody that's contributed to this. But you know who I'm really grateful to? I'm really grateful to OpenAI and Anthropic because without these AI tools, I would really be SOL. But, [laughs] um, because I have these AI tools, a lot of these controls were already in place and much easier to manage via AI. I'm just gonna talk about what they are. Um, none of these necessarily require you to buy a new product. I'll call out ones where I really love the provider that I'm using. But there's a lot of code controls you can do. Unfortunately, all these controls and AI, um, engineering capacity are available to our fraudsters, so we're gonna end up in, um, a stalemate at some point. But for now, this is what I do. So what are the technical things that I think you really need to do? Well, you need an AI gateway and controls on spend. And you can kind of do this yourself. You can build this yourself and route your AI calls to your providers yourself. We like to use the Vercel AI Gateway. I like the AI Gateway not just because it allows me to control spend, but because it allows you to control what models are available. And so one of the recommendations that I saw was that, like, you should only allow the models that you allow. And so if you just allow calls to Anthropic's messages endpoint, somebody could be like, "Well, I want Opus," or, "I want Fable 5.2 instead of Sonnet," that you wanna give them. And so making sure that in your code you only have a certain allowed set of models and do not allow calls outside that model set is really important. I'm also gonna talk about how you can implement a penalty model where you can downgrade, um, certain fraudulent or suspected fraudulent calls into a less expensive model. But definitely an AI gateway, definitely alerts and limits on spend aligned with what your business is. So spend. If you're a business of, of, of real reputation, I would call ChatPRD one of those, you gotta run spend hourly at least or live, and I think this was my big mistake. I just didn't realize how quickly, um, how quickly Something could be really exploited and costs would pile up. And so having, um, spend caps and having spend monitors is very important, and making sure that you have alerts and failovers and automations around this is very important. Big shout out to Stripe. I use Stripe Radar like it is my job. It is their fraud product. Um, I actually... [chuckles] Stripe Radar has a lot of out of the box fraud rules that you can just [clicking] turn on and it will actually tell you how many, um, payments would be blocked, how many real payments would've gotten through, all those sorts of things. I actually started using Stripe Radar quite hard, not because of token fraud, but because of card testing. I woke up one day and saw that, like, $200,000 had been, um, attempted purchases on ChatPRD. I wish, I wish we were doing $200,000 a day. Um, maybe next year, but that was a little, a little surprise. And I looked in and they were all failed requests because they were all scam kind of cards and stolen cards and they were just doing card testing. So I was using Stripe Radar for card testing, but you can also use Stripe Radar for blocking certain countries, making sure that a country's, um, a card's address matches its location, all sorts of things. Here is my tip and trick with, uh, Stripe Radar. It is, I think, easier to understand in the Stripe dashboard, although, um, you know Stripe, they make great APIs. So I use Codex to drive Stripe Radar to implement these, like, complex rules, right? These complex rules and implement them so then I can review them in Stripe and turn them on. Um, and then one other, uh, Stripe Radar style rule that I've done is start to fingerprint cards and look for multiple use of cards in multiple trials. It wasn't something I was monitoring for. And then you can see, like, multiple trials use the same credit card. And so that was one that w- took a combination of code and Stripe Radar, but I use a lot of Stripe Radar. Thanks to the Vercel team for reaching out about Bot ID. Um, Bot ID ensures that there's proof that the request came from your app, your users with, um... So even if they have credentials, it proves that they're using your app. Um, this has to be put on the actual route, and so this has to be on, like, every chat request. But you can run it for a little bit, see what, uh, your user profiles look like, tune it, and then ultimately block things. And then you can turn off enforcement and turn off monitoring for, for users that look good. But we've implemented Bot ID and this can be really helpful just to keep from some of those more technically adept scammers. Firewall, again, word to put on Vercel. I, like I run, um, Stripe Radar like it's my job, I run Vercel Firewall like it's my job. And so, so much of our fraud unfortunately comes from non-US countries and I have just found that the ban hammer is the only tool that's effective. I try to be generous, I try to be useful. So we do a lot of country and VPN level just total blocking. And you know what? It makes my life happy. Again, there are just so many benefits to being a solo founder with no board and nobody breathing down your neck at, about growth at all costs because you can just do stuff like, "I don't wanna do business in these countries," and turn off that country. And so this is, um, I use Firewall a lot. Um, I do a lot of IP blocks and a lot of rate limiting as well. So using your hosting provider's tools to control who can even access your app is another very useful tool in the fight against token fraud. Uh, you can also use the out of the box tools in your auth provider. The Clerk team has been really, really kind, um, to help us tune our setup, um, on our fraud controls, but you can turn on bot protection and often you have a lot more controls at the auth provider level. Typically, these are turned off by default, so go into your auth provider settings, toggle them all on, and then message them and ask them for help and they, they will help you. Again, as I said, Michael at WorkOS, also super, super helpful sponsor of this episode and has a lot of controls around this too, especially around token fraud itself. And then finally, you need to start monitoring not just your app, but your users and the MCPs that they use. And so, um, again, what we saw is people use ChatPRD, hook it up to an MCP, uh, that ultimately used a lot of ChatPRD compute for external tool calling and all sorts of crazy stuff. And so I try to be so generous here. Again, it's just not a thing I thought people would do with little old ChatPRD. Uh, I was not expecting it, and so we've had to be a little bit more constrained on what MCP hosts we support. And unfortunately, we're going, like, full Figma and not allow- [chuckles] not allowing anybody, but especially not allowing free tunnels and hosts for MCP servers that don't seem like they come from real companies with real concerns. So that is, um, another kind of, like, fun find in our fight against token fraud is looking for tunnels in custom MCP hosts and

30:25 – 35:30

Bots that watch the bots

Claire Vo 30:25 not allowing it. Okay, so those are, those are technical things you can do. All this code is example code, but just go ahead and ask your favorite, uh, coding agent how to implement these things and that can help. Let's talk about AI bots. So, you know, part of this is technical, like can I put these technical guardrails around our app to constrain or protect against abuse? And the other thing is, like, what are operational processes that I can monitor and run quickly against these scammers? And, um, there are four AI sort of like agent style loops that I use, um, both in ops and in the app to manage some of this. My favorite one, you knew if I don't mention Jev every three episodes, I get my AI podcaster hat taken away. But Jev is really saving us here. So one of the things that has been so brilliant about these new decision models, Jev, OpenAI's decision model, I think Google just came out one literally today, is that it can be really effective and really cheap to just look at stuff and categorize it. And so now when we start, when a chat flags our kind of like its metadata flags our abuse threshold, then we run a ZDR Jev loop on it and categorize it into do we suspect it's a real customer, trial abuse, promo abuse, co- card fraud, resale, agent farming, et cetera, et cetera, whatever your categories of fraud are. And then if it's categories of fraud, depending on its tier, we ban them or we silent downgrade them to like a GLM. Um, so this has been super helpful, kind of like Jev in the loop for fraud classification, so highly recommend that. I also run an hourly agent to look at recent signups, out of bounds, crazy looking like chat threads to see if there's any signals of abuse. If the abuse signals are high, we auto ban, um, the ban hammer goes down. If not, it pings me for feedback or sends it to our AI support desk. Or our support is generally run by AI, although yes, a lot of those emails are from me. So for, if you get an email and it sounds like me and it's signed by me, it's probably from me, but we do have, um, AI running a lot of support, and it's really important to give that AI agent toolkits to deal with fraud. And so we've given it tools to deal with fraud, fraud lists, um, allow lists, and playbooks and responses for people. When you ban people, you will get two, two types of emails in response. So we banned a, a bunch of people this week, and we get, you get two responses. You get garbled, poorly written, bad emails, like, like random, like ZN27843@outlook.com. Like you get these like fake email, fake name, poorly written, misspelled emails that say, "Why did you ban me? Give me access." Okay? [chuckles] So you get that and you're like, "I banned you because you are abusing our system." Or you get these very polite, very, um, confident, "Why did you ban me?" And I love ChatPRD, which actually many of them do. The polite ones at least come from people that are using ChatPRD well. And they say, "Why did you ban me? What was going wrong? Here's all my evidence of why I should get access to your tool." And then you unfortunately have to write them back and say, "That's a fake promo code and a stolen account, and unfortunately we have to turn you off." And so you do need to give your AI support desk tools to respond to and deal with fraud, and then more importantly, not allow it to unban, ban, provide promo codes, all that kind of stuff because people will try to use your AI support desk as a vector of attack. And then, um, finally you need to take this whole system and take all the signals for fraud and build them back into your system. And so every time, you know, fool me once, shame on you. Fool me twice... I'm from Texas, I wanna say never be, never be fooled again, but fool me twice, shame on me. If something happens, build it into your rules, build it into your ops, build it into your agents, and account for it the next time. You know, I think the really sad part and the reason why I said I was just like really bummed and might not run ChatPRD anymore at some point is I feel like this is 80, 90% of our job and it's just no fun. It's just no fun. And every s- real suggestion people give me about like how I could very easily stop fraud is the exact things that would be bad for our business. But again, you can build a virtuous cycle, you can build kind of things that feed into your code, feed into your agents automatically, and we built a lot of these loops in our business hopefully to make it a little bit more resilient against fraud. And I think a lot of this applies to, um, cybersecurity and

35:30 – 39:44

Final takeaways

Claire Vo 35:30 gaps as well. So in conclusion, if you sell AI, you sell compute, so act like it. [chuckles] So act like it. This was, this was a good headline, uh, Opus, Opus 5.5. Uh, this is a good one. But basically like you gotta watch your model bill, you gotta count dollars not requests because expensive models get expensive. Every zero dollar door gets used, every trial, every coupon code. There is no free lunch anymore. It's not like seats. It's not like when your hosting would go up a little. It is so expensive, and so I think zill- zero dollar doors have to be closed. If you sell cheap models, people, people don't want them. A funny test is we turned off Opus 5.5 in a couple countries, and guess what? No one stole tokens anymore. So people really want Opus 5.5 if I've learned anything. Your own AI will help them if you let it. So, um, because again, we're a product product and we let you build and spec out products, people were legit writing PRDs about ways they could scam ChatPRD. I am not kidding. It was wild. They used my own creation against me. Fraudsters need PRDs too, but they will, uh, they will use your AI to help them steal your AI for sure. You have to verify your client, not just the login, so, um, valid sessions and bot ID in particular. False positives happen. I banned one false positive in a big sweep of, of bans. I'm sorry, but we fixed it, and you have to keep the rules simple and the policies strong, and there's no wiggle room. So this is the conclusion. Again, I didn't write this, but I'll take it. Opus, start from I don't trust you, and then users can earn it back. What a customer forward way of thinking about, about products. Now, I mean, to be really honest, this has been a huge and very painful, and as I said, extremely expensive learning for me as a founder. But I, I'm here for you. Look, I am willing to try AI things I don't know. I'm willing to build stuff that I have no business building. And I'm gonna tell you when stuff is hard as a solo founder. I'm gonna be honest with you. And while I have over 20 years of experience in technology, I've run some of the largest teams powering huge parts of the internet, incredibly technical work. ChatPRD, as I said, is like a real business. It's a real product. I know what I'm doing, and it is hard. And again, I was just heartened by so many people in large companies, small companies, startups, technical people, you know, telling me that they're really dealing with this too, and it's exceptionally hard. So, you know, go hug your friendly neighborhood AI founder. We are not having a good time right now. Um, but, you know, fingers crossed the good guys win, the good gals win, and we can figure out a way to build amazing products that deliver awesome experiences and help you and help your business or help your life, that also don't bankrupt the, the creators and go to the people that really deserve it. So that is a very special How I AI on token fraud. I hope it was interesting to you. Everybody out there, give ChatPRD a break, man. I'm just trying to send some kids to college. So, um, be gentle to me. We will be out there fighting hand-to-hand, uh, combat, but hopefully this is helpful for you. Hopefully you learn something, and go sit down with your coding agent and let's lock some stuff up. Thanks for joining. Thanks so much for watching. If you enjoyed the show, please like and subscribe here on YouTube, or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiai pod.com. See you next time.

Episode duration: 39:46

In the uListen app, the transcript lights up word by word as you listen. Get uListen

Transcript of episode ZDpP7JXPlhc

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.