The Unfair Way to Build an AI Native Product Team

1. 0:00 – 1:44

   Intro

   1. CZCharles Zedlewski0:00

      It felt like after a while that the new party foul was flooding your coworkers' context windows. Where we all just start launching slop at one another.

   2. NHNecoline Hubner0:08

      What if we could put all these learnings into one place? And so that became this Together product repository.

   3. PSPavneet Singh0:15

      If I had to do this manually, this would have easily occupied half of my day, if not more. There are 19 tickets that were filed in the last two months alone, so this is a pretty heavily requested ask. So I kind of get a pretty good sense of what the customer problem is within like five minutes instead of maybe a full day.

   4. AGAakash Gupta0:31

      Together AI just raised an $800 million funding round, valuing them at $8.3 billion. I got their product team to show you their exact repo they use to automate their work. We've all discovered ChatGPT in 2023, and were horrified at the output and started handwriting. What is a good PRD in the AI era?

   5. PSPavneet Singh0:51

      I come from Amazon. I used to work at Amazon, so we are used to writing like 20-pager PRDs. What replaces the bulk of that is actually a prototype.

   6. CZCharles Zedlewski0:57

      We're all gonna live in a multi-model, multi-harness world. That's essentially the new bar for UX for the kind of products we build. A lot of these things don't get magically better with AI.

   7. AGAakash Gupta1:08

      Where does the line of product manager end and developer begin? Before we get into today's show, please take a second to check that you're subscribed on YouTube and following on Apple and Spotify podcasts. If you want access to all of my favorite AI tools, I've gotten them to give you an entire year of their paid plans. Check out bundle.aakashg.com for an entire year of Bolt.new, Airtable, Speechify, Descript, Magic Patterns, Linear, Dovetail, Arize, and Mobbin. And now, into today's show. Charles, so I'm fascinated by how you guys

2. 1:44 – 3:26

   Why individual productivity was the wrong goal

   1. AGAakash Gupta1:44

      work and how you guys have been using AI. What have been some of the big unlocks for your team in productivity?

   2. CZCharles Zedlewski1:50

      Um, you know, Together AI is a platform for powering, uh, AI applications and agents. And so we saw, going back to the rise, back when Cursor was starting to hit its growth spurt, how, um, uh, what an amazing, um, superpower, uh, an AI agent could be for any kind of knowledge worker in, in, in addition to, in addition to and including product people. Um, but it, it made me think a bit about what would a world be like full of product people and engineers that could all generate as much, uh, uh, code and, and, and, and content as they liked. And it wasn't clear to me that the sum of all that was actually necessarily forward progress for us as a startup, and I, I imagine it's probably the same for a lot of other organizations. So what we set out to do as a team is ask ourselves, what would it mean to use AI, but not to use AI to just make ourselves individually productive, where we all just start launching slop at one another, um, but rather AI to make us collectively more productive, that, um, we actual- that, that the, the code and the content and the research that we did actually advanced, um, the company as a whole. And that led us to a bunch of decisions about what we wanted to do centrally and what we wanted to leave people to be free to do individually, and that's largely what brought us to where we are today.

3. 3:26 – 10:41

   Inside the Together product repository

   1. AGAakash Gupta3:26

      So how do you coordinate and centralize all this information?

   2. NHNecoline Hubner3:30

      Hi, I'm Nicolina. I'm a product manager at Together. And, you know, a few months back, we all-- We've been talking about the way that we use AI in the hallway, but we never really sat down and discussed how we actually do our day-to-day work. And when we did, we found that a lot of people had unique workflows that made a lot of sense to the area that they were working in, you know, checking to see the reason behind, uh, a node failure, working with customer support, drafting reports. But then the rest of us were doing, on the day-to-day basis, we were doing things that were largely the same, researching customer needs, creating, uh, PRDs or one-pagers. And so we thought, you know, if we're really being thoughtful about how we're constructing these skills and how we're thinking about agents versus skills and now like, you know, loop engineering, what if we could put all these learnings into one place? And so that became this Together product repository. Um, it's composed mostly of Markdown files, um, some YAML files as well. And what we put in here are both context that is useful for us across the board. So we have this context document or Markdown file, sorry, this context, uh, directory, and it has a number of groups of the, the products, uh, the product grouping that we have, so customer intelligence, sandboxes. But we also have the result of strategy meetings that we have on a regular basis, where we divide up product by mission, our strategy, what we want to accomplish, uh, milestone over milestone, quarter over quarter, so that when we're thinking through our research and we're composing these documents with our agents, we can pull this context as it's relevant. And it's also useful to be able to look at the context from other people's sections. So if you're building something that is touching, uh, or going to create a, a joint workflow, so if I'm creating sandboxes that reinforcement learning users are going to make, I can of course sit down with the product manager from the model shaping team, or I can look at the documents that she's already thoughtfully composed about their entry points and how they use SDKs and how, uh, an ideal integration might look and kind of work out a really decent proposal before I then put it before her and try to do that legwork. I'm taking advantage of the work she has done. On the skills front, it's a mix of things that are, uh, useful for anyone outside of a code repository. So everything that's relevant to specific code really shouldn't live here. But if you want to do something like, uh, figure out if we are, uh, serving all of the generative media models that all of what the users are interested in on, um- Artificial analysis, I can have that skill here. So I could run that as somebody who supports generative media, um, but someone else could as well, and they can see the output of that, um, in the terminal. Or things that we do on a regular basis. So this top line status one is one that I really love. At the end of every sprint, we go through all the projects that we work on, and we create an update of what's been shipped, what's ongoing, what's coming up next. And on this skill, you can just decide who, um, what areas you're going to pull from. So for sandboxes, there's a lot of linear projects, um, but for the SDK and API, uh, that pulls from a lot of projects. Um, so you can pull directly from a repository, see what features got pushed, and then it'll form an update based off of the sources that it was fed. And then you kind of edit it, add additional updates that it missed or context that would be useful for leadership. Um, but that's something that used to take me, I don't know, anywhere from like 10 minutes to 30 minutes, depending on what I'm sleuthing through with the engineers, that now takes me, you know, a handful of minutes just to type in the, um, the prompts and then put the answers into the, um, the right location there. Um, and as you can see, this is all Claude. Uh, originally I think we, for the most part, were, uh, we adopted Claude Code. Um, but a lot of the harnesses that support open source models are, um, are used to looking at Claude Markdown files and being able to run them just as well. And so we've started to move more s- over to using our own models. Uh, one, because they're fantastic, and it's really nice to use our own models, and two, uh, it's, it's more affordable 'cause Claude is... I think li- like many teams are learning across the globe, that Anthropic and all these other models are s- starting to become quite pricey. And so this is an example of using open code. It's, uh, the same repository. This is one of the skills. It's just a, uh, a news report that I run in the morning. It looks at our, um, like our competitors, other, uh, model labs, just to collect information, um, from what happened over the last 24 hours, and I will read it over my morning coffee. It takes a minute, so I ran this earlier, but you can see it's going to read the skill and kind of look through, um, all of the listed, uh, sources that I want it to find news articles for. And then it will create a nice report that I can then click into and, uh, build out, uh, or read through, um, any information that I wanna learn more about that day.

   3. AGAakash Gupta8:57

      So if we return to the context, like who owns and maintains this? Because the worst thing is if you're gonna have like out-of-date context. So is that one... Is that like, let's say Charles sets the overall product strategy. Is it his deliverable to maintain the strategy document that is in a good format for any AI harness? Or how do you guys manage these types of behaviors and responsibilities?

   4. NHNecoline Hubner9:23

      I think for the areas that, you know, the sandboxes or the customer intelligence that I own, it would be my responsibility. And then on the general, it, it has been kind of updated based off of larger events. So if we have a, a strategy or planning session, then someone... We would sit down and decide like who's going to sit down and translate that into what material. So there's typically a lot of documents that have been written in the lead up, and so it would be a matter of just translating that into a Markdown file. Um, but you know, if the, the CEO will write letters or write documents of direction, and that's a really... It, it's a very clean moment. You know when direction is being set, and I think that speaks a lot to how, like, well our leadership communicates direction. Um, but you know. You know that this is a, a pivot. They make it very clear. They kind of, uh, lay it out for you. And so that's kind of your signal. Um, and I think as a product team, it's just, uh, a matter of us... Like I, I tend to, um, update this, I think, quite a bit, but I think if anyone else makes a PR, it's like any other PR. It's like if somebody takes the initiative to create a PR, um, make the, um, the update to the Markdown file, then, um, you know, that's that. But as soon as you want to pull from it and you realize it's missing, then you go ahead and make that change.

4. 10:41 – 13:16

   Where team skills end and personal skills begin

   1. AGAakash Gupta10:41

      And going into the skills, so how do you maintain, how do you maintain like what's gonna be a team skill and an individual? Maybe somebody has like a different way of writing a PRD. What is the sort of prescription or guidance? You should be using our team PRD. You should be customizing them. What's the line?

   2. NHNecoline Hubner10:59

      Yeah. I think when it comes to updating skills, it... There's a rule set on, you know, where each skill should live. You wanna keep it closest to the, um, the work you're trying to do. So if it's a skill that is referencing code or is housed w- within, uh- As, if it's housed within a, a set of work that a team is working on that's always in that one repository, then keeping everything co-located makes a lot of sense. And then all the rest of the skills need to go somewhere. So you're either going to have a personal repository or you're going to have a shared repository, and potentially you create a branch where you test out a skill and you say, "Okay, first I, I think this is relevant. I think this is repeatable. I'll see if I use this a few times." And if you do, then you can push it up to main, and it-- That's been my approach so far. And if it's something that I think is super niche, then it might just stay in a branch that never gets pushed.

   3. AGAakash Gupta11:57

      Makes sense. And so everybody can connect into this harness using whatever model they want, but the harness primarily is these context and skill files. Is there any other components that people need to know about?

   4. NHNecoline Hubner12:08

      So there's the, the repository, and then there's the harness. And the harness, there's many open source harnesses. OpenCode is one of the primary ones that, uh, we've, uh, we've been watching as it's evolved over time and supported a lot of the models that we support as well. Um, but Hermes has also recently published one that I've enjoyed playing around with. And as you figure out which harness you prefer in your day-to-day work or if you're looking for models that suit a specific workflow, you know, I think a lot of product managers are starting to become a bit more familiar with, um, what models are suited to different tasks. But as more open source models are coming out, I think it's really interesting to see, okay, like is GLM-- how is GLM for coding? How is it for creating these product prototypes or Kimi for doing analysis? And I really enjoy that, that experimentation because as these models evolve, you feel like you've got a lot more control over pairing the right model for the capability that you're after at the time.

5. 13:16 – 15:17

   Ads

   1. AGAakash Gupta13:16

      I've been building a lot of AI products lately. My job search OS has sixteen different agents. My newsletter has a recommendation engine. And I kept running into the same problem. I'd ship something, it would work in my testing, and then I'd get messages from users saying it's hallucinating or picking the wrong t- The issue wasn't the prompts or the tools. It was that I wasn't actually evaluating anything. I didn't have a way to see what my agent was actually doing step by step. Every tool call, every decision. That's where Arize comes in. Let me show you. I'm going to open Claude Code and install Arize with just one command. Npx skills add Arize AI, Arize skill, skill yes. Now Claude Code already knows how to instrument my agent. I tell it, "Set up tracing to Arize," and it automatically analyzes my codebase, figures out where the LLM and tool calls are, and adds instrumentation automatically. Now I can see everything, every trace, every span, every decision. And more importantly, I can evaluate it. That's the shift. Trace what's happening, evaluate where it fails, then fix it. This trace right here, my resume feedback agent was supposed to pull the company's tech stack from the job posting, but instead it hallucinated that they use React when the posting said Python. I never would have caught that without seeing the trace. And here's the part that blew my mind. I asked Claude Code to look at these traces and tell me what I should be evaluating. It came back with four eval criteria I hadn't written, things like picking the right tool and staying grounded in the input. I wrote the evals, ran them, and found that my agent was making the same kind of mistake about twelve percent of the time. Claude pushed a fix, I reran the evals, and it dropped to under two percent. That whole loop, trace, evaluate, fix, took me about twenty minutes, and now it runs automatically. If you're building AI products and not evaluating them, you're shipping blind. Try Arize FREE at arize.com and get a year free, a twelve hundred and sixty dollar value, with my bundle. Arize. Check it out. It's one of the top AI evals platforms used by all of the top

6. 15:17 – 19:56

   Running feature research live

   1. AGAakash Gupta15:17

      AI teams for a reason. Amazing. So, Pavneet, can you show us what this is like in action? How does someone day-to-day use this for customer interviews, writing PRDs, and getting things into a state that they'll hand it over for engineers?

   2. PSPavneet Singh15:32

      Yeah, Aakash. I'm Pavneet Ahluwalia. I'm the Product Manager for Infrastructure at Together AI. So how it really comes into action is basically based on two processes. One is the initial discovery and aligning process, and the second part is the build and ship, which is more automated. And really what this ensures is that as a PM, when I'm shipping a, a, you know, a PR or I'm writing some code, it's not noise to engineering. It's actually grounded in our best practices and architecture. So let me show you this in action. So I have my Claude Code here. I'm gonna first start with like see a feature that I wanna work on, right? So I, let's say I wanna do some research on a potential feature that I heard some customer complaining about. So one that I'm looking into right now is, uh, how many customers are complaining about shared storage resizing?

   3. AGAakash Gupta16:24

      Mm-hmm.

   4. PSPavneet Singh16:24

      And what the feature is gonna do is it's, it's basically linked to, uh, Pylon, which is where our support tickets lived, to Linear, which is where all our project and engineering execution is tracked, and to Notion, where some of our internal product documents live. And it's basically pulling all the information, uh, from there to show me how big is this as a customer problem, how many customers are complaining about it, give me some verbatims and some tickets that I can go and d- deep dive into.

   5. AGAakash Gupta16:50

      So you had mentioned that certain parts of the product process are more automated than the others. Which do you feel like are more automated?

   6. PSPavneet Singh16:57

      Yeah. So the way, uh, we see it is the part which is, uh, where the decisions are being made, right, which is defining the feature, defining the API surface area, um, the abstraction layer, those are very human in the loop, right, where I'm hands-on with Claude. And the code writing part, the execution part, is actually the part which is automated. And how we would typically do that is, like, I would, uh, basically trigger a goal, right? And say-

   7. AGAakash Gupta17:24

      Mm-hmm

   8. PSPavneet Singh17:24

      ... uh, give me- Production-ready PRs for a feature to resize. So exactly the feature that we are gonna work on, right?

   9. AGAakash Gupta17:36

      Mm-hmm.

   10. PSPavneet Singh17:36

       Resize, uh, shared volumes for a running cluster without destroying data, right? Just hypothetically. Now, typically, as part of this goal, I would also give it a few other tasks. For example, give me a design doc that I can review with engineering. And first, let's verify on a POC cluster. So we would typically deploy it on a cluster just as a POC to validate everything, it's working as intended. And this is where, because goals are amazing, this will spin off multiple sub-agents and actually start tracking all the pieces. And it's gonna prompt me for information along the way, for example, like a POC cluster or more information about the feature, and that's where we'll feed it the PRD or maybe a UX prototype that I built along the way.

   11. AGAakash Gupta18:28

       Hmm. So very, very powerful engineer skills. It sounds like you've have a lot of confidence and the engineering org has built skills that you can just give a goal command, like get a production PR ready, and it will go pull the relevant skills. It will be able to read your codebase and get something actually production ready.

   12. PSPavneet Singh18:49

       Absolutely. We actually have a full engineering repo where it's just our architecture, uh, details, right? It's a bunch of skills which tells it for things that we've done in the past, how to redo it, like runbooks, uh, engineering architecture, design documents, which Claude has access to through our shared repo, and, and that's why it's able to build high-fidelity PRs, uh, in, in like a first attempt, really.

   13. AGAakash Gupta19:14

       Wow, I didn't realize just how good these had gotten at AI native companies. So that really then, like you were saying, in the human-in-the-loop parts, gives you a lot more time to focus on those in depth.

   14. PSPavneet Singh19:27

       Yeah. So if you see right now, it's actually spun off a sub-agent to look at how is the one data plane working, how-- And then another sub-agent to check how is the control plane working. And now a new one to look at docs, right? Is there existing internal or external documentation on how we might do this?

   15. AGAakash Gupta19:44

       Hmm. And do you have a point of view, like you're using looks like Claude Code in the terminal, how do you-- what's the easiest way to use a shared team repo harness?

   16. PSPavneet Singh19:54

       Yeah. So actually, if you look

7. 19:56 – 27:31

   The PRD writer that interviews you

   1. PSPavneet Singh19:56

      at this, it's currently running in my OS, uh, e-- uh, directory, which has actually pulled a lot of the shared repo artifacts. So, for example, Niko shared a, a few artifacts around research. So it's actually this skill, the research, feature research skill, which is actually a shared skill that everybody uses in the team. Then there's another one, which is m-my personal favorite, which is PRD writer. This is actually pretty cool. Like, let me just actually trigger it off, right? So resize shared volumes for existing running clusters. So what this actually does is it does a turn-by-turn interview of... It'll interview me basically on what are the decisions I wanna make in this feature. And it-- And if I show you the skill, so this is the actual skill. Uh, it ingests a-- It can take a prototype. So if I already have a UX prototype, it'll ingest that. If I have a running POC, it can take that or nothing, just like a simple prompt, right? And then it has access to all the information, the co-- shared context that Niko shared about, right? It's pulling all of that in, and then it's gonna ask me questions, uh, turn by turn, and it's specifically instructed to not, you know, to push me, to challenge me on my, uh, assumptions and basically the decisions I'm making. And then it goes into drafting a PRD. And we've given it a question bank basically to kind of ground it in what type of questions typically we need to make. For example, like what are the trade-offs? Is there any one-way door decision we are making along the way?

   2. AGAakash Gupta21:31

      Amazing. So we've spinned off, I think we have currently three agents working for us, right? Research, PR, and PRD.

   3. PSPavneet Singh21:38

      Yeah. So this one is actually done. So the research, uh, agent came back. It's saying all these customers have asked for this feature in the last six months. There are nineteen tickets that were filed in the last two months alone, so this is a pretty heavily requested ask.

   4. AGAakash Gupta21:53

      Hmm.

   5. PSPavneet Singh21:53

      I have the source, so I can actually go into these tickets if I want to dig deep exactly what happened and why did it ask, why did customers ask this. And then-

   6. AGAakash Gupta22:02

      And what was Pylon again?

   7. PSPavneet Singh22:03

      Pylon is where our support tickets live.

   8. AGAakash Gupta22:05

      Hmm.

   9. PSPavneet Singh22:05

      It's the support, uh, s-- uh, platform.

   10. AGAakash Gupta22:08

       Mm-hmm.

   11. PSPavneet Singh22:09

       And then actually it figured out that we had started working on this feature, uh, because it has access to Linear as well, and it was partially implemented, but it's broken in some way, so we stopped working on it. So now I'm not duplicating, I'm not, like, replicating work. I can actually build off somebody else's work or actually talk to that engineer and figure out, hey, what happened? What issues we ran into.

   12. AGAakash Gupta22:30

       And is this hooked up to the live codebase, so it could go check, like in case there's some discrepancy between Linear and the codebase?

   13. PSPavneet Singh22:37

       Yes. Because it's in my personal OS, uh, directory-

   14. AGAakash Gupta22:41

       Hmm

   15. PSPavneet Singh22:41

       ... it has a skill which gives it, uh, access to all the internal GitHub repos, and it knows what's what and map that.

   16. CZCharles Zedlewski22:49

       By the way, Aakash, later on, I can show you like a God's eye view of, like what this looks like across all products for repos as well as across all customer feedback.

   17. AGAakash Gupta22:58

       Okay. I'm excited for that. So, Pavneet, here we've got the research report. What's your take on this research report? Is this a, like how long would this have taken you? What do you give this out of one out of ten?

   18. PSPavneet Singh23:09

       Yeah. So if I had to do this manually, this would have easily occupied half of my day, if not more. Uh, and this is actually pretty good. It gives me the brief snapshot that I wanted to know, which tells me the feature passes the smell test. There are enough customers asking for it. I know most customers are actually want to increase their shore-- storage volume, right? So I actually know what's my, like, MVP use case. And then I have a source, so I can actually dig dive into some of them to actually get a better sense of the customer pain. And it gives me verbatims as well, which kind of helps. So I know, for example, here it call- it called out a customer called Moon Lake, and we have a separate tool called Customer Insights where all customer calls live. So let me go into that. So this is the Customer Insights tool where every customer call that our sales team is getting into, all our support tickets are getting summarized and cataloged. So I can actually go into this and actually see what exactly happened with Moon Valley. So they wanted a fifty terabyte, but there's a lock icon in the UX which is preventing the action, and they are basically-- They reached out to sales team, and they basically want a self-serve workflow, right? So I kind of get a pretty good sense of what the customer problem is within, like, five minutes instead of maybe a full day.

   19. AGAakash Gupta24:25

       And this Customer Insights tool, so you guys have built an MCP server sitting on top of Gong, and this has been visualized by v0. That's what we're looking at right now?

   20. PSPavneet Singh24:34

       Exactly.

   21. CZCharles Zedlewski24:34

       It covers both Gong, Slack, Pylon. Um, yeah, at least those. So it's actually more than just, uh, Gong.

   22. AGAakash Gupta24:42

       And who maintains this? Sales ops?

   23. CZCharles Zedlewski24:44

       Hassan, you wanna speak to that? It's largely automated, but...

   24. HAHassan24:47

       Yes. Um, me and my team maintain it. So my name's Hassan. I run the developer experience team here at Together. Um, and, you know, uh, me and my team have built a bunch of, of this kind of stuff. We, we maintain it. We continue to add new features, but it's largely kind of working by itself at this point. We have, like, a daily cron job that runs every day that grabs all of the calls, all of the Pylon tickets, all of the, like, Slack channels, um, everything that happened in the last twenty-four hours, and it adds it to the database that this is running on. Um, and, uh, yeah, what Pavneet is showing is the daily screen that, like, where we show, like, five to ten kind of insights from customers every single day based on all these calls. We also have an MCP server. We have a chat where you can ask it anything. There, there's a, a lot of kind of facets to this tool as well. But, um, yeah, my team continues to, um, maintain it and, and kind of add new features here and there.

   25. AGAakash Gupta25:38

       Okay. So developer experience maintains, like, various MCPs. What other MCPs are you maintaining?

   26. HAHassan25:44

       Good question. Um, we have this one. I mean, some of them, uh, the, the-- A lot of these tools are exposed differently. Uh, this one has MCP server. Uh, for example, another tool, uh, is called Orchestrator. Charles is, is gonna show it off and, and talk about it momentarily. Uh, that one has kind of just a UI and is a wrapper on top of all our, our GitHub repos. Um, and, uh, yeah, really my team just experiments with a lot of this stuff. We build things that we think may or may be useful for, for the, the product team and, and the rest of the company. Um, and some of the internal tools kind of flop and we're like, "Well, you know, this is, this is not used very much." And, and, um, probably our two, uh, or some of our most used ones have been this, like, Customer Insights, uh, MCP and app, um, and, and, uh, Orchestrator, uh, and AgentEvals, which I'll also show off. So those are, like, the, the top three that, that have been, um, [clicks tongue] that people have been using.

   27. AGAakash Gupta26:33

       Sweet. So we get to see all three. Awesome. And Pavneet, how's our, how's our PRD and our PR agents looking?

   28. PSPavneet Singh26:40

       Yeah. So, so it's now asking for evidence. So this is where typically I would run this in a single Claude chat because this is sort of somewhat sequential work. But typically then I would take the output of this research, which is pretty detailed for, like, a starter starting point, and I would actually give it to the PRD, uh, agent to run with.

   29. AGAakash Gupta27:02

       Mm-hmm.

   30. PSPavneet Singh27:02

       So now it has a sense of the customer problem, so it can ask me more educated questions on, "Hey, what are the trade-offs? Should we build as an abstraction? Should it be self-serve? When do we pull in support?" Those kind of questions.
8. 27:31 – 35:07

   What a good PRD looks like now

   1. AGAakash Gupta27:31

      Uh, we all have gone through phases of using AI for PRDs. You know, we've all discovered ChatGPT in 2023, and were horrified at the output and started handwriting. Then we discovered Claude and said, "Okay, maybe it can write well." And then it almost feels like, you know, people were overloading our colleagues' context windows with long documents. What is the-- What is a good PRD in the AI era?

   2. PSPavneet Singh27:56

      Yeah. So that's actually a question we thought a lot about. So where we really use PRD, like, historically, like, companies started using PRDs as, like, a gating document where everybody, all stakeholders need to come together, align on it, and only then we'll start building the feature. We see it a little bit differently, where it's a tool to trigger ideation and problem-solving. That's it. And it's short. It's usually one to two pages. It, it defines the customer problem well so that everybody has a shared understanding of the customer problem we are trying to solve. It lays out some solution options, right? Doesn't have to be fully thought through necessarily, but at least, like, what are the different ways we could go solve this problem, right? And then a sample user journey. Like, based on the, uh, option we wanna go with, it'll define, uh, uh, either a API-based journey or a UX-based journey to show, hey, these are the three, four steps that a user would go through in, in the solved world, right? Once this feature is shipped. And that's it. Like, that's enough to actually have an detailed discussion around, hey, how should we build this feature? What should, uh, engineering design look like? Is it even worth building or not, right? And, and that's it. That's what the PRD template that this, uh, skill uses builds off. And actually, what replaces our typ- historical six-pager, like, I come from Amazon, right? I used to work at Amazon, so we are used to writing, like, 20-pager PRDs. What replaces the bulk of that is actually a prototype. So we built a separate skill. Let me start a new Claude window. And there's a separate skill called UX Prototype, which actually you give it that one-pager that we just discussed, and it'll give you a very detailed prompt on how to build it in like a Figma Make or any design vibe code tool. Like you can use it-

   3. AGAakash Gupta29:42

      Mm

   4. PSPavneet Singh29:42

      ... in VO, you could use it in Claude Design or even Figma, right? And that's where a lot more debate will happen because now everybody can visualize it, visualize the solution, see what's- how it's gonna interact with the user, and that's where we find like a lot of the best critical feedback comes from, from engineering, marketing, or anybody else.

   5. AGAakash Gupta30:00

      Do you find th- you're living outside of Claude Design then a lot? You're getting this to generate the prompt and then giving it to Figma Make?

   6. PSPavneet Singh30:07

      Yeah. The only reason I use Figma Make is because it's easy to share a Figma Make link and others can iterate on top of it. It's easy to click around. Claude builds like HTML files, which I, which it's hard to keep version tracking and stuff like that. Figma is just like more shareable, that's why we use Figma Make.

   7. AGAakash Gupta30:23

      Makes sense. So how do our, how do our PRD and PR agents look?

   8. PSPavneet Singh30:28

      Yeah. So okay, so now it's getting to... Now it's asking me questions, right? So it's saying, "Hey, the evidence splits into two different directions. One is multi-tenant substrate quota bumps, and the other is dedicated commercial expansion. Uh, which one do you wanna go for?" I actually wanna go in a very specific direction. Just looking at like the feedback we got from customers, uh, I wanna say, "Hey, we should focus on increasing the attached storage volume sizes for tenants." Tenants here is basically a customer cluster, right? So tenants in a running cluster.

   9. AGAakash Gupta31:10

      So here's where like your domain expertise, your human judgment is coming in.

   10. PSPavneet Singh31:14

       Yeah, like it's not a substitute for me, but it really does two things. A, it speeds up the whole PRD writing, like solves the blank page problem to a large extent, but it also catches a lot of things that I might not have thought of. Like, I often find it's asking me questions that I might have missed if I'm not spending a lot of time thinking about the problem.

   11. AGAakash Gupta31:33

       Hmm.

   12. PSPavneet Singh31:33

       So ignore BECA issues. Like, I'm just giving it like a s- sample, uh, to focus its energy and to give it like reduce the scope, so it's not trying to boil the ocean in the problem space.

   13. AGAakash Gupta31:45

       Mm-hmm. So a ten out of ten that's comes out of it, it's gonna be kind of that very short document that accompanies a prototype-

   14. PSPavneet Singh31:53

       Yeah

   15. AGAakash Gupta31:53

       ... if it, if this all goes well.

   16. PSPavneet Singh31:55

       Yeah, so I actually did this and I have that, uh, PRD. If you want, I can show it to you what that looks like.

   17. AGAakash Gupta32:01

       Yeah, let's take a look.

   18. PSPavneet Singh32:02

       So this is what the final output looks like, right? So first of all, it's telling like where's the evidence. It, it looked at... We had 20 pilot tickets across 14 customers, right? So straight up, you know there's evidence, and this is what that looks like. It first defines the customer and the business problem like we talked about. The 20 confirmed customer issues, it's name-dropping some customers just to give us a sense of which customers, right? Is it our top big customers? Is it like a long tail of customers? And then it gives you a sense of like what is the actual pain, right? It seems like we cannot update this in the UI, and that's basically the ask. They just wanna self-serve, uh, the volume resizing, and that's like pretty obvious from the customer verbatims. And then you go into goals and non-goals. This is important for scope creep, like, which is like the number one problem I feel like PMs and, and engineering faces. So it gives you some goals, like it's giving me a goal that, hey, if done well, this should basically, uh, handle over 90% of resize requests, right? It works in both directions, uh, upwards and downwards, like s- scale up and scale down. Uh, billing should adjust immediately. See, this is something I did not think of actually. Uh, when somebody, uh, reduces the size or increases the size, we need to make sure that the billing matches up and they're not over or under charged for it. And capacity block request. What happens when we don't have enough capacity, right? If it's a fully self-serve, uh, experience, when do we actually need to get, uh, support involved?

   19. AGAakash Gupta33:28

       Mm-hmm.

   20. PSPavneet Singh33:28

       And then it defines the non-goals as well. Key user stories so that, you know, whoever is reading this and when we are doing a discussion in engineering, everybody understands the key use cases. And then actually the proposal overview, right? It's giving me some API specs. Um, it actually gives like a detailed API design as well. So we're gonna add to our existing API some changes on how to, uh, basically allow for resizing. This is the user story sort of, right? Like what are the different, uh, API specs and what is the API called. And then UX as well. End-to-end user flow, uh, happy path, what are the steps that a user will follow.

   21. AGAakash Gupta34:02

       So I feel like it always gets the headers right, but sometimes the devil is in the details. Is it nailing the details?

   22. PSPavneet Singh34:08

       So let's look at this one, right? So, uh, admin sets a new size in the console, API validates. This is some internal detail actually, so maybe this part not necessarily needed in the user flow. Uh, this is actually internal detail, and then it shows the status. It shows that it's available and billing rate is updated. So it got the user journey right, but it bled some internal detail into it as well.

   23. AGAakash Gupta34:34

       So it's still gonna need a little bit of editing. You can't just immediately take this and start sharing this with your colleagues.

   24. PSPavneet Singh34:41

       Yeah, absolutely. Actually, at the top it says this is a draft one-pager PRD. It's not meant to go d- wide distribution for everyone. And that again kind of goes back to the process that we were talking about, where in this initial discovery and design phase, it's human in the loop. So the expectation is that the PM will go and make changes, maybe make some changes to the API, maybe change the scope a little bit before we circulate this widely.

9. 35:07 – 40:24

   Orchestrator, one view across every repo

   1. AGAakash Gupta35:07

      Very cool. So we've got the repo, we've gotten to see it in action. We got a little preview about an Orchestrator agent. Charles, can you walk us through this? How do you get to see this bird's-eye view at the top of the product ladder?

   2. CZCharles Zedlewski35:21

      So we built this great internal tool which we call Orchestrator, and the basic idea is each of the individual product people are sort of married up hand in glove with their corresponding engineering team. And Pavneet kind of gave you a good example of how that works, where like he wants to be- Able to go pretty far into the definition and implementation for some of the things he works on, and his sp- the specific engineering team he works with has given him the kind of skills, uh, connected to their codebase that he can go do that. From my vantage point, what I need to do is be able to, like, get a check on, like, where things sit in all kinds of different things that we're building as a company, and I don't wanna have to, like, build the equivalent of, uh, Pavneet and Hassan and Niko's environments. So I have this nice tool here called Orchestrator, and you can see that essentially all the major repositories we have as a company for all of our products are represented here. So if you see, uh, give you kind of a quick, um, example. So you have, like, uh, let me find Pavneet's, uh, T Cloud. Yeah, so you can see here, uh, we have the repo that Pavneet was just living in T Cloud, but you can see that it coexists along with many other repos. So for sake of a argument, like, one of the other products we have we call Model Shaping, which is basically the ability to adapt the behaviors of open-weight models. And, um, I can decide how I wanna interrogate what's going on in that codebase, what's going on in that product area. I can pick my harness. I can use either Open Cloud- OpenCode, Claude, or Cursor. I can pick my model. So in this case, like, I tend to be a, you know, GLM OpenCode kinda guy. Uh, and I could ask some question like, um, [keyboard clicking] what's, uh, the most recent model, um, we have enabled for supervised [keyboard clicking] supervised fine-tuning, right?

   3. AGAakash Gupta37:17

      Mm.

   4. CZCharles Zedlewski37:17

      So for we, we basically have, like, more than 30 models that you can adapt and, you know, fine-tune, but these are changing all the time. Uh, I'm not-- I don't wanna have to go understand the entire world of the product lead for Model Shaping. I just wanna understand what's going on in this one specific, you know, to answer this one specific question. Um, and you can see that essentially what we'll do in this case is we'll actually create a sandbox. We'll clone the repo. All this happening in the background. And in a minute it's gonna basically interrogate that portion of the codebase, [lips smack] uh, and, uh, it's gonna, um, go research, like, what are the most recent changes, and let me know what happened most recently.

   5. AGAakash Gupta37:54

      Mm.

   6. CZCharles Zedlewski37:55

      You can see it kinda cogitating right now. Um, uh, like the proverbial, uh, cooking show, uh, I have, like, kind of a, a, a synopsized version of, like, the conclusion. Uh, and in this case, it looks like the answer is the last model that we enabled, uh, for supervised fine-tuning is the new, uh, NVIDIA Nemotron, uh, Super 120B model. Um, so I can do this exercise essentially across, like, any product, uh, any, any portion of the codebase for that product. Um, and I can... And, uh, it's not just limited... Uh, here I'm just asking questions, but if I also find some part of the product that annoys me and it's something small, uh, I can actually generate a pull request from here as well.

   7. AGAakash Gupta38:41

      Okay. So you're mainly living in the Orchestrator, not inside CloudCode and the team repo?

   8. CZCharles Zedlewski38:46

      Uh, I use CloudCode and the team repo for if there's some requirement I'm writing myself for some part of the codebase, then I would do that. But if it's me, uh, shopping across, like, all the different products we have, and it's, like, some small UX change or things like this, then I would sooner use the Orchestrator, because this is basically not just pointed at all the different repos we have as a company, but it's inheriting all of the, uh, skills and/or MCP servers that are local to each of those repos. So I don't wanna have to build all that into my local, uh, OpenCode just to make one small change. It's a lot simpler, uh, to just, to just use Orchestrator, and it's-- and I know it's gonna have, uh, the latest, greatest skills and MCP servers for that portion of the codebase.

   9. AGAakash Gupta39:35

      Got it. So who are the other users of Orchestrator within the company? Like, what i- who is this product exactly built for?

   10. CZCharles Zedlewski39:41

       It's sort of intended for casuals, right? It's intended for... So, so it's intended for, like, uh, let's say Pavneet wants to, uh, investigate something in Inference, or let's say, uh, Necoline wants to make a suggestion on, on, uh, our infrastructure as a service. Um, they don't wanna build and replicate all of the specific skills and context n-native to that particular portion of codebase. It's a lot simpler to just look at, uh, just to use Orchestrator, where everything is kinda maintained server side.

   11. AGAakash Gupta40:14

       Mm. Okay. So this is, like, for whenever you're casually working, maybe, like, trying to learn something about another team-

   12. CZCharles Zedlewski40:21

       Right

   13. AGAakash Gupta40:21

       ... this is also creating that connectivity.

10. 40:24 – 44:19

    Shared context is a hierarchy, not a flat pool

    1. CZCharles Zedlewski40:24

       Yeah. One of the things that we talked about as a team was, like, like, like, it goes back to this point about how much is shared and how much is individual, right? And there was a point at which I thought, well, wouldn't it be great if everybody knew what everybody else was doing, and there was sort of like we all had the same context and everybody else at all times. And in reality, uh, most people don't have a whole lot of motivation to wanna understand all the depth and nuance of the context of somebody else's area. They wanna know just about they need [chuckles] to get one single question answered or to get one single problem resolved, and they're really not interested in the rest. So we sort of, we sort of set aside the idea that there was one big, broad, flat set of context that we're all gonna swim in, and it's much more like a context hierarchy, and some of us belong all the way down to the bottom of the depths of that hierarchy, and some just wanna traverse the top.

    2. AGAakash Gupta41:19

       Mm. Fascinating. Okay. So if a team, company watching this, they wanted to replicate what's being built in Orchestrator, how are they gonna spin up their own version of this?

    3. CZCharles Zedlewski41:32

       Yeah. So I mean, this, like, from what I-- from what we see in terms of our own customers, this is becoming increasingly common. Like, I was at a AI conference in, in Paris the other week, and I saw, like a, like a, like a digital, um, uh, services and marketing company called Process. And, uh, this idea of essentially Um, uh, um, uh, they, they basically already built their equivalent of the same thing, um, where they have like every, every portion of the codebase you want to change is all in one place. Everything can be spun up as a sandbox and generate its own unique pull request or its own questions. So, um, I, I think this set of, this set of tools is like not that out of reach for most software development organizations. And, um, this idea of like we're all gonna live in a multi-model, multi-harness world, and the main, the main endeavor then is, uh, how do you organize shared context? That's, that's kind of, I think, what everybody is starting to build. Um, to give a more specific answer, I mean, Hassan, what was the total, uh, like time invested, would you say, to build, uh, to build Orchestrator?

    4. HAHassan42:44

       [lip smack] I would say a few weeks of work, maybe a month of work, um, to, to, to build it, uh, of engineering time. Um, yeah, and a lot of it was kinda just like trying to figure stuff out, uh, fairly early, trying to figure out the right architecture, the right tools to use, a lot of that stuff. But as Charles said, I think like this has kind of been replicated by, by a bunch of other companies. There's like a few open source versions of this as well that, that exist out there, so it's, it's kind of easier than ever once you have the architecture. But yeah, for our team it took, it took a few weeks.

    5. AGAakash Gupta43:14

       So like if we're just s- drafting the PRD for an internal orchestrator for yourself, what are the key things? It needs to hook into-

    6. CZCharles Zedlewski43:21

       Oh, yeah

    7. AGAakash Gupta43:21

       ... like all of your codebases and your different repos. It needs to inherit all the active MCPs and skills from the different team repos.

    8. CZCharles Zedlewski43:30

       It needs a sandboxing mechanism because you're basically gonna s-- like, essentially, like each individual thing you're researching or each pull request you're gonna propose, that all gets done in the sandbox environment, so you're basically cloning a fraction of the repo, right? The portion of the repo that's necessary, uh, in the sandbox to generate that, that one pull request.

    9. AGAakash Gupta43:51

       Got it.

    10. CZCharles Zedlewski43:51

        Um, you also are gonna need, uh, some form of a model gateway or router. So you notice I have like a whole choice of models that I could use. Um, so that's, that's typically what another, another piece of the puzzle. But as Hassan mentioned, every one of these components I've mentioned so far, there are either, like either it's relatively quick to build, there are open source, uh, uh, tools and libraries, um, that lets you do these things, and there's commercial shrink wrap software if you don't feel like doing either of the previous

11. 44:19 – 52:26

    Agent evals, testing the product as an agent

    1. CZCharles Zedlewski44:19

       two things.

    2. AGAakash Gupta44:19

       Fascinating. So we're walking through the whole product development life cycle. The last step, evaluating how these things are going live. Hassan, how are you guys doing agent evaluations?

    3. HAHassan44:30

       Great question. Um, as you said, this is kind of the final, uh, step that we do. I'm gonna share my screen. I'm gonna go over, uh, this tool that, that our team built called Agent Evals. And just to give a little bit of context, uh, on Agent Evals, um, you know, I, I, I think the whole world is moving to this agent-centric way [chuckles] of doing things, right? You don't, you don't kind of manually write code anymore. Um, and, and kind of it's moving up the stack with like you just give your agent stuff to do. Um, and so, you know, my team is called Developer Experience. I think Charles is, is, is this close to renaming it Agent Experience, um, and because it's something we're, we're increasingly, uh, thinking more about and it's very, very top of mind. We've rearchite- rearchitected our docs in kind of this agent first way where it works very, very well for, for humans obviously, but also for agents. Um, you actually on some pages on our docs, like agent-- when, when an agent is reading them, we'll inject something extra, um, if we think it'll help the agent. Um, we worked on it Together MCP server, we worked on Together Skills. And so we're exploring all of these ways that we can make it as easy as possible for agents to use our product. Um, and Agent Evals is kind of the, the tool that ties them all together or the tool where, where we can actually say like, "Okay, like we're, we're pretty confident in, in, in how this works." So I'm gonna share my screen and, uh, we're gonna go over this. Fantastic. Okay, so Agent Evals, there's a lot here. Um, but Agent Evals is, is the tool we use kind of at the end of this product development, um, life cycle. Um, we have the product, it's shipped, right? Some kind of some major feature or a product or s- or second version of the product, and now we need to validate that it actually works well with agents. So what we do is I tend to work with a lot of the product managers, um, in each of the different areas, and we write a series of tests of like the main, like here are the main ways that we think users will use a specific product. So, um, let's actually take fine-tuning as an example. Uh, or we'll take, um, yeah, let's take fine-tuning as an example. So if I click on Fine-tuning, uh, this is a prompt that we give for-- this is one fine-tuning task that we're testing. So here we're saying, "Hey, we gave it a dataset," and we said, "Hey, run a fine-tuning job for this dataset, and when it's done, spin up this new fine-tune model as a dedicated endpoint and evaluate it," right? Where it's gonna send it some, um, uh, uh, inference requests, uh, and then delete the endpoint when you're done, right? So we gave it, uh, and then this actually touches on a few of our different products. It touches on our fine-tuning product and our inference product, and so some of these tasks are, are a little more simple, some of them are a little more complex. But the point is that the, the most important part is to define a series of tasks, um, and give it to this, to this tool. And what we do, um, what we do actually has a very similar architecture to Orchestrator, which you talked about, is so what it'll do is it'll spin up a sandbox, it'll spin up right now Cloud Code, uh, and it will, and it'll give us, it'll give it this prompt, right? It will give it a Together API key, and then it gives it whatever it needs for the prompt. In this case, we gave it actually a, um, a dataset, uh, and then it just watches Cloud Code do its work, right? It's just, um, it, it, it lets us do its thing and then it evaluates it at the end. And for us, this is like the best test of like can an agent actually do, uh, stuff in your product. We try to make sure-- We try to make everything in our product, um- Uh, w- we try to make it so that you can do everything in our product, for example, uh, in our UI as a human, but also, uh, as an agent through our API or CLI or SDK. So in this case, it actually got it correct, uh, but we go very, very deep here, so we run a lot of different runs. Some of these runs are just using our docs as context. Some of them use our MCP server, some of them use our, our, our skills, and we can click into every one of these, and we can get a lot of information on, on the run, how it worked. We can get some high-level info. We can get improvements, which this is one of the most important pieces of this, and, and this is something we see a lot where we'll, we'll ship a feature or, or we'll, like, be in the validation stages of a feature, and we give it to agent evals, and then it's like, "Oh, well, I had trouble doing this thing," right? I, I wasn't able to... Like for example, here I said it wasn't able to discover what fine-tunable models we have, right? Obviously, this is a, this is a big problem if it can't find this. Um, and, uh, and it said, "Hey, like, it can't, it couldn't find a docs page listing currently fine-tunable models, and so we should do this." Um, and so we make a ton of improvements to our docs based on this. In this case, actually what happened is we had a page that, um, listed all the fine-tunable models, but the agent actually couldn't find it. We were able to go into the transcript. You can go into the transcript, and you can see the full transcript of this agent and what it did, right? We gave it this initial prompt, and you can see all of the different terms. We can see the, the, the agent started exploring the repo. It started looking at the dataset. It, like, did all of this stuff, right? Like, [chuckles] the full, the full transcript. Um, and, uh, and, and we, we looked at it, and we saw, like, actually it had trouble finding that one, um, uh, that one page in the docs 'cause it's not linked in our main fine-tuning quick start. So then we're like, "Okay, well, we'll just go submit a PR to our docs and add that page to our fine-tuning quick start." So, um, the agent evals has been responsible for, for, like, dozens of, like, docs fixes that we've done, um, and, and it's been really, really valuable to really understand, like, uh, and get this bird's eye view from agents on, like, how good are agents at using this particular tool or this particular API bit that we just shipped.

    4. AGAakash Gupta49:39

       Mm.

    5. CZCharles Zedlewski49:41

       And, Aakash, like, I just wanted to, like, add on. I mean, you think about any sizable feature or product that you've built in the past, and the idea about, like, well, did you get the design right? Like, did it actually, you know, can the, can the outside user, whether it's, like, someone technical like a developer or non-technical, uh, be consistently successful using that feature? And think of how long you had to wait to validate that as, as a, as a product person before. You'd have to wait till, uh, you did a bunch of user tests, and then you're gonna get a bunch of conflicting signal, or you're gonna have a bunch of developers evaluate it, and they're all gonna say, "Well, I, like, I don't think this is sufficiently Pythonic," or, "I don't like the way you did this syntactically." Uh, and so it's like, it's really hard to get a timely signal, and it's really hard to get objective signal. Um, and here essentially, you know, at this point for a lot of our products, agents are already the majority user. Um, and so the idea is that the, the nice part about that is now did the feature work from a design point of view? This is something that we can validate immediately and continuously. Um, so, uh, it's really powerful to sort of always know where you're at, and it's really powerful to have the confidence to know that whatever your documentation says your product is, uh, you can be sure that it actually is that because essentially a few hour- every few hours we have agents revalidating that. Um, and across time what we plan to do is keep expanding the range of harnesses and models that we do this for because that's essentially the new bar for UX for the kind of products we build.

    6. AGAakash Gupta51:27

       Okay, very cool. So if your product isn't used by agents yet much, [laughs] does this still have value? Is this able to kind of simulate what a human is like, and should people still be setting this up?

    7. CZCharles Zedlewski51:42

       So in, in, if... For us it is because our users are developers, and so they're gonna use our SDK, and this is basically validating that there's, like, valid, successful paths using our SDK. If we had a more gooey, like, you know, web, you know, web front-end intensive product, we would have to adapt this tool, uh, to use more, uh, you know, like a visual reasoning model that could actually do the click-throughs and, like, interpret the screens and see whether or not that was, like, naturally intuitive to the model. So it could be extended to human-centric examples, um, but for our case where agent use is already so popular, um, it works as is.

12. 52:26 – 59:18

    Where the PM ends and the developer begins

    1. AGAakash Gupta52:26

       Mm. All right. So we've been able to cover front to back from customer research through to actually evaluating how agents would be using your feature. This is kind of the whole product development life cycle. I'm curious where this ends. We kind of drew the line here at create the PRs, create the prototype. Where does the line of product manager end and developer begin?

    2. CZCharles Zedlewski52:53

       Try... I, I, I wanna, I wanna think how to best respond to that without, like, just, like, repeating what Pavneet just said at, um, at the beginning of our, um, of our whole thing. Um, I think that the, the essence of what the product person does versus the essence of what the engineer does ironically is probably not all that different, uh, than what it was before AI, in the sense that the most valuable thing the product person can do is bring a unique insight about the market that's been well-validated by lots of internal and external context.

    3. AGAakash Gupta53:30

       Mm-hmm.

    4. CZCharles Zedlewski53:30

       And that's essentially what you still saw Pavneet do. And the essence of what an engineer does to add value to the company is to, um, arrive at a design That is the most efficient way to meet a need. Uh, that is also, um, that is, that, that, that is differentiate... That, uh, the, the essence of the engineer's, uh, job is to arrive at a design that most efficiently satisfies a need and is also something that you can maintain and extend, um, and contributes to sort of the long-term, um, architectural, uh, um, strength of the product. So, like, these two, these two, the essence of each thing I don't think is actually all that different today than it was before AI. What's different is the convenience with which the product person can reach into the, uh, to s- uh, to, to the engineering world and, and accomplish small, uh, small and medium-sized tasks. And the inverse is tu- is true as well. The degree to which the engineer can reach into the product management process and answer their own questions. So just like it's possible for me to interrogate the codebase and, uh, make small pull requests, it's just as possible for one of my colleagues in engineering to use that customer insights tool and do their own analysis and have their own standing query, uh, for, for whatever the customers have been asking for in the past month in their area. So we sort of, like, it's sort of easier for each side to reach into each other's area, um, to do small things, but in terms of, like, what each person's supposed to contribute that brings their unique talents and perspectives, I think that's actually th- the same core as it's ever been.

    5. AGAakash Gupta55:10

       Amazing. So I've been preaching to people, "Get your PM OS, get your team OS, get your company OS." You guys just demonstrated and are actually living that reality, so you're at the very top of the AI adoption curve. And I heard a really interesting, uh, observation from Chamath Palihapitiya. He said, "You know, our token costs are something like doubling every 70 days, but our actual engineering productivity is just up, like, 5%." When you look at it from a product lens on that, like, how are your costs on AI growing, and are you seeing a tangible, like, some sort of tangible productivity gain that you can point to from it?

    6. CZCharles Zedlewski55:50

       Yeah. So, I mean, we never were in a place where we were doing, like, story points or other kind of velocity measures, so I can't prove it on that level. But I would definitely say that our vel- our velocity has gained more than 5%. Um, I find claims of 3X to be very suspicious. I don't have any... Uh, like, when you get a team of, let's say, a dozen engineers and a product manager, um, there's so much of building software which is discovery, um, debate, uh, um, uh, uh, kind of reevaluation, uh, coordination, and a lot of these things don't get magically better, uh, with AI. So even if you compress the research and even if you compress the coding and the testing, um, I think that's worth a lot in terms of velocity. Um, I don't, I don't know that I, I would say our experience has been that, like, 3X or, or some huge multiple like this is, is, is the case.

    7. AGAakash Gupta56:49

       Mm.

    8. CZCharles Zedlewski56:50

       And as far as expense goes, uh, uh, I think we, we went through the same surge that a lot of folks did. It was easier for us to mitigate because we can use our own open-weight models and they're a lot less expensive. Um, but the other part is, uh, it goes back to this first point, which is, I think if, if you're focused on making teams collectively productive as opposed to yourself feeling individually productive by producing lots of output, I think it's, I think it's unlikely that you wind up with these crazy, like, you know, token budgets of three times people's salaries or things like this. I don't, I don't know that we, we ever reached that type of peak.

    9. AGAakash Gupta57:27

       Mm-hmm. I like how measured you guys were in selling the benefits of all of this, where none of it was overhyped, but we got to tactically see how it helped both individual IC PMs like Pavneet and a product leader like yourself, Charles. All four of you guys, thank you so much for dropping so much insider knowledge about how Together works.

    10. CZCharles Zedlewski57:51

        Thank you for having us.

    11. AGAakash Gupta57:52

        I hope you guys enjoyed that episode as much as I did. We... They have been kind enough to literally open source the tools, the repo, the orchestrator, how they built this. So check the link in description below if you wanna get your hands on how Together actually built this and replicate some of this in your own company. I highly recommend you do that. Podcasts are one thing, but actually implementing it, that's when you really get the ROI. So I hope you go do that, and we'll see you in the next episode. I hope you learned as much from today's episode as I did. If you can do one thing that's totally free that would help the show, it would be to check that you're following on Apple and Spotify podcasts. Check that you've left ratings and reviews on those platforms. Check that you're subscribed on YouTube. Leave a like and a comment on this video, and then share it with your friends. We're trying to make better and better podcasts. After two years, we think we've gotten something pretty good going. So let us know what we can do to make it even better, who else we should interview, and we will put on the best shows we possibly can. Finally, don't forget my offer for the bundle. You get an entire year of my paid newsletter, plus my favorite AI tools, bolt.new, Airtable, Speechify, Descript, Magic Patterns, Linear, Dovetail, Arize, and Mobbin'. That's $27,000 worth of value for just $150. So check that out at bundle.aakashgee.com if it interests you, and I can't wait to share our next episode soon.

Episode duration: 59:27