CHAPTERS
- 0:07 – 0:14
Founders’ quick tour: three distinct Managed Agents use cases
The host invites three founders to share how they’re using Managed Agents. Each describes a different product surface: meetings, sales workflows, and product analytics connected to source code.
- •Sajjad frames meetings as a full journey: before, during, and after, where agents help most with preparation and follow-through
- •Mihir describes an agent per sales account plus a cross-account layer for reporting and prioritization
- •Todd explains an agent that connects application telemetry with a snapshot of the codebase to generate insights and potential fixes
- 0:14 – 0:52
Meeting assistant: pre-meeting briefs and post-meeting task automation
Sajjad explains how their meeting product expands beyond the live meeting itself. Managed Agents support new capabilities both for preparing users ahead of time and for executing tasks after the meeting ends.
- •Agents are positioned as most valuable before the meeting (prep) and after the meeting (execution), not just during the call
- •Briefs are used to prepare users for upcoming conversations
- •Task and task-automation features handle follow-up work after meetings
- •Managed Agents are used as the underlying system to design and build these features
- 0:52 – 1:35
Sales platform: Watchtower cross-account agent built fast
Mihir contrasts per-account agents with a different-shaped cross-account product called Watchtower. They used Managed Agents to spin up this reporting and aggregation feature quickly, and it became widely used.
- •Core product: an always-on agent per account that builds memory and drives progress
- •Sales teams also need cross-account views (prioritization, forecasting, roll-ups)
- •Watchtower required different tools/features than per-account agents
- •They built it in Managed Agents in about two weeks and it now powers other product areas
- 1:35 – 2:13
Product analytics: sandboxed code snapshots paired with real telemetry
Todd explains an agent that sits in front of their analytics platform to keep instrumentation aligned with fast-changing applications. The agent snapshots code, compares it to production behavior, and proactively sends insights and potential fixes.
- •Instrumentation can drift as teams ship faster, making analytics less reliable
- •The agent connects what the codebase is doing with what’s happening in reality
- •Managed Agents are used to snapshot code versions and compare against real application behavior
- •The system aims to proactively provide insights and even fixes back to the application
- 2:13 – 3:40
Why “Outcomes” matter: don’t show a brief unless it’s correct
The discussion shifts to outcomes as a quality gate, especially for meeting briefs. Sajjad emphasizes that incorrect briefs can be worse than no brief, so they use outcome assessment to decide whether to show results at all.
- •Briefs are only valuable when they’re correct; wrong identity/context is “almost destructive”
- •Outcomes enable independent assessment of whether the agent produced the right result
- •Source attribution is checked so surfaced info is correct
- •If the agent can’t meet the outcome, the system can choose to show nothing to avoid harm
- 3:40 – 4:52
Designing the outcomes rubric: correctness plus scannable UX
Sajjad details how they structure a rubric for evaluating pre-reads. The rubric combines objective checks (like pulling the right profile) with presentation requirements so the output is usable under time pressure.
- •Rubrics reflect what matters to end users for an effective pre-read
- •Some checks are objectively verifiable (e.g., the correct profile for the right person)
- •Other checks evaluate presentation: scannable, readable, and properly ordered
- •The rubric avoids unusable outputs like overly long pre-reads
- 4:52 – 6:51
Runtime evaluation and independent verification context windows
The host clarifies that outcomes are evaluated at runtime before the meeting, and asks how “ground truth” is injected. Sajjad explains they run briefs well in advance and rely on an independent verifier agent with a clean context window to assess correctness.
- •Brief generation and outcome grading happen ahead of time (e.g., well before the meeting)
- •Verification is decoupled: an evaluator agent assesses work independently from the agent that did it
- •Independent evaluation can cross-check signals across sources (calendar, email, profiles)
- •A clean, separate context window helps prevent the evaluator from being biased by the work trace
- 6:51 – 9:44
Memory + “dreaming”: learning org-specific logic and answering complex questions
The host turns to memory and asks Mihir about Watchtower’s use of memory and “dreaming.” Mihir explains how the system learns company-specific concepts from interactions and uses agents to execute complex cross-account queries, including programmatic tool use.
- •Sales concepts vary by company (ownership, forecasting), and underlying CRM data can be messy
- •The system learns concepts through usage and stores them as org-wide memory to reduce upfront configuration
- •For complex requests, the system can write code and make programmatic tool calls
- •Work can fan out to multiple agents per account and roll up results for cross-account answers
- 9:44 – 10:36
Two-layer memory model: per-account persistent agents plus cross-account memory
Mihir breaks down memory into persistent, account-level memory and cross-account concepts learned during broader work. They describe multiple axes of memory (account vs across accounts; user vs org) that improve different parts of the product.
- •Per-account agent runs long-lived and builds deep account context over the lifecycle
- •Cross-account work teaches higher-level concepts like forecasting and processes
- •Memory is used at different levels: account vs cross-account, user vs org
- •These memory layers collectively enhance product recommendations and reporting
- 10:36 – 12:12
Sandboxing in practice: securing source access and supporting varied sessions
The host asks Todd to expand on sandboxing. Todd explains why source code must be sandboxed (secrets, tool exposure, safe actions) and highlights that Managed Agents supported both batch and interactive workflows without them maintaining all edge cases.
- •Snapshotting source code requires sandboxing to reduce risk (e.g., secrets exposure)
- •Tool exposure and data flow must be tightly controlled when interacting with code
- •Actions can include generating pull requests back to repositories, increasing the need for safety
- •They run both batch/overnight sessions and interactive sessions for fast question answering
- 12:12 – 14:14
Build vs buy: core competency, speed, and early-stage iteration
The group discusses when to roll a custom harness versus using Managed Agents. Speakers emphasize choosing managed infrastructure when the harness isn’t the differentiator—especially during rapid experimentation—while building in-house when control is central to the product.
- •Decision hinges on whether the harness is core to the feature’s differentiation
- •Managed Agents help teams focus on delivering end-user value rather than infrastructure
- •Early-stage teams prioritize speed to validate product-market fit and iterate quickly
- •Custom infrastructure may become necessary later for fine-grained control or operating constraints
- 14:14 – 17:06
Rapid shipping timelines and what teams actually tune during iteration
The host asks about ramp-up time and what teams tweak most. Speakers share fast timelines (days to weeks) and explain they iterate on prompts, tools/skills, and orchestration steps—while benefiting from managed features like compaction and caching.
- •Todd describes migrating from a homegrown setup and rolling over within weeks after quick initial setup
- •Sajjad describes building a first version in a day and scaling rapidly with short iteration cycles
- •Mihir reiterates Watchtower shipped in about two weeks
- •Iteration focuses on prompts, tools/skills, and orchestration (search, retrieval, ranking, what to surface), while managed infrastructure removes other burdens
- 17:06 – 27:26
Evals in the real world: from vibes-based to user-driven eval sets (and hard cases)
They explore how evaluation changes over time and where it gets difficult. Sajjad outlines a progression from vibes-based iteration to structured eval cohorts and rubrics, while Mihir highlights grounding evals in real customer queries and the challenges of stateful memory and mutating tool calls.
- •Early “vibes-based” iteration helps find something users actually want before formal evals matter
- •Overfitting risk exists when only internal users provide feedback; external feedback helps build representative eval sets
- •Useful eval cohorts include regressions, hard scenarios, and task-specific partitions, each with rubrics
- •Stateful systems (memory) and third-party tools with changing state make offline evaluation difficult without heavy mocking
- 27:26 – 34:41
Model migrations, then cost and attribution: where managed agents help and constrain
The group discusses switching to new model versions and what information would help teams manage model-specific failure modes. They then move to cost, attribution, and control—exploring when managed infrastructure remains worth it, what levers teams want (batch/flex modes, cost telemetry), and where constraints appear at scale.
- •Model migration is easier with a solid eval suite; teams want visibility into model-specific failure modes more than micro-optimizing prompts
- •Cost is a universal concern; teams weigh when to keep using Managed Agents versus bringing infrastructure in-house
- •Speakers want clearer cost attribution/telemetry to see which calls drive spend, especially for batch workloads
- •Requested improvements include batch/flex options, more cost levers, and UX levers like pre-warming sandboxes for latency-sensitive flows
