CHAPTERS
Why making AI agents usable for marketers is hard
Dylan frames the core challenge: building capable agents is difficult, but making them accessible to non-technical users (like marketers) is even harder. He sets up the talk as a tour of the friction points AirOps encountered while building AI products on Claude.
AirOps overview: AI Search as the new SEO battleground
AirOps is presented as a growth marketing platform for “AI search,” helping brands show up in engines like ChatGPT, Gemini, and Claude. The product workflow: measure how a brand appears, find gaps, take actions (create/refresh content), and measure impact.
From node-based workflows to the marketer complexity ceiling
AirOps originally orchestrated content through a traditional node-based workflow builder. For marketers, this created a steep learning curve (Liquid, JSON, variables) and brittle workflows that were hard to maintain as models and steps changed.
The pivot: agent experiences and the Opus tool-calling inflection point
The release of Claude Opus (noted as a major step-change) helped convince AirOps to invest in agents due to improved instruction-following and tool use. Early attempts included compiling doc-like instructions into workflows and using traditional orchestration frameworks, but both approaches proved brittle in different ways.
AirOps Next launch: Quill and Playbooks as the new agent platform
Dylan summarizes what AirOps launched: Quill (an agent “captain” for marketers) and Playbooks (a collaborative, governed, versioned way to build skills). The aim is to connect insights directly to action, grounded in brand context and search data.
Customer outcomes: faster time-to-value and measurable AI search gains
AirOps shares early results from customers adopting the new agentic approach. A case study (Parallel) reports major improvements in citation rate and share of voice, plus dramatically faster go-live times compared to prior enterprise workflows.
Friction point #1: endless use cases require ruthless workflow intentionality
Agents can do many things, but that flexibility can cause product sprawl and unclear user value. AirOps forces focus by anchoring on the marketer persona and mapping the real content workflow, including where human review must occur.
Designing Playbooks: document-based IDE + transparency + control
AirOps chose a document-centric building experience because it matches how marketers already work (like Google Docs). They preserved benefits of the old workflow builder by emphasizing transparency (what tools/context are used) and control over instructions, while building governance into the flow.
Demo: building skills with Playbooks (inputs/outputs, tools, MCP, triggers)
Dylan walks through the Playbook interface: natural-language construction, slash commands for structure, and integration with tools and external connectors (MCP). He also highlights automation via schedules, webhooks, and monitoring triggers tied to internet events or AEO metric changes.
Human review and governance in practice: Inbox approvals and Grid at scale
AirOps enforces human review by assigning approvers at section boundaries, making a specific user the gatekeeper to unblock an agent run. Governance is surfaced through an Inbox (review tasks) and a Grid for running and reviewing many jobs in parallel with collaboration.
Friction point #2: consistency and quality via harness engineering
The second major challenge is reliability: moving from deterministic workflows to agents raises consistency concerns. Dylan introduces a “car and engine” metaphor—Claude is the engine, but tools, context orchestration, and harness design determine real-world performance.
Making tools more deterministic: from primitive tool spam to specialized tools
AirOps reduced token waste and unpredictability by building specialized tools that encapsulate common repeated work (e.g., diagnosing page performance, benchmarking vs competitors). This replaces long agent “safari trips” across many small tool calls with fewer, structured, higher-signal calls.
Sub-agents for quality: compliance, writing focus, and Brand Kit context artifacts
To improve output quality without “context rot,” AirOps introduced focused sub-agents. These include a compliance checker (brand rules), a dedicated writer (narrow scope), and a Brand Kit fetcher that caches consistent context as an artifact for reuse across the run.
Results and what’s next: self-improvement loops and benchmarking taste-based content
AirOps reports operational gains: fewer tokens, faster runs, and faster enterprise publishing with self-serve workflows. Dylan closes with two upcoming friction areas: building feedback/self-improvement loops (memory, summarizing traces, forgetting) and benchmarking content agents where “correctness” is subjective.
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome