CHAPTERS
- 0:16 – 0:46
Why “making agents accessible” is hard (especially for marketers)
Dylan frames the core thesis: building agents is difficult, and the difficulty spikes when the audience isn’t technical. He sets expectations that the talk will focus on concrete friction points AirOps encountered while productizing agent experiences for marketers.
- •Agents are challenging even for developers; harder for non-technical personas
- •Goal: identify and remove friction points in real product workflows
- •Talk focus: lessons from building AI products with Claude for marketers
- 0:46 – 1:47
What AirOps does: AI Search growth marketing (SEO for ChatGPT/Claude/Gemini)
AirOps is positioned as an AI search growth platform—helping brands understand how they show up in LLM-driven discovery, find gaps, take action, and measure impact. Dylan clarifies the “AI search” problem as an emerging analog to traditional SEO.
- •AI search = discoverability in engines like ChatGPT, Gemini, Claude
- •Identify how a brand appears, locate gaps, and recommend actions
- •Measure impact of content updates/creation on AI search outcomes
- 1:47 – 2:17
Roadmap for the talk: approach, launch recap, and friction points
Dylan previews the structure: how AirOps evolved, their approach to agents, what launched in AirOps Next, and the key friction points they tackled. He also teases upcoming friction areas they’re actively exploring.
- •Background: how the product and approach evolved
- •AirOps Next launch overview
- •Two core friction points addressed in the launch
- •Additional future friction points to tackle next
- 2:17 – 4:18
Legacy workflow builder pain: complexity ceilings and brittle systems
AirOps started with a node-based workflow builder for orchestrating content, but marketers hit a complexity wall. The system became brittle to maintain as models and steps changed, and enterprise scaling required technical support.
- •Node-based workflows required concepts like JSON/liquid templates
- •Workflows had short shelf life as models and steps changed
- •Small edits broke downstream dependencies (variables referenced later)
- •Enterprise use cases needed technical guidance to scale
- 4:18 – 5:49
The shift toward agents: model capability and early attempts that didn’t stick
The release of stronger models (e.g., Opus 4.5) made tool-calling and instruction-following reliable enough to rethink UX. AirOps experimented with document-to-workflow compilation and traditional orchestration frameworks, but found them brittle for iteration.
- •Better models improved tool calling and instruction adherence
- •Tried doc-like builder that compiled workflows in the background—too brittle
- •Traditional orchestration frameworks worked but required code changes to evolve
- •Need: more flexible orchestration and faster iteration
- 5:49 – 6:19
Why Claude Agent SDK: orchestration via Markdown + skills without heavy code
AirOps leaned into the Claude Agent SDK to reduce brittleness and speed up iteration. Dylan highlights the benefit of defining and adjusting agent behaviors through Markdown/context manipulation instead of hard-coded routing graphs.
- •Agent orchestration becomes more configurable through files/contexts
- •Easier to add skills and contexts without rewriting routing code
- •Better fit for rapid product iteration and enterprise requirements
- 6:19 – 7:20
AirOps Next launch: Quill (agent captain) + Playbooks (document-based skills)
Dylan summarizes what shipped: Quill as the marketer-facing agent with access to AirOps data and brand context, and Playbooks as a new skill-building surface. The focus is moving from insights to actions while enabling collaboration, governance, and versioning.
- •Quill: agent captain grounded in AI search data + Brand Kit
- •UI designed to move users from dashboard insights to actions quickly
- •Playbooks: document-style builder for marketer-friendly “skills”
- •Collaboration, governance, and versioning for enterprise workflows
- 7:20 – 9:21
Customer outcomes and proof points: faster time-to-value and better AI search visibility
AirOps reports early results: a case study with Parallel showing citation/share-of-voice improvements and rapid go-live. Quotes reinforce the perception shift from “LLM tool” to “strategic teammate” and delegation of tedious work.
- •Parallel: +130% citation rate, +42% share of voice, live in one week
- •Enterprise workflows that used to take ~a month compressed dramatically
- •Agents described as a “mid-level strategist” by customers
- •Value: offload tedious tasks; humans add domain expertise and context
- 9:21 – 11:22
Friction point #1: endless use cases → forcing intentionality around marketer workflows
Agents can sprawl into limitless possibilities, which creates confusion for product teams and users. AirOps narrowed scope by anchoring to the actual content marketer workflow and the moments where humans must review and guide outputs.
- •Powerful agents create use-case sprawl; product must constrain intentionally
- •Anchor design to a clear persona: content marketers
- •Map the workflow: discover → research → brief → draft → SEO/AEO polish
- •Human review is central to responsible brand publishing
- 11:22 – 13:23
Design response: document-based IDE + transparency + control + enforced human review
AirOps prioritized familiarity (docs), while preserving what users liked about workflows: visibility into tools and context. Governance was built into the agent flow with explicit checkpoints and accountability.
- •Document-based “IDE” (Playbooks) matches how marketers already work
- •Transparency: show which tools and context are used at each step
- •Control: keep instructions editable and understandable even if non-deterministic
- •Governance: configurability, accountability, and human-in-the-loop checkpoints
- 13:23 – 14:55
Demo walkthrough: Playbooks, connectors (MCP), triggers, monitoring, and AEO alerts
Dylan demos how Playbooks are authored with natural language and structured inputs/outputs, plus tools marketers rely on. He shows how automations run via schedules/webhooks, “watch the internet” monitoring, and trigger on AEO metric drops for diagnosis.
- •Playbook authoring via slash commands; define inputs/outputs and tools
- •Supports MCP connectors for external integrations
- •Triggers: scheduled cadence or webhooks for always-on execution
- •Monitoring: watch queries/changes on the web to trigger runs
- •AEO insights: trigger when metrics (e.g., citation rate) drop and investigate
- 14:55 – 16:56
Governance in practice: approvals, inbox-driven review, and grid-based scale execution
Human review is enforced by assigning approvers at section boundaries, where only designated users can unblock progress. Reviews surface in an inbox and in a “grid” that runs skills at scale, enabling collaboration and consistent tone/quality checks across many outputs.
- •Section-level gates: assign approvers; only they can unblock the agent
- •Inbox surfaces review tasks and opportunities with run context
- •Run view: agent trace on one side; outputs/artifacts on the other for edits/comments
- •Grid: execute playbooks at scale; review and feedback across many jobs
- 16:56 – 18:27
Friction point #2: consistency and quality—harness engineering via tools + context orchestration
The key reliability concern is output consistency compared to deterministic workflows. Dylan frames “harness engineering” as everything around the model—especially tool design and context orchestration—accelerated by Claude Agent SDK and Managed Agents.
- •Consistency is the biggest fear when moving from workflows to agents
- •Harness engineering: model is the engine; quality comes from surrounding system
- •Primary levers: tools and context orchestration
- •Claude Agent SDK + Managed Agents API help iterate quickly and reliably
- 18:27 – 20:28
Quality lever #1: specialized tools over primitive tool-call “safari trips”
AirOps moved from many small, token-inefficient tool calls to specialized tools that return structured, comprehensive context in one shot. Examples include diagnosing page issues and benchmarking against top-ranking competitor pages.
- •Primitive tools caused long, inefficient exploratory loops
- •Specialized tool: analyze a URL and return page details + structured gaps + targets
- •“Page vs” benchmarking tool compares against top-ranking pages to close gaps
- •More deterministic context retrieval and a more “code mode” approach
- 20:28 – 23:29
Quality lever #2: sub-agents to avoid context rot and focus workstreams
Sub-agents improved output quality by isolating tasks into focused contexts. AirOps added compliance checks, a dedicated writing agent, and a Brand Kit agent that fetches and stores consistent brand context as artifacts reused across the run.
- •Start simple; add sub-agents only when quality issues emerge
- •Compliance sub-agent scores adherence to brand rules without polluting main context
- •Writing sub-agent focuses solely on drafting, not research/history distractions
- •Brand Kit sub-agent fetches relevant brand context once, stores as artifact
- •Lesson: larger context windows don’t remove the need for context efficiency
- 23:29 – 26:46
Measured impact and the moving target of friction; what’s next
Dylan shares tangible improvements: reduced tokens, faster runs, and quicker enterprise publishing during beta. He closes by emphasizing that solving one friction point reveals the next, highlighting future work on feedback loops/self-improvement and benchmarking creative outputs.
- •Results: ~8% fewer tokens for a key tool; faster execution via single-call context
- •Beta: 10 enterprise customers publishing in under two weeks
- •Friction keeps moving—continuous iteration is required for production agents
- •Next: self-improvement/feedback loops (trace summaries, memory, forgetting)
- •Next: benchmarking content agents where correctness is subjective and taste-driven
