ClaudeBuilding the best agentic analytics harness: Powered by Claude, built with Claude Code
CHAPTERS
- 0:21 – 2:22
How Claude Code changed Omni’s engineering velocity
Chris Merrick (Omni CTO) opens by describing how Claude has materially increased engineering output and helped him continue coding as the company scales. He frames 2025 as a deliberate experimentation period that turned into sustained acceleration once Claude Code (Opus) landed.
- •Omni is an AI analytics platform; speaker is CTO leading ~25 engineers
- •Claude adoption boosted commit velocity and overall throughput
- •CTO can still contribute meaningful code despite leadership responsibilities
- •Company-wide directive: experiment early because engineering jobs are changing
- •Claude Code + Opus marked the moment it became consistently useful
- 2:22 – 3:23
Omni’s culture of shipping and radical transparency (weekly demos)
He connects increased speed to Omni’s values—especially “ship it” and transparency. Omni institutionalizes demos via recorded, publicly shared Friday all-hands sessions.
- •Velocity is a core cultural value (“ship it”)
- •Transparency is operationalized through public demo sharing
- •Weekly all-hands: brief updates + extended demos (often >50 minutes)
- •CEO edits and posts demos to YouTube; accessible via omni.co
- •Public demos help customers and the community see how Omni builds
- 3:23 – 4:23
From natural language to semantic query: the high-level system flow
Chris explains how Omni uses Claude to translate a user’s question into a semantic query, which then becomes SQL executed against a warehouse. The key challenge is providing Claude the business- and data-specific context required for reliable answers.
- •User asks a question; Claude translates it into a semantic query
- •Semantic layer sits atop one or multiple databases/warehouses
- •Semantic query is translated to SQL and executed to get results
- •LLMs need business terminology and data shape to answer correctly
- •Omni’s product focus: bridge business context to data execution
- 4:23 – 4:54
Why business context is hard: definitions, terminology, and ambiguity
He highlights subtle but critical ambiguity in real organizations (e.g., what “last quarter” means). These differences must be encoded into definitions and context for consistent analytics outcomes.
- •Business terms vary across companies and even within departments
- •Example: “last quarter” can mean calendar vs fiscal quarter
- •Correct answers require encoded definitions, not just raw schemas
- •Analytics accuracy depends on shared organizational semantics
- •Semantic modeling becomes essential for dependable AI answers
- 4:54 – 6:55
What the semantic layer does: curation, localized context, and permissions
Chris breaks down the semantic layer’s responsibilities beyond simple metadata. It curates which tables matter, localizes context near definitions (similar to Claude.md close to code), enforces permissions, and supports continual improvement via feedback loops.
- •Curates messy warehouses with tens/hundreds of thousands of datasets
- •Reduces confusion from many overlapping ‘revenue’ or ‘opportunity’ tables
- •Localizes context next to field definitions for better LLM performance
- •Acts as a permissions layer to enforce data access controls
- •Feedback loop updates definitions as questions and org needs change
- 6:55 – 7:56
Meet Blobby: the agent and its evolution
He introduces Omni’s analytics agent, Blobby, and emphasizes it evolved significantly over ~18 months. The talk focuses on the phases and lessons that improved Blobby’s quality and capabilities.
- •Blobby is Omni’s agentic analyst experience
- •It has matured through iterative development over 18 months
- •Goal: explain phases of development and key learnings
- •Quality improvements came from both product and harness design
- •Sets up a demo to ground what users actually experience
- 7:56 – 9:27
Demo walkthrough: how Blobby answers a real question end-to-end
A short demo video shows Blobby interpreting terminology, finding the right semantic entities, resolving filter values, generating SQL, visualizing results, and summarizing findings. Chris calls out fuzzy matching and semantic lookup as practical necessities in real usage.
- •Understands domain shorthand (e.g., PRs → GitHub pull requests)
- •Maps the question to the right semantic model entities
- •Finds valid filter values and applies constraints (repo selection)
- •Handles typos via fuzzy matching for robust user input
- •Generates query → runs in warehouse → returns viz + narrative summary
- 9:27 – 10:58
Phase 1 improvements: AI-specific context, sample queries, and example values
Early Blobby was ‘single question, single answer’ and lacked sufficient metadata to behave reliably. Omni added AI context, sample queries, and representative field values to better ground Claude and improve answer quality.
- •Initial system needed richer guidance than labels/descriptions
- •Added “AI context” to steer LLM usage of fields/datasets
- •Included sample queries to anchor common analytic patterns
- •Added representative values to help interpret abbreviations and categories
- •Example: region codes enable mapping from ‘United States’ → ‘US’
- 10:58 – 11:59
Becoming truly agentic: task loops, error recovery, and better error messages
The next leap was adding an agentic loop with tasks and retries. Major quality gains came from explicit recovery behavior plus investing in clear, actionable error messages that the agent could use to self-correct.
- •Built a custom agentic harness with task concepts
- •Agentic loops excel at recovering from failures when given budget/tools
- •Explicitly taught Blobby how to recover from errors
- •Improved error message quality to enable self-correction
- •Hard evals improved significantly after these changes
- 11:59 – 13:00
Model shift and adoption spike: Haiku to Sonnet for longer agent conversations
As conversations became more complex, Haiku was no longer a fit, prompting a move to Sonnet. Token usage increased (by design), but customer value increased sharply as Blobby solved harder questions quickly and reliably.
- •Haiku worked for simpler Q&A but not complex agentic workflows
- •Moved to Sonnet to handle longer, richer reasoning loops
- •Token consumption rose due to longer sessions and increased usage
- •Customers reported ‘hours of work’ answered in minutes
- •Usage growth validated the investment in deeper agent behavior
- 13:00 – 16:33
Debugging with traces: ‘blobotomies’ and the split-brain sub-agent lesson
Customer pressure for consistency drove Omni to invest in trace inspection and observability. Traces revealed structural issues in a two-agent design (planner vs query generator), leading to consolidation to avoid a ‘split brain’ and improve predictability.
- •CEO demanded fixes for incorrect sessions; ‘unpredictable’ wasn’t acceptable
- •Traces/observability exposed root causes behind bad sessions
- •Original design: outer agent planned tasks; sub-agent generated one query
- •Mismatch: outer agent asked for work requiring multiple queries
- •Solution: ‘consolidate the brain’ by pulling tools into the outer harness
- 16:33 – 19:10
Unlocking better queries: switching from a proprietary JSON query to SQL parsing
Omni noticed Claude could write stronger SQL than Blobby’s structured JSON query format allowed. They revived an old SQL parser, constrained Claude’s output to parseable forms, and gained one-shot, more expressive queries (often using CTEs).
- •Claude often outperformed Blobby on hard SQL generation tasks
- •Omni had previously built (then shelved) a SQL parsing engine
- •Reframed interface: let Blobby emit SQL instead of proprietary JSON queries
- •Added guidelines to keep SQL within parser-friendly constraints
- •Benefits: fewer multi-step query chains; more efficient one-shot answers
- 19:10 – 20:40
Current architecture and tooling roadmap: checkpointing, expanding tools, and evals
Chris summarizes the present-day harness: an outer loop with checkpointing and an inner loop with an expanding toolset. He emphasizes evals not just for scoring but for trace-driven diagnosis and is building eval tooling for customers to ensure repeatable quality.
- •Outer loop handles checkpointing and failure recovery
- •Inner loop offers growing tools: dashboards, visualizations, validation
- •Tools also help improve the semantic layer and data modeling
- •Evals are central for predictability (same question → same answer)
- •Primary value of evals: trace observability to diagnose failures quickly
- 20:40 – 22:11
Building better harnesses by being users: lessons from Claude Code
He argues engineers build better agent systems when they personally understand what users value. Using Claude Code internally taught Omni what good harness UX looks like and inspired analogous approaches to exploring semantic models as if they were codebases.
- •Engineers need empathy for end-user goals to build great systems
- •Using Claude Code provided practical harness design intuition
- •Semantic models resemble codebases in how they’re explored and understood
- •Omni borrows patterns from Claude Code to improve its own harness
- •Claude is both a productivity tool and a product-design teacher
- 22:11 – 26:14
Live demo: AI-generated dashboards plus UI-based validation and troubleshooting
Chris demonstrates creating a dashboard of engineering activity via Blobby and then validating/refining outputs in Omni’s UI. He highlights the product philosophy: AI to build quickly, UI to inspect, adjust, and troubleshoot when something is off.
- •Blobby can plan and generate a multi-tile dashboard (many queries)
- •Uses ‘topics’ (domain-wide datasets) to find relevant data sources
- •Chat results can be opened in a workbook for hands-on exploration
- •Users can riff: change filters (e.g., switch repositories) quickly
- •UI supports troubleshooting (e.g., diagnosing a blank chart)
- 26:14 – 26:45
Wrap-up: Omni’s Claude-optimized agentic analytics platform
He closes by positioning Omni as an AI analytics platform designed specifically around Claude’s strengths. He notes the team, location, and invites attendees to connect after the talk.
- •Omni’s harness is optimized for Claude and Claude model family
- •Agentic analytics depends on semantic context + robust tooling
- •Omni credits strong customers and an exceptional engineering team
- •Company is based in San Francisco
- •Invitation to chat and grab a Blobby sticker
