CHAPTERS
- 0:00 – 0:56
Why code is easy for AI but operations is hard: digital vs real-world systems
Martin and Jad open with the core tension: AI thrives in the fully digital, well-represented world of code, but production operations live in a messy, partially observable world. They frame the key question as whether AI can ever “run” systems, not just write them.
- •AI’s native environment is digital; production systems interact with a more complex “actual world”
- •Operations depends on data, observability, and hidden state that may not be captured in training
- •The long-standing “dirty secret” of ops: humans manage distributed clusters by intuition and runbooks
- •Central challenge: converting operational reality into representations an AI can reason about
- 0:56 – 5:23
From Stanford systems research to security and MIT: Jad’s path to Grepr
Jad recounts his background from Stanford (OpenFlow and hardware/security projects) through an MIT postdoc and early application security work. The conversation highlights how systems thinking and security research shaped his perspective on reliability and operational complexity.
- •Stanford work on OpenFlow and hardware-side security; early cryptography (SHA-3) implementation experience
- •MIT postdoc with Nikolai Zeldovich and application security research
- •Early startup attempt: rollback-and-replay recovery after exploits; “too early” but conceptually relevant to ops state
- •Transition from academic depth to real-world engineering constraints
- 5:23 – 7:02
AppDynamics → a16z → Apply: scaling teams and discovering ops pain
Jad describes joining AppDynamics during hypergrowth and then briefly becoming a Cisco employee before moving to venture with Martin. He later became CPO at Apply, scaling engineering/product/security teams—setting up the perspective that led him to investigate observability and reliability spending.
- •AppDynamics growth (200→2000) and acquisition; immediate realization of different operating environments
- •Shift to VC and then back to operating leadership as CPO at Apply
- •Scaling engineering/product/InfoSec/design teams and the limits of coordination-heavy leadership roles
- •Talking to engineering leaders in 2023 surfaced a consistent theme: cut spend, especially observability
- 7:02 – 8:49
Why observability got so expensive: telemetry as a proxy for reliability
Jad explains that companies weren’t buying logs and traces for their own sake—they were buying reliability. Microservices and deeper instrumentation exploded telemetry volume, while the marginal reliability gained per dollar flattened, making observability feel disproportionately expensive.
- •Typical observability spend: ~5–20% of infrastructure costs
- •Observability is treated as “buying reliability,” historically with good ROI—until architectures changed
- •Microservices increased the number of components and telemetry layers, driving data volume faster than delivered functionality
- •Core insight: the reliability-per-dollar curve flattened, creating strong demand for a new approach
- 8:49 – 11:01
Grepr’s wedge: extract signal upstream, cut telemetry cost, enable downstream AI
Grepr’s initial product focus is reducing telemetry volume and cost by extracting signal early in the pipeline. That same signal becomes the foundation for AI-driven troubleshooting and reliability workflows layered on top.
- •Insert Grepr “in the data stream” to filter/compress and reduce observability spend (claimed ~90%)
- •Shift from collecting everything to identifying signal vs noise upstream in real time
- •Strategic positioning: clean, high-signal telemetry becomes input for future AI operations workflows
- •Second layer vision: use those signals to troubleshoot, detect issues earlier, and help fix incidents
- 11:01 – 18:26
How AI is changing engineering: great engineers pull further ahead
They discuss the evolution from autocomplete to agentic workflows and how strong engineers adapt faster. Jad argues AI amplifies skill differentials: engineers who can manage complexity, decompose work, and validate results see outsized gains.
- •Early phase: Copilot-style autocomplete; rapid transition to more capable coding assistants
- •Good engineers learn how to “work with” models and harnesses; AI effectively trains their interaction patterns
- •Skill gap widens: strong systems engineers benefit more than weaker ones, especially in distributed systems
- •AI still struggles with deep system-level reasoning unless humans provide structure, constraints, and tests
- 18:26 – 22:55
A practical AI workflow: Claude Code, Opus, persistent command-line agents, PRD-first
Jad details his hands-on setup (WSL on Windows, Claude Code, Opus) and an approach centered on design rigor. He starts with a PRD/design doc, uses agents for implementation and review, and then re-integrates changes carefully to avoid “bushy” complexity.
- •Tooling stack: Windows + WSL for Linux dev workflow; Claude Code with a strong model (Opus)
- •Key trick: command-line sub-agents with persistent sessions to reduce context loss and improve efficiency
- •Typical concurrency: ~3–4 sessions to avoid losing human context despite agent speed
- •Workflow emphasis: PRD/design doc first, tech-lead-style agent review, then parallel implementation and careful human integration
- 22:55 – 24:48
“A six-month project in ten days”: shipping production data-lake features with AI
Jad gives a concrete example: implementing “auto promotion/shredding” for semi-structured log data into columnar data-lake formats. He claims this production-grade capability took ~10 days solo, a task that previously would take ~6 months for one engineer.
- •Project: auto-promotion/shredding of semi-structured logs into queryable columns (Parquet/Iceberg context)
- •Hard parts: metadata commits, coordination across writers, correctness under concurrency
- •Claimed time compression: ~10 days now vs ~6 months previously for a single engineer
- •Review process shifts: heavier focus on tests, golden expectations, and regression detection in PR review
- 24:48 – 30:20
Where AI still fails: reasoning about performance, correctness, and workload-dependent behavior
Martin pushes on the gap between generating code and ensuring it behaves correctly under real workloads. Jad explains that models can improve when grounded in benchmarks and concrete query sets, but they cannot reliably infer resource needs or runtime behavior without experimentation.
- •Operational behavior depends on workload, access patterns, and production configuration—not just code structure
- •AI can optimize when given benchmarks, real query corpora, and explicit performance targets
- •Resource sizing (e.g., memory allocations) often requires experiments in realistic environments
- •Emerging pattern: models become useful when they know to run experiments rather than “guess” answers
- 30:20 – 33:48
From lagging alerts to leading indicators: Grepr’s incident-detection strategy
Jad critiques “AI SRE” approaches that begin with an alert and attempt root-cause backtracking through complex systems. Grepr instead looks for novel patterns and early warning messages, then uses AI to generate and test hypotheses—catching issues that can become incidents if ignored.
- •Many AI-ops tools fail because root-cause backtracking from alerts is brittle in complex systems
- •Alternative: detect what changed—new log patterns, new error signatures, early warning messages
- •Pipeline: deterministic detection of novelty/out-of-norm signals → send only high-signal items to AI reasoning
- •Early results: internal and customer trials reportedly caught signals that later became incidents when ignored
- 33:48 – 41:20
Can AI run production systems? Runbooks, visibility, and “debugging steps” as the learnable space
They debate whether operations is fundamentally too heavy-tailed for learning, or whether the key is learning the bounded process of debugging. Jad argues the world of “things that can go wrong” is huge, but the steps to narrow it down (like an SRE’s workflow) are more learnable and automatable.
- •Humans operate ops via runbooks plus tacit knowledge; AI must replace workflow, not omniscient prediction
- •Design goal: maximize visibility/telemetry so hidden state becomes inspectable
- •Architecture: separate anomaly/novelty detection from LLM reasoning—LLM focuses on hypothesis testing and investigation steps
- •Martin’s counterpoint: systems often historically avoid prediction via overprovisioning; ops automation remains harder than code
- 41:20 – 52:11
Why security is the harder problem: adversaries exploit rare states on purpose
They extend the same “signal → hypothesis → verification” approach to security, but note the fundamental asymmetry: attackers search for rare states intentionally. This makes the space more adversarial and less forgiving than operations, likely delaying full automation.
- •Security adds an intelligent adversary actively probing for edge cases and weak signals
- •Approach: detect behavioral/statistical perturbations (entities, cardinalities) and enumerate causal chains
- •AI’s job becomes structured investigation: propose scenarios, check evidence against observed data
- •Expectation: code → ops → security is an increasing difficulty gradient due to adversarial dynamics
- 52:11 – 57:05
What comes after code: constrained domains, economic equilibrium, and changing the world to fit AI
Jad and Martin broaden to a meta-view: AI scales fastest where the solution distribution matches training data, and slows in the natural world (self-driving as an example). They also discuss a recurring pattern in automation: rather than making AI handle full real-world complexity, we often redesign environments and workflows to make automation viable.
- •AI adoption accelerates in constrained domains where training and deployment distributions align
- •Natural-world domains are harder and often economically challenged (self-driving’s long timeline and unit economics)
- •Automation history: we frequently constrain the environment (robotics, warehouses) instead of solving unconstrained reality
- •Long-run view: value creation depends on economics, constraints, and tool-driven workflow redesign
- 57:05 – 58:57
Selling ops AI vs selling savings: the Datadog-bill anecdote and value clarity
Jad shares a go-to-market lesson: customers were excited about reliability AI, but what they urgently wanted was immediate observability cost reduction. They note code tools sell easily due to clear productivity ROI, while ops value is harder to price and attribute until outcomes are repeatable and trusted.
- •Customer reaction: “Save 90% of our Datadog bill” was the decisive purchase driver, not the AI story
- •Engineering productivity has straightforward ROI; ops savings/incident avoidance is harder to quantify and budget for
- •Ops AI is earlier and less “magical” today than code generation felt at its breakthrough moment
- •Closing thought: adoption hinges on measurable value, trust, and clear economic framing
