CHAPTERS
- 0:00 – 0:30
From Casetext to GPT-4: Why legal AI finally “worked”
Jake Heller introduces his background founding Casetext and explains how early access to GPT-4 changed what was feasible in legal workflows. He frames GPT-4 as the first model that could perform complex legal tasks at roughly human reliability while being faster and scalable.
- •Casetext spent years building AI for legal work before GPT-4
- •Early GPT-4 access revealed a step-change vs. GPT-3/3.5
- •Complex legal tasks became possible at near-human quality
- •Speed + parallelism made the product economically compelling
- 0:30 – 1:31
Pivoting to CoCounsel: productizing an AI assistant lawyers asked for
The team pivoted the company around GPT-4 to build CoCounsel, positioned as an AI assistant for lawyers. Jake emphasizes that the product succeeded largely because it delivered capabilities customers had long requested but were previously impossible.
- •CoCounsel built around GPT-4 capabilities rather than incremental AI features
- •Market pull: customers had asked for an “AI assistant” for years
- •Framed as the first assistant that can execute lawyer-like tasks
- •Success came from aligning the product with real user demands
- 1:31 – 2:31
Proof of capability: bar exam leap and real-world legal performance
Jake cites Casetext’s bar exam research to illustrate the capability jump from GPT-3.5 to GPT-4. He uses this to justify why building a serious legal product became viable at that moment.
- •GPT-3.5 ~10th percentile vs. GPT-4 ~90th percentile on the bar exam
- •Exam reportedly wasn’t in the training set (contrast with many modern evals)
- •Model reliability crossed a threshold for practical deployment
- •Performance unlocked a new class of complex legal workflows
- 2:31 – 3:01
Acquisition by Thomson Reuters and continued iteration on prompting
Following CoCounsel’s traction, Casetext was acquired by Thomson Reuters in 2023. Jake notes they continued refining what people call “prompt engineering,” setting up the talk’s focus on repeatable techniques.
- •Acquisition as a milestone enabled further scaling and refinement
- •Prompting techniques evolved with real production constraints
- •Framing: lessons learned from deploying at scale, not just demos
- •Focus shifts from story to methodology
- 3:01 – 3:32
Context engineering vs. prompt engineering: instructions + context
Jake challenges the meme of “context engineering,” arguing most meaningful prompts consist of instructions plus context. He treats the label as semantics but uses it to highlight how often teams underweight the context side.
- •Most prompts of consequence = instructions + context
- •“Context engineering” isolates one part of the real problem
- •Semantics matter less than disciplined practice
- •Sets up deeper discussion on why context quality drives outcomes
- 3:32 – 4:34
Step 1 of building AI workflows: define the customer experience and ‘skills’
Jake’s first macro step is designing the customer experience, which for CoCounsel meant a suite of tool-like “skills.” He describes these as lawyer-resume-style capabilities (research, doc review, contract analysis) implemented in a chat UI with tools.
- •Start from the ideal user experience, not model capabilities
- •CoCounsel organized functionality into discrete “skills”/tools
- •Examples: legal research, multi-document review, contract redlining
- •The UI can vary (chat, button, upload-analyze), but UX intent must be clear
- 4:34 – 6:04
Step 2: map each skill to how the world’s best expert would do it
For each skill, Jake recommends modeling the process after the best human expert with unlimited time. Using legal research as an example, he breaks the task into clarification, query formulation, searching, reading results, note-taking, and synthesis.
- •Use ‘world’s best human’ workflow as baseline architecture
- •Legal research flow: clarify → generate queries → execute → review → notes → final answer
- •Micro-steps define what the system must do, in order
- •Human-like decomposition helps identify required tools, prompts, and checks
- 6:04 – 7:36
Linear vs. agentic execution: when to hard-code steps vs. let the model roam
Jake distinguishes between predictable, linear tasks and more adaptive, agentic loops. If a task is repeatable, he suggests implementing it deterministically; if it requires backtracking (e.g., search refinement), agentic behavior can be appropriate.
- •Prefer deterministic pipelines when the task is predictable
- •Agentic loops are useful when the system must revisit earlier steps
- •Think in code-like functions for linear sequences
- •Architecture choice affects reliability, cost, and debuggability
- 7:36 – 9:07
Step 3: make each micro-step work—prompt iteration with evals
Once micro-steps are identified, each becomes code or a prompt; for hard tasks, prompts dominate. Jake’s core method is straightforward: write your best prompt, define objective evals, run them, and iterate relentlessly until it passes.
- •Each micro-step must have a clear job in the overall flow
- •Start with a best-guess prompt for one micro-task (e.g., craft a query)
- •Create initial evals (e.g., 10) with clear success criteria
- •Iterate until all pass; then scale eval coverage to 50/100/1000
- 9:07 – 10:08
Tooling and test-driven prompting: prompts are production code
Jake argues many teams fail by not treating prompts like code with tests. He recommends using tools (e.g., Promptfoo, Vellum) to run prompts across datasets, vary models/settings, and track pass/fail systematically.
- •Use prompt-eval tooling to run repeatable tests at speed
- •Promptfoo (open-source CLI) and paid tools can manage experiments
- •Prompts should be validated against representative inputs, not vibes
- •Model + settings are part of the test matrix, not afterthoughts
- 10:08 – 12:39
The grind: why ‘6/10’ isn’t failure—iterate, tune settings, anticipate user chaos
Jake describes the common moment where an early prompt only passes some tests and teams give up. He emphasizes persistence, careful instruction writing, tuning parameters (temperature, etc.), and expanding eval sets using real user inputs—because users will surprise you.
- •Most first prompts partially fail; don’t interpret as ‘AI is dumb’
- •Rewrite unclear instructions; adjust model choice and parameters
- •User inputs are messier than you expect—even from professionals
- •Beta programs can supply edge cases to convert into regression tests
- 12:39 – 15:54
Why context quality dominates: retrieval, OCR, and ‘see what the model sees’
Jake stresses that failures often come from missing or corrupted context rather than bad prompting. He advises inspecting the exact retrieved text (including OCR output and formatting) and reminding the model to rely only on provided context to reduce hallucination.
- •Model outputs can be ‘wrong’ because context is incomplete or misleading
- •Instruction: answer only from provided context (reduces hallucination)
- •Weak retrieval turns into a product problem, not a prompt problem
- •OCR errors and messy legal docs can make context unreadable; inspect raw inputs
- 15:54 – 17:55
Speed and reliability tricks: constrain outputs, stop early, and structure for evals
Jake shares pragmatic techniques to reduce latency and improve consistency by limiting generated tokens. He recommends outputting an objective scalar/label first (often one token), using stop sequences or max_tokens to cut off explanations, and structuring outputs to make evaluation easy.
- •Latency scales with output tokens; minimize generation when possible
- •Have the model ‘think’ it will explain, then stop after 1 token (stop words/max_tokens)
- •Ask for objective outputs first (number/true/false) to simplify evals
- •Break complex tasks into smaller prompts for higher correctness
- 17:55 – 20:23
Beyond prompting: reinforcement fine-tuning, per-step models, and model routing
Jake closes with advanced levers: reinforcement fine-tuning can outperform older fine-tunes with far fewer examples if you can define good objective judgments. He also recommends using different models for different steps to optimize cost and accuracy, potentially assigning specialized models per micro-step.
- •Older fine-tunes often yielded minimal gains; reinforcement fine-tuning performs better
- •RFT may need ~50–100 examples plus a clear objective judge
- •Strategy: potentially a distinct tuned model per micro-step/prompt
- •Use model routing across steps to balance accuracy and cost (e.g., smaller/cheaper models where acceptable)
