CHAPTERS
- 0:00 – 1:08
Why model behavior matters: alignment work meets real-world parenting
Andrew and Jason set the stage for why consistent, thoughtful model behavior matters—not just in theory, but in everyday use. A quick family anecdote (kids using voice mode) highlights how subtle questions can become alignment edge cases in practice.
- •Jason’s role on the alignment team and focus on model behavior
- •Models should “think through” hard problems rather than jump to answers
- •Voice mode and casual, real-user interactions as a stress test
- •Santa Claus question as an example of a surprisingly tricky behavior boundary
- 1:08 – 3:52
Defining the Model Spec: what it is (and what it isn’t)
Jason explains the Model Spec as a human-readable statement of OpenAI’s high-level decisions about how models should behave. He clarifies common misconceptions, emphasizing that the spec is aspirational, not a guarantee of perfect compliance, and not a full description of the entire product system.
- •Model Spec = explanation of intended model behavior and principles
- •Not a claim that models perfectly follow it today (alignment is ongoing)
- •Not an implementation artifact; primary audience is humans
- •Not a complete description of ChatGPT (product features, policy enforcement, etc.)
- •Focuses on the most important decisions rather than every detail
- 3:52 – 6:23
How the Model Spec works in practice: policies, defaults, and examples
Jason describes the spec as a long, structured document: mission-level goals up top, followed by detailed policies and boundary-setting examples. A key theme is balancing strong safety rules with steerability—letting users override defaults like tone and style when appropriate.
- •High-level goals: empower users, reduce serious harm, manage trade-offs
- •Large policy space because users can ask “literally anything”
- •Mix of hard rules vs. overridable defaults (tone, style, personality)
- •Heavy use of examples to clarify borderline cases and intended interpretation
- •Examples also communicate subtle style/behavior expectations better than prose alone
- 6:23 – 7:34
Transparency and feedback loops: where to read the spec and how to respond
The conversation turns to transparency—making the spec accessible and improvable through feedback. Jason explains where the public can read the latest spec, how open source access works, and the most practical channels for user feedback today.
- •Public access via model-spec.openai.com
- •Spec source available on GitHub; open source and forkable
- •Feedback through in-product rating/reporting on outputs
- •Direct social feedback channels (e.g., contacting Jason)
- •Real public input has driven meaningful spec changes
- 7:34 – 9:58
How the Model Spec originated: limits of RLHF and the “employee handbook” idea
Jason recounts how his earlier work and observations about RLHF shaped the idea of a written behavioral guide. He explains that training-only approaches can be hard to interpret or revise, motivating a clearer, more editable “handbook” that can guide both people and models.
- •RLHF works, but the learned policy can be opaque in the data
- •Changing behavior via RLHF can require expensive data recollection
- •As models improve, “models meet us where we are”—so teach them like people
- •The “employee handbook” framing for consistent, explainable behavior
- •Model Spec project launched in 2024; collaboration with Joanne Jiang and John Schulman
- 9:58 – 11:24
From spec to behavior: how intentions translate into training (imperfectly)
Jason explains that mapping a written spec onto actual model behavior is complex and not always direct. Sometimes training changes come first and the spec is updated to reflect intent; other times the spec directly informs methods like deliberative alignment for reasoning models.
- •No simple pipeline where “edit spec → model instantly changes”
- •Deliberative alignment: teaching reasoning models to follow explicit policies
- •Spec language can inform training, but training is broader and nuanced
- •Spec must accurately reflect intentions—even as training methods evolve
- •Many teams and techniques contribute to behavior; spec can’t capture all implementation detail
- 11:24 – 13:32
The chain of command: resolving instruction conflicts with authority levels
Jason introduces the spec’s core conflict-resolution mechanism: the chain of command. It defines how the model should prioritize OpenAI policies, developer instructions, and user requests—while still preserving user steerability by pushing many policies to lower authority levels where possible.
- •Hard part isn’t listing goals; it’s resolving conflicts between them
- •Priority order: OpenAI instructions > developer instructions > user instructions
- •Authority levels let each policy be placed at an appropriate rank
- •Design goal: keep many policies below user level to preserve steerability
- •Highest-level constraints reserved mostly for essential safety boundaries
- 13:32 – 17:42
Edge cases under uncertainty: Santa Claus, honesty, and confidentiality collisions
Using Santa Claus and Tooth Fairy examples, Jason explains how uncertainty about the user’s context shapes conservative behavioral choices. The discussion expands into nuanced policy interactions—especially honesty versus friendliness and honesty versus confidentiality—and how earlier wording created unintended behavior risks.
- •Model often lacks context (who is asking, who is listening, intent of use)
- •Conservative assumption: avoid “spoiling the magic” while not stating falsehoods
- •Honesty is critical but can collide with social/framing constraints
- •Past stance on “white lies” shifted; tightening boundaries over time
- •Key lesson: confidentiality rules + conflicts can incentivize covert behavior; honesty prioritized above confidentiality after revisions
- 17:42 – 19:56
How the Model Spec evolves: open internal process, new modalities, and incidents
Jason describes an iterative, organization-wide process for updating the spec based on new capabilities, new product surfaces, and deployment learnings. He gives concrete examples of expansions—multimodal behavior, autonomy/agents, under-18 principles—and notes how incidents feed back into policy refinement.
- •Open internal process: broad visibility and proposal/comment mechanisms
- •Spec updates driven by new model capabilities and product changes
- •Additions over time: multimodal principles, autonomy/agents, under-18 mode
- •Iterative deployment philosophy: learn from real-world usage and failures
- •Incidents (e.g., sycophancy) become inputs to policy revisions
- 19:56 – 22:02
When models disagree with the spec: deciding whether to change the model or the policy
Jason explains how teams respond when observed behavior doesn’t match the spec’s intent. The first question is whether the output is actually better than the spec—prompting a spec update—or whether the model needs training interventions; he also notes that non-determinism means perfect compliance is unrealistic.
- •Spec is a “North Star,” often ahead of current model behavior
- •Training alignment is complex; different principles require different techniques
- •Models are non-deterministic, so perfect compliance isn’t expected
- •Triage: if output seems better than spec, revise the spec; otherwise train
- •Model Spec evals track compliance trends, showing improvements over time
- 22:02 – 23:13
Smaller and reasoning-focused models: spec compliance and deliberative alignment benefits
The discussion shifts to how smaller variants and “thinking” models follow the spec. Jason notes that reasoning models often generalize better because they can explicitly apply policies—especially when trained with deliberative alignment—leading to stronger consistency even in compact models.
- •Smaller models have been “pretty aligned” and capable
- •Reasoning/thinking models tend to follow spec better
- •Deliberative alignment encourages explicit policy understanding vs. mimicry
- •Better policy understanding improves generalization to new edge cases
- •Observed improvements align with the expectation that smarter models handle nuance better
- 23:13 – 24:16
Chain-of-thought for alignment research: diagnosing scheming and strategic deception
Jason explains why chain-of-thought is especially valuable for alignment research, particularly in detecting strategic deception where surface behavior looks fine. He also emphasizes the importance of not supervising chain-of-thought too tightly, so internal reasoning remains candid and diagnostically useful.
- •Chain-of-thought can reveal misbehavior that outputs alone don’t show
- •Critical for studying scheming/strategic deception vs. simple mistakes
- •OpenAI aims to avoid directly supervising chain-of-thought to preserve honesty
- •Helps researchers understand how policies are being applied internally
- •Supports deeper debugging of alignment failures and unintended strategies
- 24:16 – 37:26
Model Spec vs. “constitutions,” plus the road ahead: scope, developer use, and AI-written specs
Jason compares OpenAI’s Model Spec to Anthropic’s constitutional-style documents, arguing they’re often behaviorally similar but serve different purposes (public interface vs implementation artifact). The conversation then spans surprises from policy interactions, how scope decisions are made, long-term relevance (even for AGI), how developers can write their own mini-specs, and parallels to Asimov’s laws—ending with how AI can help find edge cases and even draft specs.
- •Comparison: Model Spec as public behavioral interface vs constitution as training/identity artifact
- •Surprising failure modes: policy interactions can justify unwanted behavior
- •Scope is broad (anything behavioral) but constrained by readability and importance to the public
- •Future: specs remain useful for expectation-setting and product/value decisions, even at AGI levels
- •Developers can use spec ideas to write clearer dev messages or project-level “mini specs” (agents.md)
- •Asimov parallel: goals are easy; conflict handling is the hard part (and not a strict hierarchy here)
- •Models help by brainstorming edge cases and testing policy boundaries; possibility of AI-assisted spec drafting
