By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

AI prompt engineering: A deep dive

Some of Anthropic's prompt engineering experts—Amanda Askell (Alignment Finetuning), Alex Albert (Developer Relations), David Hershey (Applied AI), and Zack Witten (Prompt Engineering)—reflect on how prompt engineering has evolved…

Alex AlberthostDavid HersheyguestAmanda AskellguestZack Wittenguest

Sep 5, 20241h 16mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

CHAPTERS

Jump to publisher timestamps

Chapter summaries

Roundtable setup: prompt engineering from research, consumer, and enterprise angles
Alex Albert frames the session as a wide-ranging discussion of prompt engineering, explicitly aiming to compare perspectives across research, consumer use, and enterprise deployment. The panel then introduces themselves and the kinds of prompting-related work each person does.
- •Alex Albert sets the goal: explore what prompt engineering is and why it matters across contexts
- •Alex describes his role in developer relations and past work on a prompt engineering team
- •David Hershey shares that he helps customers adopt language models, including prompting and system-building
- •Amanda Askell introduces her focus on fine-tuning and making Claude "honest and kind"
- •Zack Witten describes work ranging from customer prompting to educational materials like a prompt generator
Defining prompt engineering: communication, iteration, and systems integration
Zack defines prompt engineering as getting the most out of a model through clear communication and experimentation. The group expands the definition to include iterative trial-and-error and integrating prompts into a broader system, not just writing a single instruction.
- •Zack frames prompting as clear communication and understanding how the model interprets instructions
- •The "engineering" aspect is linked to rapid trial-and-error enabled by a restart/clean-slate workflow
- •Zack emphasizes that prompts often must be integrated into a larger system, not used in isolation
- •David describes prompts as a way to "program" models, involving data availability and system constraints
- •The group notes prompts can be treated like code: versioning, tracking experiments, and managing changes
What makes a good prompt engineer: iteration, edge cases, and realistic user behavior
Amanda lays out qualities she looks for in strong prompt engineers, emphasizing iteration and anticipating failure modes. David adds that real-world usage often differs sharply from idealized inputs, so prompts must be tested against messy, realistic user behavior.
- •Amanda highlights clear communication, willingness to iterate rapidly, and diagnosing misinterpretations
- •She argues being a good writer isn’t as tightly correlated with prompting skill as people assume
- •A core skill is anticipating edge cases and testing unusual inputs (missing data, empty strings, unexpected formats)
- •David notes enterprise chat inputs are often typo-filled and poorly formatted compared to what teams expect
- •Alex stresses closely reading model outputs (not just issuing instructions) to catch mismatches like superficial "step by step" compliance
Making instructions complete: stripping assumptions and using the model to critique the prompt
The panel discusses how prompts frequently fail because the author leaves unstated assumptions in their head. Amanda describes tactics for using the model itself to point out ambiguities and even propose revisions after mistakes.
- •David describes the challenge of writing down all necessary context the model doesn’t share
- •He observes many prompts only make sense to the author, not to an outside reader (or the model)
- •Alex notes models often won’t ask clarifying questions unless explicitly instructed to do so
- •Amanda has the model list ambiguities instead of following instructions on the first pass
- •Amanda sometimes asks the model why it failed and to rewrite the instructions to prevent the error; she says it works often enough to be worth trying
When to trust outputs: high-signal evaluation, consistency checks, and the “mythical better prompt”
Amanda explains she doesn’t trust the model by default; she stress-tests it and looks for consistent behavior across carefully chosen cases. The group also discusses knowing when to stop iterating—when a “perfect prompt” is unlikely to exist for the task.
- •Amanda describes "hammering" on the model to determine reliability, especially in unusual areas
- •She argues a well-constructed set of a few hundred tests can be higher-signal than thousands of loose ones
- •David emphasizes learning from the full text output—how the model got there, not just pass/fail
- •Amanda says prompt quality can determine whether an experiment succeeds or fails at the margin
- •David warns about getting stuck chasing a "mythical better prompt"; they discuss checking whether the model is in the right “ZIP Code” of understanding
Case study: Claude plays Pokémon—prompting limits, multimodal friction, and deciding to wait for better models
David describes an experiment connecting Claude to a Pokémon Red emulator and the difficulty of getting reliable visual interpretation from screenshots. The discussion becomes a concrete example of how prompting can improve performance incrementally, but also how to recognize hard capability ceilings.
- •David explains hooking Claude to an emulator and having it press buttons based on its interpretation
- •He reports the model struggled with screenshots; iteration moved from "no signal" to "some signal" but not to reliability
- •David describes a workaround: overlaying a grid, describing segments, reconstructing an ASCII map, and anchoring the player location
- •Zack notes that some text prompting intuitions don’t transfer well to images; he finds multi-shot less effective in multimodal settings
- •David highlights a breaking point: inability to consistently identify/describe NPCs, making continuity impossible; he chose to stop and wait for better models
Honesty vs personas: role prompting, metaphors, and giving models “outs” for weird inputs
The panel debates common prompting advice like assigning the model a persona, with Amanda favoring direct, honest context and task descriptions. They distinguish between misleading roleplay and helpful metaphors, and discuss designing prompts that let the model safely say it’s unsure.
- •Alex asks whether honesty beats personas and incentives; Amanda says capable models don’t need deception and she prefers direct task framing
- •Amanda argues that if the task is to build an eval, it’s better to say so than pretend it’s a school quiz
- •Alex shares a case where a metaphor (grading a chart like a high school assignment) helped specify the evaluation standard
- •David warns role prompting can become a shortcut that omits key product/task context; he prefers explicitly describing the real system context
- •Amanda emphasizes adding “outs” for edge cases (e.g., output an "unsure" tag) to avoid forced, confident nonsense and to surface bad inputs
Model reasoning and chain-of-thought: usefulness, limits of personification, and structuring reasoning
Alex raises whether chain-of-thought is “real” reasoning or a useful scaffold; the group focuses on pragmatic value rather than philosophy. They agree structured reasoning often improves results, while noting reasoning text can sometimes be inconsistent with final answers.
- •David says debating whether it’s “reasoning” can distract; he focuses on the fact it improves outcomes
- •He observes performance improves further when you structure reasoning and iterate on how it should be done
- •Alex suggests a test: swapping in plausible-but-wrong reasoning to see if conclusions change; he notes related internal experimentation
- •They discuss that reasoning text can contain wrong steps yet still reach correct answers, and that humans can do this too
- •Alex notes non-reasoning filler (e.g., random tokens) doesn’t help the way reasoning does, suggesting it’s not just extra computation space
Prompt format, typos, and pre-trained vs RLHF behavior: when polish matters
Zack and the group discuss whether grammar and punctuation matter, distinguishing between iteration-phase messiness and final prompts. They contrast how pre-trained models may mirror typos more, while RLHF-tuned models often infer what the user wants stylistically.
- •Zack says correctness in grammar/punctuation isn’t strictly necessary, but attention to detail often correlates with better prompts
- •He’s comfortable iterating with messy prompts but would clean up a final prompt as an easy check
- •David highlights that pre-trained models are more likely to continue typos once they appear (mirroring text patterns)
- •They discuss that RLHF models are trained to infer desired style and polish (e.g., not returning typos even if the user wrote them)
- •Alex mentions using pre-trained-style behavior to generate realistic typo-filled inputs for testing user traffic patterns
Enterprise vs research vs chat prompting: examples, diversity vs reliability, and scale of deployment
The panel compares prompting goals across research, consumer/enterprise systems, and one-off chat use. Differences include the need for reliability and formatting at scale, versus exploring diverse behaviors in research, and the role of extensive examples.
- •Zack contrasts research prompts (seeking diversity/range) with product prompts (seeking reliability and consistent formatting)
- •He notes product prompts often include many examples; research may avoid examples that cause the model to latch onto narrow patterns
- •Amanda says when she uses examples, she may make them intentionally illustrative and unlike the real data to preserve flexibility
- •David distinguishes one-off chat prompting (iterate until it works once) from enterprise prompts (must work across huge volumes and varied inputs)
- •They note chat keeps a human in the loop; enterprise designs must handle the “minimum effort” user and broad input distributions
Improving prompting skills and the future: pushing boundaries, model-assisted prompting, and a flipped relationship
The group shares practical ways to get better—reading good prompts, repeated practice, and attempting tasks near the model’s limits. They close by projecting how prompting may evolve as models improve: more collaboration, model-led elicitation, and a shift from “temp worker” to “expert consultant” dynamics.
- •David recommends reading strong prompts and closely reading outputs to understand why they work
- •Amanda suggests giving prompts to someone without context, practicing repeatedly, and enjoying the iterative craft
- •David advises trying to make the model do something you think it can’t, because boundary-pushing teaches the most
- •They discuss jailbreaks as a mix of out-of-distribution inputs, training-data quirks (e.g., multilingual behavior), and understanding next-token prediction dynamics; Amanda says she isn’t sure of a full explanation
- •On the future, they predict more model-assisted prompt creation (meta-prompts, interview-style elicitation) and a shift where models ask clarifying questions like an expert designer working with a vague client brief

Publisher timestamps

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.