By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Latent SpaceOpenAI’s New Agent Stack: Computer Use, Decisions API, UltraFast, Dots—Ari Weinstein & Nikunj Handa
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:00 – 1:16
Computer Use is nearing “superhuman” speed—and OpenAI is racing to ship it
Ari describes a recent jump in Computer Use performance: it’s now often faster than an average human at completing software tasks. The hosts react to how quickly this capability moved from prototype hacking to an imminent launch, with the team focused on shaving latency.
- •Ari says Computer Use is now faster than the average human in many cases and aims next for expert-level or superhuman performance
- •The hosts emphasize how sudden the development was, with recent work driven by internal hacking and rapid iteration
- •Ari describes a prototype built quickly by infra/inference folks and the current focus on “hill climbing” latency
- •The plan is to launch once a latency target is met
- 1:16 – 2:46
DevDay recap: Dots, GPT‑6.1 Sol, Agents API w/ Computer Use, App Shots, and native Mac usage
Ari recaps key DevDay announcements centered on Computer Use. He highlights Dots as a new assistant product, GPT‑6.1 Sol for cost/speed benefits, and the Agents API gaining Computer Use so developers can build on the same tech used in Codex/ChatGPT.
- •Ari lists major announcements: Dots, GPT‑6.1 Sol, Agents API with Computer Use, and demos of existing Computer Use features
- •Ari says GPT‑6.1 Sol has strong cost/speed advantages for Computer Use and compares costs relative to Astra
- •Agents API now exposes Computer Use used by Codex and ChatGPT for developers
- •Demos include App Shots and native Computer Use on Mac that can run while the user does other things
- 2:46 – 3:36
Decisions API vs. Computer Use: fast parallel inference without reasoning
Asked whether everything is the same model stack, Ari distinguishes Decisions API from the Computer Use setup. He frames Decisions as a smaller, non-reasoning model that runs inference in parallel for speed, but may be weaker at long-horizon tasks.
- •Ari explains Decisions API does inference in parallel and is designed to be very fast
- •He notes it does not have reasoning and uses a smaller model than Computer Use
- •Ari says this can limit long-horizon sophisticated tasks compared to Computer Use
- •He calls combining these approaches an open research area and is eager to see what developers build
- 3:36 – 4:58
Dots as personal cloud computers: each Dot gets a dedicated Linux desktop in the cloud
The hosts dig into what makes Dots different: they’re persistent assistants paired with their own virtual Linux computers. Ari explains this expands what agents can do by running full desktop apps and browsing the web, enabling delegation of many everyday workflows.
- •Dots have access to an entire Linux virtual computer in the cloud, not just a browser or the user’s machine
- •Ari says this enables running full desktop applications alongside web browsing
- •He argues Computer Use is powerful because it can do anything a person can do with human-designed software
- •He suggests starting by identifying repetitive time sinks and delegating them to an agent
- 4:58 – 6:11
Everyday “delegate it” use cases: meal prep ordering and creator workflows like YouTube automation
Ari and swyx trade concrete examples of tasks that become practical with persistent Computer Use. Ari shares a meal-prep ordering workflow that went from hours to minutes, while swyx highlights creator tooling where platforms don’t expose APIs.
- •Ari describes using Computer Use to complete a complex meal-prep order faster than he could manually
- •He says the agent did the task in about 15 minutes versus his two hours
- •swyx cites creator automation needs, especially on YouTube where many features aren’t available via API
- •Ari notes he’s heard similar YouTube usage patterns from OpenAI’s developer experience team
- 6:11 – 9:14
Why Computer Use is “180 degrees different” now: better models, debugging, code-writing, and multimodal harnesses
Responding to skepticism that Computer Use hasn’t advanced, Ari argues capabilities have changed dramatically. He points to stronger models that can debug and retry, plus harness improvements like executing JavaScript for multi-step actions and using accessibility/DOM interfaces.
- •Ari says models used to start tasks but often got stuck; now they’re much better at debugging, retrying, and introspecting failures
- •He notes Computer Use increasingly writes and executes JavaScript to perform many actions at once
- •Ari describes using multiple interfaces: screenshots, accessibility representations, and Playwright depending on the task
- •He says progress comes from both harness and model acceleration over recent months
- 9:14 – 10:43
Measuring gains: harness permutations, safety checks, and a shifting cost–speed frontier
The hosts ask how OpenAI measures Computer Use improvements and what drove the reported speedups. Ari explains internal measurement varies by harness configuration and product safety needs, but gains are consistent and come from both harness and model changes.
- •Ari says measurement is complex because production products include extra safety checks and different configurations
- •He reports they see consistent gains across measurement approaches
- •Improvements come from both harness changes and model improvements
- •Ari highlights that GPT‑6.1 is even more cost-effective for Computer Use than the baseline cost improvement versus Astra
- 10:43 – 13:20
Big “aha”: accessibility + DOM access reduces scrolling; App Shots provide token-efficient rich context
Ari describes a key breakthrough: giving the model richer representations so it doesn’t operate purely screenshot-by-screenshot. The conversation then explains App Shots—often mistaken for plain screenshots—as a way to capture raw text and accessibility metadata in a token-efficient form.
- •Ari says older systems wasted time scrolling and taking repeated screenshots; richer modalities let the model ‘see’ more at once
- •Accessibility and direct DOM access help the model view whole pages and batch actions via code
- •App Shots can expose raw text and raw accessibility representations when inspected
- •Ari says substantial work went into making these representations token-efficient and effective for models
- 13:20 – 17:01
Roadmap constraints and bottlenecks: waiting on websites, event-driven triggers, and long waits in customer service
Asked about the next two years, Ari frames the goal as superhuman speed and more real-time agent experiences. He notes that as agents get faster, benchmarks become limited by external software load times and the complexity of triggering actions immediately after pages or states are ready.
- •Ari predicts the next frontier is Computer Use reaching expert-level or superhuman software operation speed
- •He says faster agents could reduce the ‘activation energy’ of delegating tasks to agents and shift user defaults
- •He identifies bottlenecks across model, inference, harness, and representation—and increasingly external load times (e.g., websites loading)
- •He discusses event-driven approaches when possible and the complexity when waiting isn’t tied to a clean event (e.g., customer service response delays)
- 17:01 – 18:43
Agents API guidance: trust, permissions, safety checks, and user consent for consequential actions
With Computer Use now in the Agents API, the hosts ask what developers should watch out for. Ari emphasizes building trust through reliability, scoping access, and requiring user consent—especially for actions like payments—while encouraging developers to build on OpenAI’s harness because the models are trained on it.
- •Ari is excited developers can use the same Computer Use implementation powering Codex and ChatGPT
- •He says building a custom Computer Use harness can be valuable but is hard, and using the in-distribution harness may have speed/cost/accuracy benefits
- •He stresses building trust through reliability and appropriate safety checks
- •He highlights asking user consent before consequential actions (e.g., payments) and limiting access to only necessary websites/apps
- 18:43 – 20:30
Computer Use in the dev lifecycle: agents that build and then visually test/QA what they built
The conversation turns to how Computer Use changes coding workflows by letting an agent test software it created. swyx describes closing the loop on the software development lifecycle, while Nikunj adds a “visual play test” approach for catching design issues and cloning app flows screen-by-screen.
- •swyx says Computer Use enables agents to test what they build, reducing the need for the developer to act as QA
- •He describes nested usage where a Computer Use agent helps develop Computer Use itself
- •Nikunj describes using visual play testing to catch design issues that code review might miss
- •Nikunj notes Computer Use can drive screenshot-based documentation of app flows and support cloning interfaces screen-by-screen
- 20:30 – 22:48
API platform updates: async tool calling, mid-turn steering, and WebSockets for bidirectional execution
Nikunj introduces himself and explains how new model capabilities get exposed in the API. He highlights async tool calling so tool execution doesn’t stall the model, mid-turn steering to inject messages during reasoning, and WebSockets enabling bidirectional communication that supports these behaviors.
- •Nikunj says the API team works closely with post-training and research to expose new model capabilities
- •Async tool calling lets tools run without pausing the model’s execution, enabling continued reasoning and later check-ins
- •Mid-turn steering allows injecting messages/instructions while the model is reasoning mid-turn
- •He attributes much of this to WebSockets enabling bidirectional communication (distinct from GPT Live in his explanation)
- 22:48 – 24:32
UltraFast inference: reducing overhead and pushing frontier-speed interactions
Nikunj discusses UltraFast as a major inference effort, describing how the inference team shifted from cost efficiency to maximizing speed. He ties UltraFast to WebSockets by emphasizing the need to reduce tool-call overhead and improve back-and-forth latency when using tools.
- •Nikunj describes UltraFast as a project focused on making models run as fast as possible after earlier efficiency/cost work
- •He says it’s been impressive watching the inference team iterate rapidly to squeeze performance
- •WebSockets reduce overhead for tool-heavy workflows by avoiding repeated request/response setup
- •The hosts discuss model naming and retiring older usage limits as the platform evolves
- 24:32 – 33:34
Decisions API origin story and mechanics: JEV-inspired, built on Luna with constraints and parallel batching
Nikunj recounts how interest in JEV-style systems sparked an internal push to build Decisions API quickly. He explains it’s built on existing Luna weights (not a newly trained model), using constrained structured outputs, optimized TTFT, and parallel batching of multiple questions to achieve speed, with iterative deployment to learn and improve.
- •Nikunj credits JEV and Diogo’s team for inspiring a fast-moving internal effort
- •He says the project wasn’t on the agenda weeks earlier and was driven by OpenAI’s hacker culture and rapid prototyping
- •He explains Decisions API uses Luna weights and adds constraints (including structured outputs) plus inference optimizations for speed
- •Multiple questions are run in parallel as a batch; the first version is described as ‘zero-shot’ on top of Luna with plans to iterate based on feedback
- 33:34 – 38:17
Developer focus areas: Responses API performance, caching & pre-warming, and context compaction for long-running agents
Nikunj outlines what he wants feedback on: new products like Agents/Decisions API, and especially low-level performance work in the Responses API. He details caching guarantees, longer cache windows in preview, pre-warming caches, and approaches to context compaction (server-side thresholds or manual /compact), including ongoing experiments.
- •Nikunj asks for broad feedback on Agents API and Decisions API to guide roadmaps
- •He says Responses API work is centered on latency improvements and deeper caching for long-running agent threads
- •He describes cache guarantees (including a longer window in preview) and pre-warming to pay a cache write fee upfront for faster reuse
- •He explains compaction options: automatic server-side compaction at thresholds or manual /compact for full control, plus new techniques being explored in the Codex harness
- 38:17 – 40:23
Where the platform goes next: higher-level primitives, memory vault ideas, and the “AI cloud” analogy
Closing out, Nikunj reflects on the challenge of designing the right level of abstraction in AI APIs. He contrasts low-level primitives plus example harnesses with potentially higher-level objects (like memory and storage abstractions), while swyx frames the broader effort as building an AI-native cloud with exposed building blocks like caching and pre-warming.
- •Nikunj draws on prior experience to discuss building higher-level primitives on top of core APIs and says the right balance is still unclear
- •He references earlier attempts and the shift toward Agents API, asking how much flexibility vs. abstraction builders want
- •He floats ideas like memory vaults and higher-level API objects to abstract storage concepts
- •swyx compares the effort to building an AI-native cloud, with infrastructure-like primitives such as caches and pre-warming exposed to builders
