a16zWhy OpenAI Is Racing to Make Computer Use Superhuman
EVERY SPOKEN WORD
50 min read · 9,827 words- 0:00 – 0:51
Intro
- AWAri Weinstein
We've gotten computer use now to a place where it's faster than the average human, and I think we're getting really close where it is superhuman.
- SASeema Amble
What sorts of tasks do you think are ready to hand over to the agent versus the ones, like, okay, it still requires some checking and correcting?
- JSJames Sun
Computer-using agents now since Astra is now capable of doing anything that a person can do. We think a lot about what new consumer experiences can we enable with a faster, better model. And then on the enterprise side, that's kind of naturally why the safety becomes very interesting. How do we give the ability to control the amount of human in the loop there is, and then the auditability?
- AWAri Weinstein
It's unlike any other thing that we've built where the developer has to do some work to adopt it, someone has to build a connector or MCP server. You know, computer use works with everything.
- FSFabrizio Serafini
Now we have, like, capable models. They're getting cheap enough to run at scale. What should companies think of automating?
- JSJames Sun
I think of it as in two big categories. The first category is like-
- 0:51 – 5:43
What GPT-6 Astra unlocks for consumers and enterprises
- SASeema Amble
Welcome to the a16z podcast. I'm Seema Amble. I'm joined by my colleague Fabrizio Serafini. In August, we wrote about, uh, computer-using agents and how they're starting to do the real work inside companies. Uh, today I'm joined by Ari Weinstein and James Sun from OpenAI, um, who are going to help unpack what is going on with computer-using agents from their perspective and, uh, share some thoughts on the future. Ari, James, thanks for joining us.
- JSJames Sun
Of course. Happy to be here.
- AWAri Weinstein
Yeah, thanks for having us.
- JSJames Sun
Thanks for having us.
- SASeema Amble
Ari, kick it off and tell us a little bit about, um, what in Astra has made computer-using agents move forward and what, what's really unlocked.
- AWAri Weinstein
Yeah. Um, so Astra's an amazing model. It's good at so many things. Our computer-use models were already so good before Astra. You know, like, Sol itself is amazing at computer use, so, um, Astra really helps with a few things around the edges in my sort of anecdotal experience. One is it is a little bit faster on average because it uses fewer thinking tokens to do the same task. It's better at certain kinds of creative work. Like for example, we had this demo where, um, in our launch video where you were talking to it and it's drawing things on Microsoft Paint. Um, it's way better at that sort of thing, which I think probably transfers to other kinds of creative tasks also. And then the last thing I find is it's just more reliable around the edges. Like, you know, it's kind of... I feel like I trust it a little more. Like, it's sort of the kind of thing where there's a little difference that is hard to perceive but really important between something being ninety-nine percent reliable and ninety-seven percent reliable, and I feel like it's sort of just a little, a little higher in a way that feels, uh, pretty, pretty concrete.
- JSJames Sun
Yeah, and I think adding on to some of those things, Astra's also our most aligned model.
- AWAri Weinstein
Yeah.
- JSJames Sun
Right? And so, you know, talking about trust a little bit-
- AWAri Weinstein
Yeah
- JSJames Sun
... um, you know, if you look at the system card for it, it is much better at being resistant to prompt injection. Um, it's much better at not taking any kind of, um, unintended actions. And so when you think about it for like enterprise work, especially for companies using it for sensitive things, whether it's like procurement or, you know, talking to vendors or talking to customers, it's going to be just a l- you, you're gonna feel a lot safer kind of using it, and that's really, really exciting. And then so some, some fun things on the creative side, I think internally we had, um, you know, Astra play video games against-
- SASeema Amble
Right
- JSJames Sun
... uh, you know, 5.6, and it wins every time. And so there are some cool tweets that people have made where, you know, Astra was able to beat Fallout in like 30 hours or something like that. And so it's also just really amazing at playing video games. So [laughs] I know that probably translates to something, but [laughs] yeah, it's very exciting.
- SASeema Amble
May- maybe just to double-click on that point, um, consumer versus enterprise use cases of computer agents, very, uh, very different implications around kind of like all the scaffolding around-
- JSJames Sun
Mm-hmm
- SASeema Amble
... I would say the, um, computer-using agents. How do you guys think about where things are more exciting-
- JSJames Sun
Mm-hmm
- SASeema Amble
... uh, between consumer and enterprise if, if you were to pick?
- JSJames Sun
I think, you know, our goal at OpenAI is really about building products that's useful for everyone. And so, um, you know, actually it's interesting 'cause Ari is definitely more of a consumer guy here, and then I spend a little bit more time on the enterprise side. Um, I think the consumer is always the place where we try to push the frontier on what's possible. Uh, you know, uh, with Astra, like Ari mentioned, there is-- it's a much faster compared to the previous models. And so we think a lot about, like, what new consumer experiences can we enable with a faster, better model? And then on the enterprise side, that's kind of naturally why the safety becomes very interesting because I think at the enterprise, whenever we talk to large companies, I think the use cases for computer use is actually fairly obvious, right? Inside enterprises, there's lots of legacy software that doesn't have any MCP tools. They are software that's been used for, like, 15, 20 years, and so there's... And, and, and a lot of times it's actually kind of hard to use for a normal person.
- SASeema Amble
Right.
- JSJames Sun
Lots of, like, UI that is, like, from the last, you know, [laughs] decade. And so you don't even know how to operate some of the key, key, key things. And so we've seen companies build skills that kind of teach the model to be able to use them better. And so I think maybe the way, the way to think about it is enterprise is where, you know, the capabilities have been there maybe for a little bit longer, and the companies are catching up but trying to understand how to deal with it. And consumer is where we're trying to figure out the next types of use cases that's not maybe possible today with software. Maybe you can share a little more.
- AWAri Weinstein
Yeah, I mean, the consumer use cases are so fun for me because it's kind of like how can you put your agent to work doing the things in your daily life-
- SASeema Amble
Right
- AWAri Weinstein
... that are so annoying and save you tons of time and just have more time to focus on the things that, um, that you're excited about. Um, I think it's been really fun to see Dot, uh, where, you know, each dot gets its own computer, and you can ask it to do things for you in the cloud.
- JSJames Sun
Mm-hmm.
- AWAri Weinstein
Um, it gets a full web browser. It can buy things for you. It can, um, you know, book haircut appointments and, uh, you know, a-a-anything else you want it to do. So, um, so that, that's been really fun, and I think there's a, a lot more on the horizon that's gonna be really fun.
- 5:43 – 8:05
How the OpenAI team uses dots
- FSFabrizio Serafini
Yeah, no, that was gonna be my question on, on, on Dots. I think, James, you probably use a Dot to walk through product feedback.
- JSJames Sun
Mm-hmm.
- FSFabrizio Serafini
It'd be cool if you could share, share more about it. Or I'm just so curious to know on, on your end, guys, like, how are you using Dots and, like, how does com-computer use play into kind of what you can do with them?
- JSJames Sun
Yeah. I mean, I think Dots is really cool for a few reasons, right? One is, like, really proactive, and so, um, I've been very surprised a lot of times when in both my personal life and work, when Dots will just flag something that, um, I just completely slipped through the cracks, right? Just the other day... So internally, like, on our, our email system, whenever we get an external email for an invite, it doesn't actually show up on our calendars, and so I end up mis- missing a bunch of, like, certain meetings. And so Dots now is, like, catching these things for me. It's like, "Oh, there's this invite that's in your inbox, it's not on your calendar. Do you want me to, like, accept it in your Gmail so it shows up?" So that's been a lifesaver. It's a small thing, but it matters a lot. Um, but I also have it proactively kind of looking at... I think it's really good at proactively looking for tasks where it's, the shape of it is not super, uh, let's say concrete every single time. So an example that I also have it set up to do is, um, we use Slack internally, and there's, there's a ton of messages that's going on at all times. And, uh, I used to have this Slack job, uh, using automations in Codex that's like every hour or every 30 minutes go and try to pull things, and I've just ended up tuning out a lot of the notifications because it's just, like, sometimes useful, sometimes it's not.
- SASeema Amble
Right.
- JSJames Sun
With Dots, I can give kind of a high-level goal. It's like, "Hey, if anybody in my reporting chain says anything, flag it to me," right? If anybody who's giving some critical feedback, use your judgment, uh, about certain products, go for it. And then because it has a cloud computer, um, you know, I've had this pretty cool workflow where when someone reports, they usually have like a Sentry log. I could have it go use the cloud computer, um, you know, actually drive my local, local browser while the Dot's gonna con- control both your local computer as well as the cloud computer, log into Sentry, pull the logs, analyze it, give me feedback. And so it's saving that entire loop of figuring out which feedback's important, looking at it, then going to Sentry, downloading their logs, feeding it into Codex, asking for thoughts, and all that can now just kind of be automated. And so, um, yeah, Dots is great for, for at least these proactive and kind of long-running tasks where they're not the same shape every
- 8:05 – 11:24
Approaching superhuman: speed, cost and control
- JSJames Sun
single time.
- SASeema Amble
We talked a little bit about reliability, speed, token usage, and efficiency. How do you guys think about balancing all of these things, uh, as you, uh, as you keep making improvements on what you're, what you're releasing?
- AWAri Weinstein
I think we're really excited about Computer Use at OpenAI, and we've invested in it a lot, and we've gotten to a place where I think we have the fastest, cheapest, and most reliable Computer Use product. Um, and I think there's actually so much further we can take that. Um, we're-- we spend a lot of time looking at speed. You know, speed is really interesting because on one hand, the power of Computer Use is it can do tasks in the background, so in some cases, it doesn't matter how long it takes because you can g- be go- going and doing other things while it's accomplishing the task for you. But also, sometimes you wanna watch it to make sure it does it correctly. You need to wait for it because it needs to complete this before something else can happen. And so the speed is so important because, um, you know, you don't wanna get this feeling like, "Oh, I could have just done that myself, and it would've been so much easier." So, um, you know, we've gotten Computer Use now to a place where it's faster than the average human at using a computer, and I think we're getting really close to a place where it is superhuman, meaning it is faster than an expert computer user, like probably one of us at this table. Um, and I think that's gonna actually be a sea change when that happens. I think it's gonna change the way we all interact with computers because all of a sudden, anything that you, you know, that, that you could do manually, it could do faster. So unless you really want to do it yourself for some reason, which I think actually there may be many reasons people would wanna do that, but in a default case, you probably just want the agent to do it for you, and I think that's gonna let us build all kinds of cool user experiences. So speed is something we think about a lot. And then obviously, we have a strategy of offering multiple different models at multiple different, you know, qualities and price points, and so for different use cases, you may wanna use a different model. Um, at the same time, our, um, our less expensive models are getting much better at Computer Use over time, and so I think there will actually be more variation over time in which models people wanna use for which tasks.
- JSJames Sun
I think that's what I was saying. Like, I think another thought is, like, how do we give customers, you know, control over the safety, reliability, autonomy versus speed, right?
- AWAri Weinstein
Mm-hmm.
- JSJames Sun
Like, if you have... Like, you know, when, when the model's slow, you can kind of watch it do stuff, but then you're limited to that, to the tasks that you can do because you can't just sit here watching the thing doing it. Like, it's actually less effective, right? Because it, it's slower than, than you. Um, but once you can get to a point where you can no longer observe and audit everything it's doing, um, how do we give, especially enterprises and businesses, the ability to control, um, the amount of human in the loop there is-
- AWAri Weinstein
Right
- JSJames Sun
... and then the auditability, right?
- AWAri Weinstein
Mm-hmm.
- JSJames Sun
Like, I think a few months ago, uh, it was part of Astra. It's not a few months ago. As part of Astra, you know, we, we kind of launched a set of enterprise controls that let, um, businesses choose, one, like how much human in the loop is there? Like, do they want the agent to automatically approve everything? We have this really awesome system called Guardian, right? Our auto, I think auto-approve in the app that, uh, is a really smart model looking at every single action that the Computer Use agent is taking to decide if it's safe, if it's, if it's aligned to the original intent. Um, a consumer might be like, "I just want it to do everything, and I don't wanna be asked for approval." Whereas an enterprise is able to kind of decide which type of actions actually go through. And so I think the, the, the control part becomes more and more important as our Computer Use agents become also superhuman at doing
- 11:24 – 13:51
Agents on the open web and the ad problem
- JSJames Sun
things.
- SASeema Amble
Totally. I think as you try to, you know, roll out more enterprise use cases, I think employers are excited about the efficiency, but they're also, you know, wary of-
- JSJames Sun
Yeah
- SASeema Amble
... like what is, what are, what's sort of being unleashed.
- JSJames Sun
Yeah.
- SASeema Amble
Maybe even on the other side, have you guys seen the response from application software around, okay, now there's computer-using agents accessing my legacy workflow software-
- JSJames Sun
Mm-hmm
- SASeema Amble
... and how do you, do you guys think at all about that? Are you more focused on just the, the agent that's being used by the, you know, consumer enterprise user?
- JSJames Sun
Hmm. Yeah, I think there's like-- I think this problem becomes really interesting on the open web, actually. I think when it's, like, within software that is, like, approved and used by the company, it's actually more straightforward-
- SASeema Amble
Right
- JSJames Sun
... because it's like, oh yeah, 100% we wanna have Uh, people automate and do work in here that they don't have to do tediously themselves. But when it comes to the internet, I think you kind of have a few different camps of folks, right?
- SASeema Amble
Right.
- JSJames Sun
One is we run a-- our business model is not in line with having agents use our product, right? You think about advertisers, publishers whose business model is entirely on advertisement. If the agent's using the website for them, then where's the ad revenue coming from?
- SASeema Amble
Right.
- JSJames Sun
And I think most places, a agent's ad impression does not actually count towards real, uh, revenue. Um, and so that's, again, an interesting question that we're always working through and working with our partnerships team on. Um, there's another question kind of around, um, for the companies who do really wanna embrace having computer use agents using their sites, a lot of them are adopting kind of standards that let humans or people and agents to be able to use it together. So, like, WebMCP is a standard that we're really excited about. We, we added support for it in the in-app browser, and so you can kind of imagine these s-scenarios where the agent is elevating the experience of using the site, but is not entirely replacing it. So, like, I think browsing the internet is still very fun, in my opinion, right?
- SASeema Amble
Yeah.
- JSJames Sun
I still like doing shopping. I still like looking at all the hotels and flights and places. And so, you know, you can, you can imagine places where, um, the, the, the agent does all the tedious work, but it kind of carves out, uh, an user experience where you're doing the browsing, being like, "Okay, now I wanna finish the, the transaction," and then put in my credit card and do all this stuff, then you can hand it back to the agent. And so WebMCP is kind of an interesting way where your interface with the website is still the site itself versus an API, but then the person is kind of using it together. And so tho-those are just some thoughts.
- 13:51 – 17:21
Which tasks to hand over to an agent
- JSJames Sun
Yeah.
- SASeema Amble
Shifting gears, wanna double-click a little bit on what it means to do the job or-
- JSJames Sun
Mm.
- SASeema Amble
-for get the end-to-end job done. Um, would love to hear a little bit from you guys around, you know, what works well, how do you guys define the right, the right narrow, repeatable task-
- JSJames Sun
Mm.
- SASeema Amble
-uh, for, you know, and what sorts of tasks do you think, um, are ready to hand over to the agent versus the ones, like, okay, it still requires some checking and correcting as the, on the user side?
- JSJames Sun
Mm. I mean, generally, I've found the model to be pretty capable to do anything that you can do on a computer now. Like, um, and, and you know, one exercise that I try to force myself to do, um, is that whenever I have a task, the first thing I think about is just, can I just have the agent do it? Now, there are obviously some sensitivities around, you know, if I'm sending a message to someone, I wanna feel like it's coming from me. Um, but in general, I think the way, uh, I think about it is computer-using agents now since Astra, um, is now capable of doing anything that a person can do. And, uh, there are questions around, I think before, of like, oh, there are some, like, squishiness around, like, specificity and requirements. I've, I've generally found that the model is getting pretty good at whenever you're asking it to do something and there's some vagueness, it will actually... You know, with Astra, I think we launched this, like, uh, flexible input, right?
- SASeema Amble
Yeah.
- JSJames Sun
And so it will actually ask you about, "Hey, like, I want some details." And if you don't answer it in time, it'll just keep going and then pull the context as necessary. I think there are some things around, you know, if you have repeatable workflows, it's nice to be able to kind of define, you know, the, the requirements are at least similar to, like, writing a product spec. Um, but in general, I, I generally feel like unless there is something incredibly sensitive that you wanna be in the loop for, I j- I just reach for the computer use model for everything I d- I do now. I'm curious if it's the same for you.
- AWAri Weinstein
I-- Yeah. It, it, it is. I'm, I'm trying to, uh, uh, you know, build that impulse to-
- JSJames Sun
[laughs] Yeah.
- AWAri Weinstein
-to reach for it, uh, because, you know, it's so good, and sometimes it's so easy to get locked into-
- JSJames Sun
Yeah
- AWAri Weinstein
... the old way of doing things. Um, I, um, I think s- the really cool thing about computer use as a technology is the universality of it. You know-
- JSJames Sun
Mm
- AWAri Weinstein
... it works with every application. It's unlike any other thing that we've built where, you know, the developer has to do some work to adopt it. Someone has to build a connector or MCP server. You know, computer use works with everything, and it works by using the application the same way a human uses it, um, which I think is something worth understanding if you haven't spent much time with this technology. It-- Like, it-- Computer use uses computers the same way a human does. It makes the agent have all the capabilities you do, um, which I think is so powerful.
- JSJames Sun
Yeah. And I guess, like, adding on to Ari saying, like, part of it is, like, I think over time I've stopped thinking about it as, like, am I using the mo- like, a computer use task or a non-computer use task.
- AWAri Weinstein
Mm-hmm.
- JSJames Sun
And frequently they are, like, connected, right? Like, the way we think about computer use, um, is that, like, computer use is not, like, a thing you pick up and just say, "I wanna do something with it." It's really, like, the universal kind of API that the model has to do whatever it takes. And so if you have an ambitious task and you want the agent to do it, likely the more complex it becomes, the more likely it will probably touch computer use at some point in that journey.
- SASeema Amble
Mm-hmm.
- JSJames Sun
Um, and so, like, maybe the only time that I tr- I don't reach for computer use for some cases is if the task is so simple that then it's, like, easier for me to just go and, like, click two buttons and do it. But, you know, as computer use gets faster, even faster, then the question becomes, maybe I don't even need to do that. I just, you know, drive voice, and it just uses the computer with me. Um, so tho-those, those are some interesting thoughts there too.
- 17:21 – 20:04
Getting enterprises to trust computer use
- JSJames Sun
Yeah.
- SASeema Amble
Do you think there's anything you have to share with, like, an enterprise user-
- JSJames Sun
Mm-hmm
- SASeema Amble
... to get them comfortable with that? Uh, obviously, you guys are totally built on computer use, but-
- JSJames Sun
Yeah
- SASeema Amble
... you know, I think, uh, there's obviously, like, hesitance sometimes on-
- JSJames Sun
Totally
- SASeema Amble
... the enterprise user of like, okay, well, yes, the human would click through it and probably do it worse, but, uh, there is that feeling of, like, there's accountability and-
- JSJames Sun
Yeah
- SASeema Amble
... and, you know, uh, if that person's trained and all, you know, all the classic kind of rebuttals. I'm curious-
- JSJames Sun
Yeah. Yeah
- SASeema Amble
... what you have to convince people around.
- JSJames Sun
For sure. Lots, lots of conversation with enterprises kind of over the last little bit, and I think the, the main thing is- We think about it as two layers, right? The first layer is like, how do we create deterministic controls? I think that's really what gives, um, you know, enterprises a lot of s- feeling of safety, is when they can... Even though our model is the most aligned ever, and it's, I think better, you know, better than most people at pr- you know, protecting against prompt injections and unintended things because it's like just, you know, the intelligence is so smart. But, um, I think there is just a feeling of safety when the enterprise admins can basically say for certain tasks or for certain data sources are just too sensitive to be used. Um, and so we see kind of enterprise-- So what we, we launched was two things, right? One is the ability to be able to deterministically create a sandbox around your agent. So being able to say it can use certain apps, it can't use other apps, it can use certain websites and not certain other websites. And then two is, you know, this kind of part of a human in the loop, like what type of actions does it require a human to, um, be intervened? And so we might say, okay, for any time there's like a download and upload, a person has to kind of approve it. Any time it goes to a new website, a person has to approve it. Now again, these are all like dials an enterprise can kind of go through. Um, and so it's really like that's one, and then two is really just showing the enterprise admins just how, you know, all the evals that we've run internally around how aligned these models are even compared to like, you know, both internal and kind of third party, um, kind of evals.
- AWAri Weinstein
I think there's also some baseline stuff that we do across the board to just try to help people build trust with these kinds of products.
- JSJames Sun
Right. Yeah.
- AWAri Weinstein
You know, we know it's so new, it's such a magical and sometimes alien experience for people.
- JSJames Sun
Mm-hmm. Yeah.
- AWAri Weinstein
And so, um, you know, we've put a lot of work into the design of the product, so you can always see the cursor if you want to watch it perform a task, and it sort of animates in a fun way, so it's not boring to watch.
- JSJames Sun
Mm-hmm.
- AWAri Weinstein
And, um, and then, uh, like James s- was saying, this is very con- um, customizable for enterprises, but also for a regular user. You know, we ask you for permission before using a new application or website, um, uh, depending on which sort of modes you're in. And, um, I think that helps people sort of not feel like it's gonna go do some runaway stuff on their computer. You kind of wanna have this trust building mo- moment where you figure out what it feels like and what you, um, want it to do for you, and then, um, and then you sort of feel more comfortable over time.
- 20:04 – 25:36
The best enterprise use cases: boring is opportunity
- FSFabrizio Serafini
The Codex cursor is so cute. I love it.
- JSJames Sun
Yeah.
- FSFabrizio Serafini
It's like such a... I, I know it's a small detail, but like watching it use the computer gives you like confidence, and it makes the experience nice. So I think there is a kind of like a user experience point, uh, that's, that's worth, uh, uh, that's definitely worth inv-investing on. And I guess, like, I'm curious, like in, in a broader sense like, okay, now we have like capable models-
- JSJames Sun
Mm-hmm
- FSFabrizio Serafini
... they're getting like cheap enough to run at scale. In your view, what should kind of companies think of automating? Like what would you-- What-- Where do you see the most promising kind of use cases within enterprises? Are there some workflows, maybe something around the ERP, maybe something around kind of, uh, how like, again, humans and agents can work together to get tasks done. But like if you had to like pick where, where are you most excited to see your models perform, uh, in the world, like?
- JSJames Sun
Yeah. I, I guess, like from my perspective, I'd love to hear Ari's, it's like two, two... There-- I think of it as in two big categories. The first category is like automating, you know, tedious workflows. That's like the more boring one, and I'll give you some boring examples like- [chuckles]
- FSFabrizio Serafini
But boring, boring is opportunity, as you said. Yeah.
- JSJames Sun
Boring is opportunity. Yeah, exactly. Um, like ERPs is a really big one. Um, and so all the, all the acronyms you can think of, ERPs, CRMs, uh-
- FSFabrizio Serafini
All the systems of record. Yeah
- JSJames Sun
... yeah, all the, every single system of record.
- FSFabrizio Serafini
Yeah.
- JSJames Sun
Right? And so, you know, we've seen folks, you know, procurement software. Like, you know, you are going to vendors, and every year you're ordering a bunch of lab equipment. Uh, instead of having this giant spreadsheet with like 1,000 rows that then the ops team... So it's like, you know, worked with one company. It's like normally what they have done is you-- they have this giant spreadsheet that gets filled out by the teams themselves, then an ops team takes that giant spreadsheet, puts it into an internal, you know, procurement software, and then that gets like, you know, finalized as a, a bunch of lines. And then a sales team then goes-- procurement team then takes that giant software output and then goes out and reaches out to vendors. Um, Computer Use can automate all of that, right? So like procurement is a big one we see. Um, finance is another really interesting category we've seen. And so, uh, you know, there's a lot of different ledgers within a company, right? And so you have like ledgers for expenses, ledgers for all these things, and so at the end of the month, there's a lot of reconciliation. And a lot of these reconci- uh, financial software don't have APIs to be able to quickly pull things. Um, and so the finance team being able to have Computer Use look through all these ledgers, reconcile, pull out anything that's like interesting, um, to flag either to a person or then do some investigation. Also a very boring but useful workflow. Um, for their sales team, a l- lot of sales teams uses a lot of different software for like transcripts, for getting data, all these kind of things. And so having Computer Use go and do prospecting and then going and, uh, putting that into a CRM or some kind of like, you know, s- outreach software and then all that. So all, all the system records is the boring part. Um, and then like market research is another one that we see a lot. Um, and so, you know, in my free time, I've been building this like, uh, video annotation software for, for badminton. I play a lot of badminton and like I like to film myself playing the game and then, [chuckles] uh, having the agent kind of go and, and look at stuff. And I was like, okay, how do I make this thing really useful? There's some s- software out there for like pickleball, for squash, for tennis, and so I just had Computer Use go log in, create an account on these soft- on these softwares, try it, take screenshots, give me feedback, and then try to build it back into the app. And so these are all kind of like tedious workflows that could be done. And then I think the, some of the more like interesting creative work of like enabling more ambitious work is like, you know, we talked about, um, the model now being really good with like visual, visual design. And so, um, circuit design, you know, being able like architecture work, hardware engineering, game design, these are also areas that I, I think we're really excited about where the Astra can take. Um, less, you know, less common for enterprise to think about those right now because they're thinking about the boring opportunities. But I think as folks understand the capabilities of the model more, that's where, um, I think we'll, we'll kind of see more usage next. But curious if you have any thoughts. Ari?
- AWAri Weinstein
Yeah, I think kind of similarly on the consumer side, there's a lot of things where it's sort of automating drudgery or just-
- JSJames Sun
Yeah
- AWAri Weinstein
... doing things you don't wanna do. It's filling out DMV forms for you.
- JSJames Sun
Mm.
- AWAri Weinstein
It's helping you, uh, unsubscribe from a bunch of emails in your inbox. Um, and then I think there's also things that are more like, um... I, I'm, I'm hearing from a lot of people stories of people doing things that they just wouldn't have time-- they wouldn't have had time or wouldn't have thought to do otherwise. Like, I don't-- for me personally, I'm, um, uh, maintaining spreadsheets about my finance and spreadsheets about meal planning and-
- JSJames Sun
Yeah
- AWAri Weinstein
... uh, I have a friend who's a personal trainer who's making customized things for his clients and, um, you know, some of these things, um, would just be so, so, uh, time prohibitive to do in your, in, in y- in your personal life. Um, but this makes it so easy. Um, I use Apple Numbers, which is like a obscure spreadsheet program, but the model still knocks it out of the park, you know?
- JSJames Sun
I don't think I've ever heard someone use Apple Numbers.
- AWAri Weinstein
[laughs] It's a surprisingly great spreadsheet software. Um, and then I think there's a third category of thing which I'm kind of the most excited about, which is just starting and I think, uh, uh, l- you know, on the horizon, you know, um, which is really just having a different way to use your entire computer or interface-
- JSJames Sun
Mm-hmm
- AWAri Weinstein
... with software. I think you saw in our Astra ad, uh, really cool examples of people just talk to their computer with their voice. It's such a natural way to use your computer, and I think over time we're gonna see the form factor of software itself change due to this technology, and I'm, I'm really excited about that.
- 25:36 – 32:38
Will agents run legacy software or replace it?
- JSJames Sun
On that point, okay, form factor of software changing. Um, if- imagine if you're an enterprise user, you've used to, you're used to using whatever clunky ERP from... was built, you know, 30 years ago.
- AWAri Weinstein
Yeah.
- JSJames Sun
Um, there's a world in which you now, you know, unleash, uh, a set of agents that just operate that software, and you don't really care. Um, there's another world in which, um, folks are a new n- like AI native player is using, computer-using agent.
- AWAri Weinstein
Yeah.
- JSJames Sun
And I'm curious if, do you think the world is gonna go more in the direction of like keep the legacy folks, um, and just have this layer on top? Or do you think, you know, computer-using agents can absolutely be part of, uh, new software players? And this is something I think a lot about, so I'm curious to get, get your guys' thoughts.
- AWAri Weinstein
It is such an interesting moment where simultaneously the cost and complexity of creating new software is so much lower, so you-
- JSJames Sun
Mm-hmm
- AWAri Weinstein
... can imagine software changing so much. But also there exists so much legacy software in the world-
- JSJames Sun
Right
- AWAri Weinstein
... that Computer Use is so well suited to interact with. So I expect both of those things to continue for a long time. You know, if you look at the software that's in production today, a lot of it was written 10, 20, 30, 40 years ago, and I, I, I don't think that that stuff's gonna go away overnight. I do think that in some way the existence of Computer Use, um, uh, puts, may put more pressure on companies to make software work better with agents-
- JSJames Sun
Mm-hmm
- AWAri Weinstein
... because all the software is going to work with agents by default, and I think WebMCP is a great example of this-
- JSJames Sun
Yeah
- AWAri Weinstein
... and I think we'll see more stuff like that. Um, so yeah, I, I sort of expect both to be, to be super important. Um, but I, I do think it'll like kinda shift over time. Um, what do you think, James?
- JSJames Sun
Yeah, I, I, I agree with you. I think it's like, um, the, there, the, the way... there's like a lot of arts and craft, and y- you understand this 'cause Ari builds a beautiful product with his team. [laughs] The cursor was all him. Um, and so the, uh, um, I think there's like a lot of art and craft into building like great products that people can hold in their hand. And, um, I don't know, like for me, if we go, if the internet becomes just this like faceless y- you know, thing that is just agents talking to each other and, and people are not using anything anymore, I would be very sad. Um, and I, and I don't think it will be like that because like there is like something about craftsmanship and art- artisan software that's like just feels nice to use because the person who designed it really, really understands, um, like the experience that the customer wants to have. So like my wife loves, is recently got into playing this like, uh, you know, board game. I for- I forgot the name of it. It's where you connect like five pebbles to like win. I think it's like Omok is the name of it. And so anyway, uh, so she's trying to get good at it because she's playing online with friends and is getting her ass kicked. And, [laughs] and so I was like, "All right. I'm gonna build you like a tactical trainer for, um, you know, going through different gambits so you can like get better at it." Like that's, for example, something that like I, you know, created a first version. I was like, "Oh, I know like she likes to do it in a certain way. She likes these type of, uh, ways to learn more than others." And so I was able to like, you know, craft the software to be something like feels really good for her.
- AWAri Weinstein
Mm.
- JSJames Sun
And like these kind of personalized software, I, I know it's like I think maybe today still feels like only really tech people will do. But I, I think if you like think about this a few years out, it's like if anybody can build anything, and they can build these really craft software in a very short amount of time for like niche use cases that are like so well-tailored for people, I think that's a beautiful world. Um, and again, you know, at the end of her training, the lessons and mistakes she made, that could be an agent that summarizes it and gives it to her in a text format. But when she's actually playing with the, with the AI and, and, and, and, and trying to get better, that's something that the person has to do. So I do think it's, it's, um... I, I'm more, I'm bullish that I, I don't think UI is going away. And then, uh, and I think kind of at a little tangent to your original question, but I, I do think that, um, the pressure of cheap software will make it so that software has to be better. Um, but I don't think it will just be like, yeah, everything will have an MCP and an API and, you know, Computer Use will never be useful. [laughs]
- AWAri Weinstein
Totally.
- JSJames Sun
I love this point so much. I think that like humans are so hardwired for direct manipulation. You really wanna be able to, you know, move and, and, and do things with your own hands.
- AWAri Weinstein
Yeah.
- JSJames Sun
Um, and I, I think that it's really exciting to think about how software in the past was so one size fits all-
- AWAri Weinstein
Mm-hmm.
- JSJames Sun
Yeah
- AWAri Weinstein
... and had to be, you know, often Uh, a lot of the software we interact with every day is designed for millions, tens of millions, hundreds of millions, billions of people to use.
- SASeema Amble
Mm-hmm.
- AWAri Weinstein
And I think we will enter an era where any piece of software could be designed for 100 people or one person-
- SASeema Amble
Hmm
- AWAri Weinstein
... and, um, just how much more, uh, personal and, and, and unique-
- SASeema Amble
Yeah
- AWAri Weinstein
... that kind of software will be.
- 32:38 – 37:31
Advice for founders building on computer use
- JSJames Sun
months.
- SASeema Amble
I, I love that you guys, yeah, talk about this 'cause I, I think it's so underappreciated the importance of like human-AI interaction, especially if you want like a product to be successful. So like being thoughtful of like, how can I kind of... Like humans are used to collaborate, so you want any agent to be able to like kind of fit into the team. I'm just curious, like if you apply that to a new startup, let's say you're like, I don't know, starting a company tomorrow, and, uh, you wanna use computer use to like, um, uh, solve a workflow within the enterprise, how would you think about it? And, uh, would you be kind of... What would be the, the things that jump to mind in terms of like, oh, I wanna know when it fails, well, I want it to be able to communicate with people, or I want it to be like a, I don't know, a black box. Just like maybe if you had to talk to founders right now, what advice would you give them as they try to use your models and capabilities to just solve problems for enterprises?
- AWAri Weinstein
In my mind, it depends on, it depends so much on the modality of-
- JSJames Sun
Hmm
- AWAri Weinstein
... you know, are you trying to... Are you a small team and you want sort of individual on the team to be able to use some piece of software that, that your, um, that, that your team is using but in a more efficient way? Um, or are you maybe building something that itself wants to interact with existing software, in which case maybe you have some, you know, machine in the cloud somewhere that's, uh, doing computer use in the background. I think that will really affect the modality of how you want the human agent collaboration to be, right? Um, and, um, yeah, you know, in the former case, I think a lot of our products are serving this use case, um, you know, where you sit down at ChatGPT and, uh, you know, there's a lot of great, uh, you know, collaboration you can have there. Um, and in the other case, you know, maybe you're building something with, with the API, um, you're, you're, um, uh, doing, doing your own computer use implementation, then, uh, yeah, then it really is interesting to figure out when, when do you want the human to be in the loop? Is the agent telling you when something's wrong? Is it telling you when something went right? Uh, you know, is something summarizing that so that you don't get bombarded with notifications saying, "Hey, this happened, this happened, this happened," which actually you're gonna ignore. Um, um, I don't know. What, what do you think, James?
- JSJames Sun
Yeah, I mean, I, I, you know, both Ari and I were startup founders before-
- AWAri Weinstein
Yeah
- JSJames Sun
... we began OpenAI, and so, you know, I'm, I'm still very bullish on startups. [chuckles] You know, I think the question, right, uh, uh, kind of comes down to like what... Is there still a moat, right? [chuckles] Is there something that startups can still tackle while labs are, are doing things? I mean, in general, I just think there's like, like two, two things. Like one is like I think there's so many things to build, right? And like even though we're quite big now, thousands of people, we are still feeling very stretched thin internally.
- AWAri Weinstein
Oh, yeah.
- JSJames Sun
And like, you know, it's like the teams are not very big for all the problems, and so if you have a problem that you really deeply understand, you have industry expertise, you understand how regulatory works, the go-to-market motion, how to service the, those customers, um, it, it is still, I think, a very green field out there. And, um, the, the tools that, that you have through our APIs, like, uh, Ari mentioned with computer use, um, I think, I think will still let you accelerate and speed really, I think at the end of the day, speed and pain [chuckles] is kind of the moat that-
- AWAri Weinstein
[chuckles]
- JSJames Sun
... that exists today. Um, I think the other thing I, I think about a lot is like there are s- in my opinion, still a lot of like vertical industries where, um, the diffusion of AI is like much slower. And so, um, you know, uh, I was talking to a company the other, the day who, uh, uses computer use to help restaurant groups to, you know, take pictures of like their back Storage, you know, and then using that y- the, the vision looks at kind of like what is gonna be missing based on a projection for tomorrow's traffic. And then it takes that to then use Computer Use to go to, like, a Shopee or some kind of, like, procurement software to then order your groceries for the next day. Um, and then based on that, it will go into your ERP to adjust, like, the timetable for your, like, employees who are, um, who we might need more people to help because in the last year, at this time of the day, um, there was a lot more traffic coming in from, like, like, a holiday or something like that. And so, like, you know, those restaurant groups are probably not, like, tech companies that are, um, super, you know, familiar with all the tools that are available. And so I think, like, you know, living in San Francisco, we see a lot of thinking, "Wow, like, the labs can do everything, the Codex-"
- SASeema Amble
Sure
- JSJames Sun
... "and, like, you know, ev- why doesn't everybody just install Codex and just have it do stuff?" But, like, for the average American and the average small to medium business owner or the average enterprise, um, even understanding what the tools are capable is one, connecting tools in a way that makes sense is two, deploying it is three, servicing is four. And so there's still a lot, in my opinion, where if there, if there are complex workflows that require a lot of disparate software, Computer Use is a great way to join them, complete the entire end-to-end workflow, and then building it in a format and UI that is, feels good to hold, um, for those customers feels s- still like a really, really big opportunity that, um, you know, we, uh, I think is still open. [chuckles]
- SASeema Amble
Right.
- 37:31 – 43:59
What’s next for computer use
- SASeema Amble
Tell us about what's next. What's, what are you excited about, um, that's on the frontier that you can share around, uh, around computer-using agents?
- JSJames Sun
Ellen?
- AWAri Weinstein
Well, I've, I've-
- JSJames Sun
[laughs]
- AWAri Weinstein
... sort of gestured towards it already, but we're really excited about speed. We're really excited about making Computer Use superhuman, and, um, we're really excited about things that blend together different kinds of interfaces, that bring together sort of multimodal inputs and outputs, um, so you can have more kind of fluid experiences with your computer. I think right now a lot of our Computer Use product is very background focused. It's very much like, um, let me give the agent a task so it can do that while I do something else, which is actually a tremendous strength, right, of the product. But I think that in the future we'll also see more of this, like, real time collaboration between you and the agent. Um, so, um, I'm really excited about this area. Um, and, um, um, and other, otherwise just, uh, sort of better, cheaper, more, more available, easier to use.
- JSJames Sun
More, yeah, exactly.
- AWAri Weinstein
Yeah.
- JSJames Sun
Being able to actually access everything.
- AWAri Weinstein
Yeah.
- JSJames Sun
Yeah. [laughs] I think, I think there's... You know, we, with Dots, right, we're, we're giving folks these, like, computer in the cloud. ChatGPT Work also has a browser in the cloud. I think it'll, uh, honestly, like, for, for me, like, one thing I'm excited about is just making the experience feel a little bit more co- more coherent.
- AWAri Weinstein
Mm-hmm.
- JSJames Sun
Like, you know, today you can install the Chrome extension, and it, that uses your, like, password manager and everything in Chrome, and then we have an in-app browser in the desktop app, which has a different set of cookies and different passwords. Your computer in the cloud also has different things, and for the average consumer it's k- kind of confusing.
- AWAri Weinstein
Mm-hmm.
- JSJames Sun
It's like, why do I have these four different browsers that all don't talk to each other? If I open up a tab in one, does it, like, show up in the other? And so, like, I think a lot of, um, what I'm excited about is how do we make Computer Use go from this product that is used by, like, enterprises and, like, really hardcore tip-of-the-spear kind of users to something that's actually understandable for, for people? It's kind of like an oxymoron because I think Computer Use is actually the most easy-to-understand AI product-
- SASeema Amble
Mm-hmm
- JSJames Sun
... because-
- SASeema Amble
Right
- JSJames Sun
... it's an AI using your computer.
- SASeema Amble
Yeah.
- JSJames Sun
But then because of the, like, uh, complexities of, like, the different, like, states you have to manage and the different places you're using it, the implementation details sometimes seep through, and then it becomes kind of hard to grok. And so, um, being able to kind of make that consumer grade and easier to understand, more co- more coherent, um, is something I'm really excited about for the next quarter.
- SASeema Amble
Yeah, and I think it, it comes back to the point that there is still a human that's-
- AWAri Weinstein
Yeah
- SASeema Amble
... setting off the agent, right?
- AWAri Weinstein
Yeah. [laughs]
- SASeema Amble
And they have to... The, the, the better the UI and controls and the easier and more enjoyable it is to use, the more they're gonna get out of it, and that's-
- AWAri Weinstein
Absolutely
- SASeema Amble
... still principle.
- AWAri Weinstein
On that point, one other thing I want to gesture towards is, um, I think another challenge of the product today is you, you have to put some thought into how you're gonna use it-
- JSJames Sun
Mm-hmm.
- SASeema Amble
Right
Episode duration: 44:13
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode 9v42ZOdiBJY
