No Priors Ep. 33 | With Replit's CEO & Co-Founder Amjad Masad

1. 0:00 – 3:55

   Introduction

   1. SGSarah Guo0:06

      This week on No Priors, we explore Replit and its new AI tool, Ghostwriter. Replit now has more than 22 million users and recently started a partnership with Google. With Replit, anyone from beginners to experts can quickly build and release fully functional live apps in seconds, now with the help of AI. Amjad Masad, CEO, co-founder, and head engineer at Replit joins Elad and me this week on No Priors. Welcome.

   2. AMAmjad Masad0:29

      Thank you.

   3. SGSarah Guo0:29

      Amjad, just to start, you were working as an engineer at Facebook. What made you decide to start Replit?

   4. AMAmjad Masad0:35

      My journey with Replit actually starts way back in, in college. I was really interested in program languages, and I found it really difficult to just, like, keep setting up the development environment. I didn't have a laptop at the time, and so I was like, you know, "Why can't I open a browser window and start coding?" Turns out nobody has done that, and I was, I was kind of naive and, and thought I, you know, could probably try to do it, and got a prototype up pretty quickly. But my friends loved it in college. Everyone started using it. It was just, like, a simple webpage with a text box and, and a button that says run, which is still kind of in a lot of ways what Replit is. But then that started me on a, like, multi-year journey to actually build the first kind of in-browser sandbox, and at the time, we had a bit of a breakthrough. We were the first to compile Python, Ruby, and other languages to, to JavaScript. I open-sourced that project. Uh, that got me a job at Codecademy. I was a founding engineer there. And after that, I went to Facebook, and I was one of the founding engineers in React Native. We, like, made React and, uh, Jest, uh, Babel, and, uh, a bunch of tools that, uh, JavaScript developers use today. Our goal was to make mobile development as fun and easy and fast as web development, which I think we succeeded at. You know, towards the end of my time at Facebook, I just wanted to see what to do next and kind of looked around and, you know, whatever happened to this online coding space. And there was a bunch of online editors, but none of them really, uh, felt like web native, like you can just share a URL and someone can hop in and code with you, kind of like how Figma or Google Docs work, and, uh, we decided to start in 2016.

   5. SGSarah Guo2:25

      You have made a huge bet on AI as, uh, a company. Can you just talk about Ghostwriter and how it came about and the investment in this area?

   6. AMAmjad Masad2:34

      Yeah, throughout my career, working on code that handles code, right? Whether it's at Codecademy for teaching programming, whether it's my own project, whether it's React and building the runtime around React Native, I always felt like our tools that were handling code, whether it's, like, compiling it, parsing it, minifying it, all that stuff, were very kind of rigid and very laborious. Um, in a lot of ways you're building sort of a classical intelligence system, very algorithmic, um, and a lot of heuristics and, and all of that. And I always thought that you can probably apply machine learning to it, and I started reading around whether anyone had done it. There was this seminal paper in 2012 called On the Naturalness of Software, and basically a bunch of researchers try to apply NLP to code. And what they found is that actually code can be modeled like any language. That's why they call it naturalnesses, because, hey, code is kind of repetitive like language you can do, things like n-gram, basic n-gram model can actually s- start to generate code that, that is compilable. That had a huge impact on me. And every year or two while starting Replit, I would go look at the state of the art on ML on code, and nothing ever really worked o- all that well up until GPT-2. And you

2. 3:55 – 11:09

   Impact of AI on Code Generation

   1. AMAmjad Masad3:55

      can, like, take GPT-2 and fine-tune it and, like, try to make it write code and was, like, kind of okay, but obviously GPT-3 was like, "Okay, it's here." And that's when we started building what became Ghostwriter. Initially it was a bunch of tools kind of sprinkled all over the Replit IDE, explain code, you know, generate code here. And then we wrote our kind of autocomplete product, which is kind of like Copilot. As you're coding, it kind of gives you suggestions. And then we added the chat product, and that all became sort of Ghostwriter, our, our AI suite.

   2. SGSarah Guo4:29

      It's amazing that this idea of type-ahead code autocomplete with local context has become as widely adopted as it has. Like, your tool and Copilot are sort of a handful of tools that people, like, very much accept have changed workflow. Maybe if we just project out a little bit, where do you think that the, like, next big leaps in productivity in software development come from in terms of AI? W- what else is gonna happen?

   3. AMAmjad Masad4:55

      It's an amazing tool, but I think it's still very primitive, and we haven't fully explored the full capabilities of even GPT, but also transformer models applied to code in general. There's a lot to do. I think there's a lot to build on the sort of the layer just around the models, so the basically how do you give models better context, how do you give them tools, the ability for the model to actually be able to go out and read a file, install a package, evaluate code, the ability for these models to be more agentic and be able to kind of write entire features o- uh, by themselves. And so that, that's all the work just around the model itself with the model in the center. And then there's a lot of work on the models themselves that haven't been, you know, fully explored. The way we train models today is we just give it a large corpus of, of, of code, but you can imagine different ways of training models, and we've played around with different techniques at, at Replit. There are ways to make the models... give models more intuition about how...... code execution happens as opposed to just static code. 'Cause you're training them on static code, they understand the structure and syntax of code, but they don't really understand the semantics of code. So you can imagine a way to train models where you're not only feeding them code, but you're also feeding them the results of the evaluation of that code. You can imagine ways of training models not just to be good at code, but also be good at the editor, the debugger, uh, everything that comes around code itself. And so I feel like we're in the very early beginnings of it. I thi- I think for professional programmers, perhaps it's a, like a marginal improvement, and, like, it- it some people claim it's a huge margin. Based on our research of how people are using our products, we think it's 30 to 50% more productivity on coding itself. Obviously, there's a lot of things other than coding, such as debugging, deployment, meetings, being able to translate specs into- into code, and everything that we do just around the act of coding itself, AI hasn't touched yet, and I think it will touch every part of the software development lifecycle. I think actually the people that are getting the most amount of value today are- tend to be the beginners, tend to be the people who don't have the skills typically to, like, start a startup, and we're seeing a lot of those on Replit. Just today, we've profiled a- an entrepreneur that has a quarter million ARR startup in a few months, and they haven't... You know, earlier this year, they didn't know how to code.

   4. EGElad Gil7:34

      (laughs)

   5. AMAmjad Masad7:34

      And I think this is where most of the impact is right now. But I think if you project forward, those people will also get a lot more tools and gonna get more productive, gonna hit superhuman productivity. But the professional programmers, this is where actually you start to have that 2x, 5x, 10x multiple in productivity.

   6. EGElad Gil7:51

      What- what do you think is the timeline on that? Do you think that's a year away, three years away, five years away? I'm just sort of curious about when we both see really massive productivity gains, and then also when we move towards this more agentic world that you mentioned, where, you know, you have AI go out and do things on your behalf in terms of generating a set of features or grabbing a- a packet from- a package from somewhere or doing other things for you.

   7. AMAmjad Masad8:13

      My feeling is this is within reach, even in the current capabilities. Like, if you hold constant the current capabilities, this is when reach- within reach in the order of, you know, six to 18 months. We're in the demo era of GPT models. Right now, we're doing the hard work of building infrastructure on Replit. Like, we are building this service that sits inside the container, uh, that embodies the AI in the development environment such that it can do things like go to the internet, like install a package, like read a file, write a file, start another project, git clone, do all of that stuff. That's a lot of work. And that's not just one model, it's probably multiple models. And so within the current capabilities, there's a lot more to do on the infrastructure side. So I think we're within months of actually having really cool basic agentic experiences where you can give a high-level task to a model or to an AI, uh, such as build a login page, and for it to do a decent enough job to give you, like, a starting point for that. Now, if you try to project, like, the actual advancement in AI itself, increased context size, increased performance, better prices, potentially, like, even new- net new advancements or new ways of training code models, it- it- it's- it becomes really hard to predict. Like, I think software could fundamentally change in the next four- three to five years, where it looks totally different to someone sitting- sitting today and looking at it into the future. Maybe it looks something like programmers tend to be more like engineering managers, like even- even ICs, you're just, like, working with employees. Uh, and those employees are m- mainly AI, and you're- most of the time, you're reviewing code and giving instructions, and you're not actually, like, spending 50% of your time typing. I- it's hard to know, but I'm confident to say within the current capabilities, there's still a lot to do over the next six to 18 months.

   8. EGElad Gil10:13

      And then, I guess one other thing that you have as a real advantage for your product is just you have such user liquidity, right? So you have, I think it's on the order of 22 million users or something in that, that range. And that's- obviously can create an enormous feedback loop for training your own models, for reinforcement learning, et cetera. There's a paper that recently came out from Google comparing a- a very specific type of reinforcement learning through human feedback versus AI feedback, and it roughly netted out, right? You could start to use AI for certain tasks to effectively train itself. Like, how do you think about that in the context of code and sort of where you're heading?

   9. AMAmjad Masad10:46

      I think it's easy to overstate data modes. You know, you need to be measured about that. I think the real advantage of Replit, uh, is actually just the platform, just the end-to-end platform. 'Cause I think, you know, you can be- you can try to build these code models, but without having the platform to be able to apply it on, to be able to get feedback from, uh,

3. 11:09 – 14:35

   Breaking Down Barriers to Entry in Development with Replit

   1. AMAmjad Masad11:09

      it gets very difficult. But also, even if you just have a code editor or editor extension, that doesn't give you all the feedback that you would want, because you would want feedback from deployment, you would want feedback from execution of the- of the code, and not just user feedback. I think user feedback, you're right, for RLHF type things. I don't think that's the real advantage. The real advantage is the end-to-end journey from the first line of code, to the deployment, to getting crashes in production, to making edits, to all of that feedback cycle, which typically, in most cases, is divided between GitHub and VS Code and AWS and all of that, and the real magic of Replit is putting all of that together. And this is, I think, where we're gonna get the kind of richness of training data that allows you to train over a large action space, and that's- that's really exciting.

   2. SGSarah Guo12:06

      Can you talk about the decision to, um, train models versus use existing APIs? Like, was it just a latency thing? How did that happen?

   3. AMAmjad Masad12:14

      So when Microsoft did it, like they took GPT-3, they distilled it down and they hosted it on their own infrastructure. They did a bunch of, uh, caching and a lot of things like that. If you're someone who's just using the OpenAI API, you couldn't do any of that. And it was very expensive, it was very slow. Um, and even now, you know, there's no completion models. Now all the models are chat models. You know, they- they're not gonna be releasing any completion model going forward. And so if you wanted to build a completion-based product, it was actually fairly difficult to do it using commercial APIs. And we felt like we can, we can make a model that's both cheap and fast and good enough for that autocomplete use case. And we found that the three billion parameter size is kind of the nice sweet spot where you get enough raw IQ from it. 'Cause like one billion felt kind of too dumb in a lot of ways. And it was gonna be cheap enough and fast enough to host and- and be able to do a fast inference from. Um, and it was a time after, you know, the Chinchilla paper first came out and LLaMA had been announced, and the idea of just training them longer had just been in the air and- and we're like, "Okay, what if we- we apply all these open source tokens on a smaller model?" And then we applied some more tokens from Replit's data that gave us a 50% improvement over the model that we open sourced. So we open sourced the base model 'cause we think that's sort of the ethical thing to do because we trained on open source data and we wanna give it back to the world. And our state-of-the-art model was better than, uh, most open source code models at the time, even things 20X its size. But- but the need for it came from a product need. Like, we needed to build that product. I think a lot of founders now see it as a point of pride or status to train a model. And the moment we put out the three billion parameter model, like three or four cop- copycats came out of, uh, after that. And I- I- I'm not a fan of what's happening in- in the open source world where everyone's competing over these silly benchmarks. I think it's important to start from a product and start from a user and customer need and then go from there. Right now, if we could do what- what we can do with LLaMA, we'll just use LLaMA. If we have to train a model, we'll

4. 14:35 – 20:32

   The Impact of Open Source Models, Meta/Llama

   1. AMAmjad Masad14:35

      train a model. And that's how we always approach building this company is start from the customer and the use case and then solve any technical challenge that you- you get along the way.

   2. EGElad Gil14:46

      One of the interesting debates I- I've been hearing about LLaMA recently is how long will Facebook continue to sponsor them as open source models? And, you know, so far it seems like LLaMA 2, for example, is the one that a- that a lot of people are viewing as the most performant with LLaMA 3 perhaps being a big step forward in terms of catching up relative to, uh, GPT-4 and sort of other closed source models. What do you think happens if- if Facebook stops open sourcing? Do you think somebody else steps into the void? Do you think everybody's kinda on their own? Like I- I'm a little bit curious how you think about the open source world relative to models given how important LLaMA has become so rapidly.

   3. AMAmjad Masad15:23

      So when I came to the realization early '22 that we can't build the product that we wanna build using commercial APIs, I was kind of depressed for a moment. It was like, oh, you know, we- we didn't have, you know, a lot of funding or whatever. It's like okay, you know, Microsoft's gonna be the only one that's gonna be able to build these kind of products. And, uh, I got a meeting with Zuck actually, and I pitched Zuck on this idea. And- and I remember around the time he did the Open Compute Project. So what Open Compute was, like Google and AWS are gonna keep all their data center secrets because it's a- it's a company advantage. But Facebook did not sell compute, did not sell cloud. And there's a lot of other companies that did not sell cloud. Cloud was a complement for them, was not a competitive advantage. And so they pooled their resources together and they worked on this Open Compute Project to be able to optimize computers for data centers, and it was a huge success. And what I told Zuck at the time was like, "Hey, why didn't you do that for LLMs?" Obviously open source AI is gonna be great for you because you can build better products as a complement. Uh, and so it's sort of the traditional, like to the- the classical like commoditize or complement business strategy. And at the time he just, uh, kind of nodded his head and I don't know if- if it was a novel idea. Maybe he was already thinking about it. But that's basically what he's talking about. I- I think in one of, um, one of his interviews he talked about Open Compute as an analogy for what they're doing with LLMs. So I think it makes a lot of sense for them to do that. What I'm actually surprised by is how little industry participation they've gotten. And I think the difference between LLMs and Open Compute is that the AI safety angle makes- makes it a little toxic for a lot of companies to touch. Like they wouldn't want to be associated with something that spews something that- that is toxic or they don't wanna touch. So that's another wrinkle that I didn't appreciate and- and it takes a certain amount of guts to release an open source language model, especially with political heat that Meta's getting from that. Now what happens if they stop? Uh, yeah, I think- I think it would be bad. I think it would be bad for the ecosystem. Will there be another player that will emerge? I think the- the problem a lot is that, as you know, there isn't a lot of guts in the industry. That's our problem. (laughs) It's like, and Zuck has balls, right? (laughs) So...

   4. EGElad Gil17:51

      Yeah, no, I think it's been super impressive what Meta's done with LLaMA and I think it's really helped push forward the entire industry in an amazing way. And so really kudos to the company for doing it. I think beyond guts, which I agree with, there's also the cash side of it. I actually think the money exists in the ecosystem to pay for open source and to help train models. I think the talent base is much harder and the magic of what I think Facebook did with LLaMA is they had an amazing talent base and then the capital to train something and then the guts to release it.And if you look at every other open source software wave, almost every major one had a big corporate sponsor, right? Linux was sponsored by IBM to the tune of like a billion dollars a year for some period of time. WebKit was obviously sponsored by Google and Apple. Like, there's lot- lots of the really big things that were important in terms of fundamental pieces were sponsored by a company at one point or another. And I think in the absence of Meta providing not only capital and guts, but also talent, there may end up being a gap. But that's one of the reasons I was asking the question, is they have that really amazing intersection of guts, money, and people.

   5. AMAmjad Masad18:55

      Right.

   6. EGElad Gil18:56

      And I- I- you know, it's- it's unclear to me what the replacement there is.

   7. AMAmjad Masad18:59

      Yeah, it's also unclear, like you said, which other companies would be able to participate in that. And I'm sure they tried to do outreach and they tried to work with other companies. Uh, I think on a long enough timescale, uh, obviously there's- there's gonna be open source models that are as good as commercial models, uh, but, uh, the current landscape makes it hard to- to kind of imagine how it would play out. And also, there's this whole question of, like, if the architecture gets more and more complex, it actually makes it harder and harder to open source, right? Like, there are tools and there are services. Tools especially are easy to open source because you can... it's a binary, you can run. And basically, that's what we have today, you know? You can just run, it's- it's like a binary blob, right? But if there's, like, mixture of experts and there are components, stateful components for running this stuff, then it becomes exceedingly harder to create an open source alternative.

   8. SGSarah Guo19:58

      I'm curious if we can talk about bounties for a second.

   9. AMAmjad Masad20:02

      Sure.

   10. SGSarah Guo20:02

       Because you're somebody who, like, clearly believes in agentic development, and then you stuck a marketplace in your product that allows developers to collect dollars for code that they've produced. Do you think we end up doing that with development agents or how does that play out?

   11. AMAmjad Masad20:18

       So, uh, the- the idea behind bounties is that at Replit we have all these talented people across the world that are learning how to code and that are actually, like, becoming great programmers

5. 20:32 – 24:26

   Bounties, Agents who Make Money

   1. AMAmjad Masad20:32

      that kind of no one knows about. And I felt like it was almost, uh, our duty to, you know, make it known that there's so much talent around the world that you can kind of leverage. And a lot of these people don't have any resumes, don't have a lot of skills to be able to get- go get jobs themselves. So, part of it was this community building activity. Can we get all these people money? And like, we have people in Africa, uh, India, Thailand, uh, Serbia, all sorts of countries now that are, like, making money on bounties. The other thing was my view at the time, or my thesis that I wanted to test, was it's gonna be a lot easier and it- it's gonna be a lot cheaper to be able to get basic software tasks done because AI, like I said, will have, in the short term, the most impact on beginners. So, beginners are actually getting a two, three, 10X multiplication on their productivity. You learn a little bit of code and now you can actually build a product that makes 200K ARR. And so the idea was that, okay, how do we leverage this, uh, this opportunity? How do we monetize it? And so, okay, human plus AI, a human who knows a little bit of code plus AI could actually be a really great developer. And so yeah, turns out it was right. You can go to Replit and get a- get a startup, a prototype for a startup or an MVP for a startup for as little as, like, $50 or $100 in- in some cases. M- my hope as well for Replit in- in general is that there is some kind of currency or way to transact built into the- the platform. And that was upstream of the idea of bounties. Uh, bounties was- was kind of the first step into that direction. My view is that there's something broken about open source where the currency in open source is sort of stars, right? Like, how many stars you get, and it's sort of a fake currency, and creates all sorts of weird side effects. We talked about how open source tend to be... the most impactful open source projects tend to be corporate sponsored, right? But that doesn't really have to be the case. I think the software economy is missing some native currency for exchange of value. Um, and open source is good and- and- and great. We want more of it, but also we want it to be sustainable and we want the people to be rewarded who are contributing to it. And then there are all sorts of things you can do with a platform that has concept of money and transactions built into that. So, if your development environment and your deployment environment has a currency built into it, what kind of collaboration and ways of doing things does it unlock? One idea is that you sort of publish a module or publish a component or publish a service to the network and I- and you kind of price things per function, and I can sort of call these functions and be able to have a wallet in my application and be able to kind of spend money on the different functions and different, uh, databases, different services that I want to- to compose to create an application that someone can consume. And AI was, uh, was an interesting sort of addition- additional component to this because I think agents, uh, in the future will also be able to- to use money to- to do transactions, do things in the world, right? So, right now our AIs, again, we're using them as these copilots, right? Uh, they're basically something that's helping you do some kind of action or task, whether they're- they're predicting the next step. They're literally doing the- the, you know, auto aggressive action. But what...But, you know, what if we get to this place where we actually have these agents that can act, uh,

6. 24:26 – 32:29

   The Missing Data Spec-to-Code

   1. AMAmjad Masad24:26

      semi-autonomously? What would you want them to do? Uh, obviously we want them to interact with the wider, wider internet, so that they're coding, they're evaluating code. But you also would want them to be able to spend money and make money. And I, in the future, I could see people creating, um, bounty hunters that are fully automated. So can you create an AI that can go earn money for you while you sleep? And I think that that would be a really cool, cool idea. Um, and, and finally, I think on the data side of things, one thing that I thought was missing from the data landscape around code is complete project spec or product spec to actual end product. And, and so we have this data now, where you have a literal, like, complete end-to-end project spec and the actual code accompanied with it.

   2. SGSarah Guo25:19

      Yeah, we host this college hacker group for my fund, um, for this commit hackathon. And it is very cool seeing how sure some of the, like, 19-year-olds are that AI is going to change development such that they don't really need to, like, learn to code in the same way before, right? Like, and if you ask them, like, "Well, do you need to know architecture?" And they're like, "Nah," like, you know, "It's more prompt guidance." Like, "Do you," like, "Who's gonna debug?" And they're like, "Oh, another agent," right? And these are people who, you know, 20-year-old MIT kids that are, like, designing and building their own GPUs, they're very interested in computing. But the orientation toward, I would rather orchestrate some of Amjad's, like, bounty hunter-type developer bots is, is very strong. So, I'm excited to see if that happens. I think so, they'll be some of your first users.

   3. AMAmjad Masad26:09

      I, I think embedded in their answer is this attitude of, like, if it becomes a problem, I will learn it, right? If the AI can't do it, I will learn it. And I think that's the right attitude for hackers, is that you learn just in time to be able to do that task.

   4. SGSarah Guo26:24

      Yeah, absolutely. I'm not gonna, like, move myself up the MIT computer science curriculum hierarchy. I'm gonna go figure out what I wanna build and then kinda work backwards, and look, I can jump 20 steps forward because of Ghostwriter now. It's a very cool setup.

   5. EGElad Gil26:38

      And one of the things that you mentioned was, you know, different aspects of design, payment, um, different activities, et cetera. Do you, do you think from a payment perspective anything needs to change in terms of financial system? Is it crypto? Is it just centralized payments and it's just the, the standard old rails?

   6. AMAmjad Masad26:55

      You know, I'm, I'm very pragmatic, uh, about, like, technology. Like, if Stripe gets us there, Stripe gets us there. If it is Bitcoin, then, then it's Bi- Bitcoin. I, I think it's a combination of both. I, I don't view them as that separate. Actually, like, one interesting historical tidbit is that Stripe, Square, and Bitcoin started the same year. Um (laughs) -

   7. EGElad Gil27:17

      And Satoshi was behind all three, it turns out.

   8. AMAmjad Masad27:20

      Right (laughs) . Maybe (laughs) . It's actually Jack.

   9. EGElad Gil27:23

      Yeah, it's Jack and Patrick working together.

   10. AMAmjad Masad27:26

       (laughs) Right.

   11. SGSarah Guo27:27

       (laughs)

   12. AMAmjad Masad27:28

       But, you know, it, it, it was obviously i- in the air, this idea of, like, the internet is missing this core feature. Um, money is a coordinating, coordinating mechanism, right? Like, there's so many cool things you can do with money as a, as a primitive. Not only for people to be able to kind of buy and transact and, and value, but, you know, all the things around the people in the Ethereum community that it sort of invented with staking and ... When I was designing bounties, I really didn't want it to be this, like, rigid thing where people apply and you accept them, whatever. What if you could, like, stake your currency and say, "I'm gonna do this task, and if I don't do it, you can take $100 from me. Uh, I'm gonna do it in five hours, and for every extra hour, you can slash my funds by X percentage," or what have you. And so you have this market dynamic where people can kind of bet on how confident they are they're gonna be able to do the task. And, uh, and then, um, the coordination happens sort of semi- uh, automatically, 'cause, like, the biggest bet, you know, takes that task, and so on and so forth. So, money is, could be a programmable primitive, uh, and I think that's true, and it's, it's really hard to argue with. You know, um, I think there are a lot of drawbacks to the current financial system and all the problems that, that come with it. I still believe, I think Bitcoin is gonna be long-term the TCPIP of money, but there's gonna be a lot of serv- services that could also be Stripe and others and Square that it's built on that, like, on that sort of rails.

   13. EGElad Gil29:10

       Cool. I think we covered a ton of ground. Thanks so much for joining us today. I- it was a really great conversation.

   14. AMAmjad Masad29:15

       Yeah, likewise. Yeah, thank you for having me.

   15. SGSarah Guo29:17

       (instrumental music) Find us on Twitter, @nopriorspod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.

Episode duration: 29:37