No Priors Ep. 79 | With Magic.dev CEO and Co-Founder Eric Steinberger

1. 0:00 – 0:45

   Introduction

   1. SGSarah Guo0:00

      (music plays) So welcome to No Priors. Today we're talking with Erik Steinberger, the co-founder and CEO of Magic. They're developing a software engineer co-pilot that will act more like a colleague than a tool. And Erik has a really fascinating background between work on- at Meta on different types of games, running, uh, Climate Science, which was a nonprofit focused on the climate world, and now, of course, developing a- a incredibly interesting AI model and system. So welcome to No Priors today, Erik.

   2. ESEric Steinberger0:30

      Uh, thank you so much for having me. It's great.

   3. SGSarah Guo0:32

      So- so you have a super eclectic background. Um, could you tell us a little bit more about your, you know, what you worked on in the early days, how that evolved into working on AI, and sort of the path you've taken?

   4. ESEric Steinberger0:43

      Yeah. Um, yeah, thank you. Um,

2. 0:45 – 4:01

   Eric’s journey to founding Magic.dev

   1. ESEric Steinberger0:45

      so I- I guess when I was 14, I just had my midlife crisis and, uh, thought I had to do something important with my life. Uh, and spent a year trying to look at everything. Uh, it was pretty stupid. And, uh, basically I looked at the things like string theory and like all the things a 14-year-old would look at and be like, "Okay, what can I spend my life on?" Uh, and, uh, eventually, uh, my mom got me a book on AI and I didn't read it, I'm sorry. Um, but it was like the idea, um, was sufficient. Uh, so I was like, "Okay, this could do anything." Uh, and so- so you should just do that, and then it does everything. And then, um, it seemed plausible that, uh, you'd need to do reinforcement learning. Um, so I didn't know how to code at the time. Um, then, uh, learned to code, uh, uh, over a couple years. This was sort of in high school times. Uh, and then, um, it seemed plausible that you'd need to do reinforcement learning because otherwise you'd not be unfounded. Um, so- so I sort of just started working on RL, uh, played around with things for a bit. Um, and, uh, e- eventually, uh, reached out to someone at DeepMind, uh, to basically I was like pitching this like multi-page email that was like, "Could you like do like a mini PhD thing," where I'm like, "I'm a complete newbie, but if you can bash me, just please bash me like every two weeks and like tell me how to be a good researcher." And so- so eventually, um, I got like reasonable and then did some actual research work and- and worked with, you know, a few other people, including Norman Brown, um, who on developing new RL algorithms, uh, to be more sample efficient and just generally better and faster or whatever, uh, was the goal at the time, um, to- to- to solve, uh, whatever environments we were interested in at the time. Um, so yeah, that's how I got into it. I- I have no background in language models when we started, um, Magic at all. It- it just seemed... I just was like totally not on my radar. Uh, I was like, "Oh, wait a second, like if you take this and this and put it together, like- like maybe this works." Um, and so- so then- then I- I sort of, it felt like this huge relief of, uh, uncert- like, uh, uh, certainty relief of like where AGI would come from, uh, 'cause you just put those two things together and then- and then they will work, uh, was the sort of hope. Um, but yeah, yeah, my original background is in RL, and, uh, trying to come up with algorithms that sort of, yeah, just like have better structures, uh, to be more sample efficient and faster or better conversions.

   2. SGSarah Guo2:49

      And a lot of the emphasis on Magic is sort of twofold. Um, on the one hand, you're doing a large scale, uh, custom model, um, specifically in part focused on code, and then you're also building sort of out the product suite that can really help, um, address, uh, coding and working, uh, on the- the software development side. How did you decide to start Magic, and why- why focus on that, um, versus other aspects of AI?

   3. ESEric Steinberger3:12

      It sort of came from a place of working backwards from ETL. If you, uh, your end goal is to have a system that can do everything, uh, you can reduce that to building a system that can build that system. And so, that minimal system is a system that writes code and comes up with ideas and can validate those, uh, uh, by writing code and running experiments, which is still like of- in the same order of complexity as the full thing, but at least we don't have to train . Uh, and like, you know, we don't have to think about 10 billion other use cases that, uh, everyone, uh, building, uh, general domain products has to think about. Uh, we only have to think about code. So it's a lot simpler in all aspects except compute, and slightly simpler on the aspect- and slightly cheaper on the aspect of compute. I, uh, think it's not a lot cheaper. I probably overestimated how much cheaper it would get, um, on the compute side, uh, at the beginning. Uh, but that... the other things are simpler, I think.

   4. SGSarah Guo3:58

      And you've taken a slightly different architectural approach, right,

3. 4:01 – 10:53

   Long context windows for more accurate outcomes

   1. SGSarah Guo4:01

      than what a lot of people are doing in terms of just going with a traditional transformer model. Is there anything you can talk about there? And, you know, you also had a very early emphasis on one context window. And so I think you were the first model, um, that was publicly announced at like five million tokens, and I think that was like a year ago or something now, right? So you were way ahead of the curve on that.

   2. ESEric Steinberger4:18

      Yeah, that was a year ago.

   3. SGSarah Guo4:18

      Um, so I was just curious like how you chose the specific architecture that you decided to focus on and how you decided to focus on context windows before most people thought it was a thing, and, you know, I think you've been very sort of pioneering on a lot of these areas.

   4. ESEric Steinberger4:29

      Um, thank you. Yeah, it- it seems important for models to have the ability to learn from long histories of their own and their collaborators' actions, as well as take into account, uh, a- a large amount of, uh- uh, fast changing data. And so, if you imagine having 10,000 employees, uh, or, you know, everyone on Earth having their own model, uh, and wanting to feed on all their data, you can now fine tune everyone's model, uh, maybe do some lower tricks. Uh, but- but in practice, uh, context just works better. Uh, and- and that's... Like- like in context learning is the magical part that came out of transformers. Like this- this is- this is what makes them great. I think of that as some sort of a- as an online optimizer in a sense that instead of compressing, uh, a set of data, you're trying to learn an optimizer. So- so the perspective we take on models is- is instead of... Uh, this is wh- one of my colleagues put it this way, I find it- find it, uh, uh, very fitting, so I'm- I'm- I'm stealing his- his quote here, is instead of bringing the data, uh, to the, uh, compute, um, we're bringing the compute to the data. Uh, so- so you- you have a set of stuff, and our model acts on that stuff, rather than having like a giant model that, um, you know, you have to sort of work around. So the- the whole system is designed for this. So- so yeah, like a y- a year ago, we- we announced five million, um, and, uh, uh, by- by the time this is out, uh, uh, we- we might- we may have, um, announced a larger number. Um, uh, the- um, the main reason being that you would want to deploy these things-... for very long horizon trajectories and you want them to spend a lot of time thinking and you want the model to remember all of that. Uh, and you can't really do that by fine-tuning because you'd have to fine-tune every, like, whatever many thousand tokens your context window is long.

   5. EGElad Gil6:13

      I don't know if people would think of this as a trick, Erik, but, like, uh, you know, can you give us some intuition for why, you know, quality of output would be better than, um, a retrieval-based system?

   6. ESEric Steinberger6:22

      Uh, yeah. I mean, so you can come at this from two perspectives. I can explain it mechanistically and I can... uh, the other cheap way is to just point out Richard Sutton's Bitter Lesson. Uh, uh, retrieval selects a subset of data for one completion. Our model sees all the data all the time. Uh, clearly, a subset of data for the whole completion is a subset of all the data all the time. So if retrieval was optimal, our system could learn it. Um, and, uh, uh, i- it just turns out that it's not optimal. That's the sort of mechanistic, I guess, explanation or logical explanation as to why long context will be better. Uh, you could make arguments around the quality of long context if it weren't sufficient. If it wasn't sufficiently high quality, uh, maybe having a short context window and pulling in some data is better. So you obviously have to re-evaluate this, like, you- you know. But in principle, yeah, on the assumption of Richard Sutton's Bitter Lesson, you would want the thing that can learn your heuristic rather than the heuristic.

   7. EGElad Gil7:13

      The, uh, the other area of, um, expansive exploration right now amongst researchers for better AI code generation tends to be test-time search. Um, so, you know, m- more compute at inference time. Uh, you know, speaking of a, speaking of a mutual friend in Noam, like, how do you think about this?

   8. ESEric Steinberger7:31

      Well, so you can think of model performance as some function of training compute times some function of inference time compute. Now, th- those are specific functions that are just scaling law things that you can, like, model, but the, the general way to think about it is some function and some function. And then you would want to estimate how much inference you're gonna do and how much, what your total budget is, and then you would want to create the optimal trade-off, um, in your allocation of money. You also want to consider the distribution of outputs. There will be users who will want to spend less money and users who will want to spend more money, and this is likely going to follow some sort of very sort of spiky distribution where there will be, like, four users spending a million dollars and, you know, four billion users spending 10 cents and a curve in between. It seems, um, strictly beneficial to be able to provide that choice. So instead of training the, putting all the compute into training and having that $1 million inference performance be purely from the training compute, which is just hilariously inefficient, you can, you can allow the user to choose their, their thing. So, um, or rather you can just deploy multiple things. The reason I can talk about this now is because everyone, like, is d- is getting, like, everyone gets this. Um, but, but basically you clearly want to be able to regulate the amount of compute use at this time. Now, it turns out this is actually not trivial. Um, (laughs) like, like, doing this is hard. Uh, like, c- finding, like, the right algorithms to do it. Um, that being said, people have done it in RL for a decade and so to those, um, you know, uh, I don't, I don't want to name people. I, I don't... Noam obviously is one of them, but there are others at other, other labs as well. Like, there's a set of people to whom this is, like, uh, the opposite of a surprise. But, um, it's still not trivial 'cause it's the general domain. There is no game.

   9. SGSarah Guo9:07

      Yeah. The analogy I've heard, um, is sometimes when you're asked a question, you kind of pause and think about it, and that's almost like your inference time compute. So you're basically investing some sort of resources, uh, to actually consider the problem at hand versus just spot react to it.

   10. ESEric Steinberger9:22

       Correct. And there, there are things you have to learn during training. Like, if you are asked to write a piece of code in a... and you've never learned coding, you can spend... yeah, the inference time compute you have to spend is ridiculous. Uh, like, like, you're gonna have to, at inference time, learn programming. Which, which may be... I actually think this is possible, but, uh, I, I... and it's also crazy. Um, and, um, the... this, this is like, clearly this is shared among, like, everyone who, like, posts a query. So, so this... it's stupid not to bake this into training. Even the best mathematicians in the world for the frontier of mathematics require a long time to solve the problem. Uh, so, so, so, like, I would love to have a Terrence Tao in my computer, but I would then still need to run Terrence Tao for a year, uh, of, of human thinking time. And, and, like, Terrence Tao will not just token by token spit out, uh, a proof for Riemann or whatever, um, you know. And so, so I think to achieve things like that, um, through pure training compute deployments, uh, inference time compute work would need to drastically fail simply because it is a shortcut. Like, you can also train this model for a quadrillion dollars. Uh, maybe you can't actually because there's no data. But, uh, say you could. Um, well, what if I just need a billion? Like, uh, you, you get the idea. So, so I think it's, like, this fundamental trade-off that you, you want to be able to bake into... and the humans can do this exactly as you say. Uh, um, and this applies to, to all parts of the, the workflow that you ask them to do.

   11. SGSarah Guo10:44

       And I guess if the goal of the company is eventually to build AGI, um, how does that impact your choices from a design perspective relative to some of these trade-offs? Or is it more

4. 10:53 – 15:18

   Building a path toward AGI

   1. SGSarah Guo10:53

      you were going to iterate till we have a system that's very good at writing code and then it bootstraps its own next version? Or how do you think about your roadmap relative to AGI itself?

   2. ESEric Steinberger11:01

      Yeah, that's a great question. So I remember one of our first conversations. We were talking about sort of the step function relevance of safety risks, let's say, right? Where, like, there's a lot of stuff people are panicking about that really doesn't matter in the short term, uh, for, like, the grand scheme of society. And, like, some people will get pissed with me saying this, but it's just what I actually believe. Uh, I, I just don't think this is, like... the, the current complaints are, like, similar to what we have seen in, like, all other technologies and, like, totally resolvable. And, but, but then there comes, like, the evolutionary one (laughs) and I was like, "Okay, shit." Like, humans succeeded in the world because we're smarter than chimps and we're smarter than bunnies. And, uh, you know, like, like, bunnies do not rule the world. Uh, and, um, that would be cute probably, but also, um-

   3. SGSarah Guo11:41

      (laughs)

   4. ESEric Steinberger11:41

      ... uh, it, it's, it probably wouldn't be as nice for us, uh... and, like, here we are turning ourselves into, uh, bunnies and apes and, and creating this thing that, you know, we're all thinking is gonna be way smarter than we are. This is insane. And, um, so the reason I think, like, the sort of recursive approach that, that you're mentioning here, uh, hinting at... and obviously, you know, this is what... you know, thi- this is, this is what we were founded to do. Um, it's exactly the... I, I think the only way to, to sort of...... reasonably approach this is to iteratively ask your model, uh, to solve, uh, uh, a- a- a- a- alignment and safety at that stage. Not- not, you know, surely you can also ask it to solve your product level problems, but like that- that's- that's nice, but th- that's not- that's not the fundamental objective. The fundamental obje- objective is to iterate towards AGI with a safety boundary and there's just no knob like this. There is just no knob like this in the, uh, human world. Like, you- you- you can say, like, "Oh, I'm gonna, like, spend X percent of my resources on this," but- but that doesn't indicate an outcome. Um, like, you- you- you can't actually control it. Uh, but- but if it's, like, compute, you can- you can somewhat do it. Um, so- so I think it's just a- the most promising path we have, uh, both on the, um, progress towards better models because you can... If y- if you just have, like, a th- 100,000 brilliant researchers in your computer, uh, th- that is better than having 100, uh, especially if you can connect all their brains. Um, and- and- and- and then if you can say like, "Okay, we think this is safe to do. Like, let's go do it," and then you have the next thing and then you're like, "Okay, like, maybe we should use that thing to do some more safety research," y- you can bootstrap. And so- so just very pragmatically, uh, even if there was goodwill, I just don't know that we have the, uh, the mechanisms to prioritize safety without automation. So- so yeah, that's the primary goal. Um, now the good thing is as we pursue this, if we're successful. If we're not successful, then I'm, uh, deeply sorry to all of our investors, um, and (laughs) including you.

   5. EGElad Gil13:30

      (laughs)

   6. ESEric Steinberger13:30

      Um, but if we do succeed, this will, uh, especially if we are, uh, uh, first or early to succeed, um, drop a beautifully, uh, profitable apple off a tree, uh, that- that happens to automate a good chunk of what we call work today, which is another thing that I think is, like, a responsibility of our company to do, just to say like, "Look, if- if this type of thing works, it's not really an assistant, it's- it's- it's just, like, you just put it there and, like, you talk to it and it's like a colleague." And- and that's great. Like, I think e- even the economy does the same thing, uh, same outputs with less input and... Or even more output with less input. This is fantastic for the world if we- if we just do it well. Like- like- like, capitalism and competition can be great and- and, like, progress, uh, the entire history of progress comes from this. Like, we'd all be farmers if we weren't in favor of automation. It's in- i- it's, like, it's scary, I get it, and- and, you know, if you don't think about it all, like, for the whole way, like, it's very scary, especially if you're affected that you're not in one of the lucky seats I'm in, uh, for example. But- but- but ultimately it's good. And, uh, so- so- so yeah, I just- I just basically, like, when we started tried my best to think through this and- and this seemed like the most productive path, um, forward. And- and again, it... I think, uh, uh, just, like, we're lucky and to be in a position where we both get to pursue an incredibly valuable, uh, product that- that's a new generation of thing, um, that just d- doesn't really exist yet. It's- it just there is no, like, thing that does your work for you. Um, I think ChatGPT was one of these, you know, mo- m- monumental moments of a new type of thing. Uh, but you just, it... Uh, the first AI assistant, this is, like, mind-blowing to all of humanity. And- and I think this will happen once more and then maybe once more when you, like, can just, like, query the solution to Riemann. Uh, and- and then- then that's probably it, uh, uh, uh, at least in this domain. Uh, and- and so I hope we can be a part of the second one, maybe the third one, um, um, and on the product side. Uh, but we- we chose the domain we chose because of what I, what I said previously.

   7. EGElad Gil15:17

      Can you...

5. 15:18 – 17:34

   Defining what is enough compute for AGI

   1. EGElad Gil15:18

      Let's talk about those two pieces separately. Like, you said, um, you think it'd be slightly cheaper than a more generic AGI eff-... Like, the last conversation we really had one-on-one was about how much compute you might need, right?

   2. ESEric Steinberger15:29

      Yeah, I was wrong.

   3. EGElad Gil15:30

      Na- um, so, like, I guess, uh, contrast this effort to, like, a more generic effort because clearly the large labs that are using a great deal of human-generated coding data, they care about the use case. So, like, you know, what makes the efforts different?

   4. ESEric Steinberger15:43

      I think a lot of it is, uh, direct competition. Um, now, I think that, like, there are totally things I could say here that sound, like, entirely believable, um, as, like, major differentiators, but I think fundamentally they are and I'm gonna talk about them, but I just also wanna highlight that I think at the core, uh, the wider world has understood that code is very helpful and there are ways to deploy compute, uh, to improve coding performance. And so therefore, because m-... a lot of computers deployed in this domain, uh, uh, the- the need for compute is- is large. Um, that said, also in parallel with the release of this, we're gonna be announcing, uh, uh, wha- wha- what is, uh, going to be one of the largest clusters, um, uh, to- to ever be built. Um, so- so we are correcting for that. I- I was wrong when we initially talked. Uh, you just definitely need a lot of compute. Um, now there's still, I think, a notion of enough, uh, but it- it might be a lot. (laughs)

   5. EGElad Gil16:41

      (laughs)

   6. ESEric Steinberger16:42

      Um, the, um... The- there's like this interesting, right? 'Cause there is enough compute for search, for Google Search. I don't think... If you- if you took GB- uh, if you took Gemini, uh, 1.5 Pro and you put it into Google and you, like, did all the, you know, fine-tuning properly and then you swap it with G- with Gemini 4, I don't think anyone would notice. Uh, unless, like, you could, like, prove Riemann, right? But- but, like, it's- it's good. Like, 99% of users, use cases will never notice the difference, so, like, this does the job. If, like, AI overviews works now. Um, and, uh, I- I- I think this is gonna, uh, uh, like, similarly be the case for some, for- for each. Like, there's gonna be a model that can prove Riemann. You can make it 100 times smarter. Y- you have your proof, right? So- so for each thing, there's gonna be improvements after a certain amount of compute and, um, I think I underestimated that number, uh, and, uh, I underestimated how much others would focus on code. So- so anyway, just to clear up our, I think, our 101, um, you- you were right.

   7. EGElad Gil17:33

      So,

6. 17:34 – 20:03

   Achieving Magic’s final UX

   1. EGElad Gil17:34

      I guess there's the model side of it and then there's sort of the productization of that model or the ways people access or interact with that. Is there anything you can share at this point regarding those types of things?

   2. ESEric Steinberger17:43

      As we have built the UX, uh, each iteration of the U-... Like, in next UX internally, um, we thought about launching it. We were at this interesting stage of it feels like an uncanny valley, um, where, like, well, you know, clearly you can see signs of life. For the first time, I mean, completion is a trivial one, right? Like- like, we decided not to launch completions 'cause it's just obviously gonna get killed by the next thing and then we're like, okay, it- it's gonna take us a few months to get a prototype of the next thing. We got a prototype of the next thing and then... Like, you're looking at this and it's like-... you can see signs of life, but, you know, like, you guys tried it, uh, when you decided to invest. So- so it's, you know, it's like, would you be using this to, like, write your ... no, no, not yet. Uh, but yeah, we can train the next model and then, like, that model can do it. But then that model, like, can also do all these other things that, like, would go into final shape. So you just enter this, like, stupid recursive loop, uh, until the point we got- we got to this- we get to the point where we're like, "Okay, like, what's the final UX? Let's just, like, let's just make sure this never happens again." And so- so, um, we- we're trying to, like, meet the bar of the- that UX now, um, which I hope we will, um, sooner rather than later. The closer you get to it, the sort of dumber it feels to launch this thing before it because you're gonna replace it in a few months. So if it's good enough for that, like, you know, how hard can it be to add these last little few things? So I do think there is a difference between ... like, you can launch an extremely capable assistant before you launch full automation. That's fine. But launching a sort of mediocrely capable assistant, like we- we might do it, we have a deadline eternally by which if we don't have the, like, ab- mor- ab- nor- you know, nobody has this right now. Like, there is no amazing, can do everything and just, like, feels like a true genius colleague on your team, um, but- and if we don't hit it by that time, we'll launch the other thing, but, um, I would prefer hitting it. Uh, it's just the honest ... you know, the reality is, um, things are hard, uh, things are ... some things, some projects are, uh, going great and some are delayed and some are just th- you know, uh, there are 100 fires all the time. This is just how every hard engineering project goes. Everyone who's listening to this and has ever worked on an engineering projec- project was like, "This is just how it goes." Uh, I think we- we, you know, there are a lot of things we learned along the way, um, and, um, there's nothing we're stuck on. Uh, it's just things are, you know, "God, there's this thing we didn't think about. Okay, let's fix it." And so- so I feel very optimistic.

7. 20:03 – 22:09

   What makes a good AI assistant

   1. ESEric Steinberger20:03

   2. EGElad Gil20:03

      What makes a assistant, like, more mediocre versus amazing? Is it-

   3. ESEric Steinberger20:08

      Trust.

   4. EGElad Gil20:09

      ... reliability? Okay. Yeah.

   5. ESEric Steinberger20:11

      Like, they just trust it. My engineers, when I have one of them write a- write a c- piece of code and another one review it, like, looking at it, they see why- why would I look at it? (laughs) Like, it went through these two guys, you know? So- so, like, what's the point? Uh, and-

   6. EGElad Gil20:25

      So is the eval bar like, "I'm not gonna do code review?"

   7. ESEric Steinberger20:28

      For the ... so that's like the fully automation thing, right? And then you can launch something where you're like, "I'm gonna do code review, but it's not frustrating," uh, if you have to do code review and it's, like, really taxing and you have to fix half of the problems, like half of the PRs or whatever. And so I think the bar for this product is just high. It's not that, like, we are, like, so ambitious and, you know, like, that too, but I- I just genuinely think that there is a gigantic market that gets unlocked in a step function moment where users decide that they're no longer go- they're no longer, uh, going to use, uh, VS Code to write code and send it to their colleagues. They're gonna use Magic or whoever ends up doing this well, uh, first, uh, to- to write their code for them and then briefly look at it and correct every now and then what- what has been done. And then eventually not, right? But that is a step function moment, I think. It's not ... like, you're not gonna use this for, like, 5% of your tasks. You're gonna use this for 90% of your tasks or zero.

   8. EGElad Gil21:26

      You don't believe that this is something you can cut by use case, right? It will be trustworthy on some set of things.

   9. ESEric Steinberger21:32

      No, I think if you can, the leap to doing all use cases is small.

   10. EGElad Gil21:37

       Mm-hmm.

   11. ESEric Steinberger21:37

       Like, you can build a UI builder and then it's like a normal UI builder, or you can build a true great UI builder driven by AI with some added features for that vertical. But then you can do the same thing for all the other verticals, just add the features, you know? And so maybe your product team needs to do one by one. That's feasible. Like, that- that's totally imaginable. And maybe your go-to market needs to be one by one. But the model, I don't think so.

   12. SGSarah Guo21:59

       I guess, um, to your point on having a very high trust team, how did you think about the team that you assemble?

   13. ESEric Steinberger22:05

       So now this is really easy, um, 'cause we've raised an unbelievable

8. 22:09 – 27:10

   Hiring at Magic

   1. ESEric Steinberger22:09

      amount of money from great people and we've got things to show, uh, that even risk averse people who don't think from first principles can understand that this makes sense, um, or who need that initial, like, seed of trust. But I did find it very hard to recruit when we got started, to be honest, because you know when- when Dario Amodei goes out and starts a company, this is trivial, (laughs) like, "Hi, I made GPT-3." Um, and, uh, when, like, you know, there are others like this, um, it's easy, um, to- to- to establish that, um, that trust in- in the outcome. Um, so the strategy we adopted at the start was to hire people who might be, uh ... we have one guy who was just, like, depressed at Amazon, um, working on a, like, Alexa AI, and he just hated it and he- he was, like, so great. Like, he just knew everything, every single paper. I- I was-

   2. EGElad Gil23:06

      (laughs)

   3. ESEric Steinberger23:06

      He was like ... he is RAG. Um, like, I would- I would go like, "Yeah, like, you know, let's talk about this," and he would just know everything. "And let's talk about this," and he would know everything. "And let's talk about this," and he would know everything. Um, and you know, like, he- he studied engineering in college and then just got into mi- ... like, he's- he- he's one of the most brilliant people, um, I get to work with. And, uh, and so we hired him and, like, he did a bunch of stuff. Like, he invented a new sharding dimension (laughs) like for model training. And this is just like a random, you know, you- you have to, like, do ... you have to really pay attention to identify these people. Uh, but then there is an amount of drive and loyalty that you just don't get if you poach, like, the obvious guy, right? Um, and so that's the type of person we have. Uh, and- and we- we have a decent number of them now. Uh, we've gotten really good at identifying them. I would like to have, like, roughly four times as many if we could, but, uh, uh, that said, like, again, I think with the series of announcements that's, you know, going out in the- in the batch that this podcast is going live, that again, will get, you know, uh, easier and better. But, um, I love our culture. It's just, everyone cares about the mission deeply, you know? What I- what I said earlier about why we do what we do, that- that, you know, safely bounded AGI recursion, like, it just takes a lot of-... brain power and understanding of the world to comprehend that this is the right thing to do, uh, and, and or a lot of trust to trust an organization with doing that in the first place. And, um, everyone cares deeply about this but, you know, we, we don't, we don't do it for like the, you know, here is like your marketing sign or whatever, you know. We, we don't have the ... that, that's ... it's just not who we are. And at the same time everyone is deeply productive. They all, you know, when there's, uh, eh, one, one of our, um, primary, like one of our core engineers who writes like the inference engine or like is one of the two people writing the inference engine and, and sub- stuff like kernels and, um, when he joined I was like, "Why do you wanna join? What do- what do you wanna do?" And, um, this was before we raised this giant stack of cash that's getting announced now. It's, uh, it was like the, the tiny amount still, uh, compared to other labs, uh, or I guess compared to any lab (laughs) by a large margin. Uh, and, and he was, he was just like, well, you know, he saw this as an opportunity to, uh, he wanted to be a, one of the best or he said he wanted to try to be the best kernel engineer outside of NVIDIA. And he didn't say this in an arrogant way. He just said like, "This is a ton of work. I've done this for the last few years and I just need to be in an environment where I'm sufficiently challenged to do this." And he's been grinding every day and, and just having like that level of ambition, but not with, like, the typical San Francisco, uh, you know, but, but, but it's, it's, it's, it's like quiet, uh, with humility, drive. You just come into the office every day. You're not like working until 3:00 AM because you prove that you look like this is our culture were to grow. You do it sometimes because you're just so obsessed and you try to be healthy. You do work all the time because you just care so much. But we're not buying the IP by poaching someone from like a lab who tells us how they train GPT at ... Never done this, will not do it. Um, we just do our thing. We have our plan and, you know, we have brilliant people who I'm delighted have trusted us and, uh, uh, spent their energy and their best years on our company.

   4. SGSarah Guo26:14

      What does AGI look like?

   5. ESEric Steinberger26:15

      I think you just talk to it and it does everything. And it asks you questions. That's important.

   6. SGSarah Guo26:21

      And do you think the existence of that ... I guess one could argue it can increase GDP, but it may also, um, decrease, uh, a lot of sort of human driven activities. Um, how do you think about the eventual implications or impact of AGI?

   7. ESEric Steinberger26:37

      It's gonna take longer. Um, I'm gonna try to compress it but it's actually quite complicated. One of the biggest problems with this question is that everyone tries to simplify it, um, by picking one side of the argument. For example, and this is just one example. Um, centralizing power is terrible, so therefore you should open source everything. If I stop now, this is reasonable, right? At least don't make a quote out of this 'cause I don't believe this. Um, and then you could say, well, I should not give these ... I should not ... I should learn how to do PR. I should not say these sentences. Anyway, you can

9. 27:10 – 32:44

   Impact of AGI

   1. ESEric Steinberger27:10

      keep this in. So there's one way to say this, and then the other way to say this is, well, this is like nukes. We all fear this existential risk thing. Like, maybe we care less about the, you know, we, we ... or we care. It's not that we care less, it's just that we think these intermediate problems are completely solvable. Uh, but, but you can't open source how to build a nuke. This is just terrible. So, so the problem is that both of these things are true, and the problem is that there are 10 questions like this and both of the answers are true in all of them. Um, and so what you get is people arguing on X claiming one side and ignoring the other. Um, and so, so I think there are like 10,000 possible futures and which one we end up with depends entirely on which answer we choose or whether we find the sensible middle ground and we, we manage to have a rational debate. Um, the reality is I really truly believe that capitalism and competition are the only chance we have to provide a, an optimizer that is capable of getting us to the right place. I think we need the right guard rails to do that. Not stupid guard rails. I'm not saying any guard rails. I'm saying the right guard rails. Um, and then by the end of it, all work on a computer at least and probably like robot factory stuff, I know less about that, I haven't run the cost structure, uh, but probably that too. I just don't know the cost structure, uh, will be automated. Uh, and humans will do other things. Um, I don't think they'll do, um ... I, I don't think we'll be required to do work for financial gain but we will probably be able to. Uh, property will be a thing. I think if you own apartments, uh, like that won't go away. Uh, if, i- i- there- there's Etsy. Etsy is a, uh ... Etsy is a great proof of concept for what happens after AGI. It's just completely useless. Uh, like you could just buy a made in China product. It looks the same, but it's not made by the human, you know? So that's a thing, I think, like that will grow really big. Whoever owns Etsy, I don't know, but you will get rich, uh, if I'm right.

   2. SGSarah Guo29:03

      Josh, are you listening? (laughs)

   3. ESEric Steinberger29:06

      Um, so, so just do your thing and, and hold through if you don't get automated. Maybe you do actually but your company doesn't. Um, the, um, the ... So, so that might be one way this could go. I think games will be huge. Um, people who are competitive, like all three of us I would guess, um, will be deeply frustrated by the fact that they can no longer fulfill their desire for competitive interaction through work. I, look, I care much more about the positive outcome of what I do than I care about winning personally, but this is a hell of a lot of fun. Like, I love what I do every day. I get to build AGI. I mean, holy shit. And like, I, I mean, isn't it ... it's great to compete against other competent people. Like this is ... if it, if it wasn't in such a serious environment, you know, I would just be enjoying it. Now I have to be conditionally enjoying it but, um, enjoying it and it's gonna go away. So, so weirdly I think like we're the ones who are like harmed the most, uh, on like a meaning level, um, 'cause, 'cause like we won't be able to contribute to society as much, uh, 'cause, 'cause the work I feel like at least I can speak for myself, like I'm doing, I feel tremendously full- is fulfilling and meaningful and that's gonna be deleted. Uh, as much as it sucks to say and hear this, I just ... it is gonna be deleted. Uh, and then, and then my ability to be competitive is gonna be deleted 'cause, you know, DeepBlue beats everyone at chess and like hopefully Magic or some AI system will beat everyone at coding and then like what am I ... you know, and then like there'll be some CEO system and, and then like the, the responsible decision will be to, you know, (laughs) like have that thing be the CEO. And then, so, so that will happen at some point. You can debate how long it takes but that doesn't really matter for the argument to be honest.

   4. SGSarah Guo30:35

      It definitely feels like 10 to 20% of society will be-... deeply frustrated in a post-AGI world and there may be, you know, 70% that's indifferent or happy, and then maybe another, uh, 50%, whatever it is, and then the rest will, will be very excited and thrive and...

   5. EGElad Gil30:54

      There's a, there's a book that is, like, worth reading, skimming, from Ryan Yvent all the way back in 2016 called The Wealth of Humans, and it explores the question... Like, it's not, like, really focused on AGI but, uh, it explores the question of, in a time of abundance, where does your identity come from and how do we keep people, like, happy and productive in that society? I think it's really interesting because it goes beyond, like, a surface level question of just like, "Oh, like, if we can make UBI work, like, are we all okay?" And the answer is no. Right? Like, I wasn't here for UBI to begin with, or any sort of income.

   6. SGSarah Guo31:22

      Well, you could actually argue that abundance has created more issues in society than one would expect if you just look at fragility and other issues and, you know, what people consider actual problems in the world versus real problems and, you know. One could argue that's an outgrowth of abundance and relative peace that we've seen for 30 years. And so, it's an interesting question to ask. In the limit, what does that look like?

   7. EGElad Gil31:44

      S- still on this abundance train, but yes, it's an interesting question. Um, I have one more for you, uh, Erik. Um, so you mentioned the Riemann hypothesis a few times. Like, what is the thing that you really wanna try in terms of new knowledge that you hope Magic will be a- able to answer? Right? 'Cause you could, you could say Riemann, you could say Navier-Stokes, you could say P versus MP. There's a bunch of, like, interesting problems in math or maybe climate or whatever else.

   8. ESEric Steinberger32:08

      Yeah.

   9. EGElad Gil32:08

      But if you're truly ambitious, like, there's gotta be a question.

   10. ESEric Steinberger32:11

       Look, my honest reply is that I think all of these questions are gonna get answered and my North Star, at least personally, and I think most, this is true for most of the company, um, at least, is that I just want the world to be in a good place in 30 years. And after all this is done and dusted and this is the past and we talk about it the way we talk about mobile phones, um, I just want the world to be in a good place. Uh, this is the largest transition we have ever faced and, um, work will get automated and that's crazy. Uh, we'll

10. 32:44 – 36:09

    Eric’s north star for Magic

    1. ESEric Steinberger32:44

       have to find new ways of finding meaning, uh, and that's crazy. Uh, uh, uh, uh, the economy will be... like, I don't even know. Um, governments are gonna have to figure that out. (laughs) But I'm curious about a number of things, but it's just not the North Star. Um, it's a nice side effect that I think is just... I mean, in a way it is the North Star, right? Like what are we building? We're building this, like, automation engine that can answer all our questions. But in a way I think this is just gonna happen, so you don't need to try. The thing you need to try is to make it go well. And if you make it happen and go well, y- all these questions are just gonna get answered. Like, this is a side effect. It's, uh... If someone's gonna ask, like, "Oh, what's Riemann?" I, I'm gonna look it up, I'm gonna try to understand the proof, I'm gonna fail 'cause I'm not smart enough, but I'm gonna try. Like, this will be interesting, I'm gonna spend, like, a few weeks on it and, and y- you know, it's gonna be fun. But, but that's just not, that's just not my North Star at least. Um, I just want everything to be fine in 30 years. If it is, the world will be amazing. And like... Because all the ways in which it could not be amazing are, like, terrible.

    2. EGElad Gil33:44

       (laughs)

    3. ESEric Steinberger33:44

       So, so if we simply keep it not terrible, I think it will be amazing. Um, (laughs) because I can't come up with, like, a mediocre AGI future. It doesn't exist. Like, I don't really... Like, get all the stuff we need and we find a great way to, like, live together and not use it as a weapon, uh, and, and, um, and then, and then, like, everyone has all the things, n- nobody is starving and, like, we all have, like, infinite computer stuff and, like, like, a- and it's, it's... we find new meaning and... Like, all this is... Like, and we're not dead. Like, that's a good start. Uh, like, the, the, the, um... That world is amazing and so all the failure modes are terrible. So, so really... I'm sorry. It's just, like, the only thing I can think about is, like, the, the bimodal nature of this distribution that we're rushing into. And really what's happening is that this, like, smooth distribution of... like, this cloud of uncertainty is, like, slowly collapsing in, in, like, this bimodal thing, and, and everyone is sort of gonna progressively understand this more. And, uh, and, and we just ha- as, as sitting in a chair that I'm sitting in, I just don't feel like I have the right to think about anything else. Like, like, this is, th- it's, uh, this is just a, the responsibility and, like, you know, people will look back in history and, like, all these questions are answered and that's amazing but, but, like, great that, like, this stuff did go wrong, you know?

    4. EGElad Gil34:57

       Distribution skews right.

    5. ESEric Steinberger34:58

       I'm sorry this is not the answer but...

    6. EGElad Gil35:00

       No, no, no. It's okay. The distribution skews right. It's gonna be, it's gonna be good. Like, I, I think an interesting product, like, y- user experience question is in, you know, an era of coworkers rather than, let's say, completion, co-pilot type products, like, how do you think about Magic interacting... Magic is, uh, you know, first and foremost a model company, but how do you think about it interacting with all of the other tools and interfaces that developers use today, like the IDE and whatever e-... Like, does it matter?

    7. ESEric Steinberger35:29

       Great question. Very good question. I changed my mind on this, like, four times. Um, again I was just like we... Like, nobody has a cl- e- everyone's, like, trying things so it really just matters what the market wants in, in terms of product. Uh, we'll, we'll just do whatever the market wants. Our current state of belief, uh, is that you want the system to behave like an employee so it uses the tool set that you give to your staff members, uh, in an interface that is either the same or specifically crafted for AI to be better, uh, than a human could use a tool. For example, Grafana log ingestion. I'm sure we can come up with better ways for AI to use this than humans are using it. So, so maybe there... I, I anticipate that companies will integrate into Magic

11. 36:09 – 37:46

    How Magic will interact in other tools

    1. ESEric Steinberger36:09

       and, and maybe others. Uh, hopefully. I'm g- I'm guessing there'll be competition (laughs) once this is a thing. Um, but, uh, uh, it'll be... Uh, I, I think it'll be, um, such that, uh, systems will be... AI systems will be on a level with the human and tools will be below it. Uh, that, there, that we will not view this as a one-to-one integration. That we'll, we will view it as all these tools, um, are being adapted for AI the way websites had, uh, uh, optimizations made for Google search crawlers, uh, crawling our w- I think there will be, uh, tool optimizations made for AI. And for those that don't have it, the models will just use it natively and, uh, the agent, the model, uh, is the main thing that matters and everything else will get solved for you by other companies. And, uh, it's the same way how, like, all, you know, this army of wrapper companies is gonna get swallowed by, uh, AGI, uh, companies doing their own agent stuff. The, the, the same thing is gonna happen there. Like, if you build your own tools, it's just gonna get swallowed by simply the model learning to adopt everyone el-... This is just... Um, I just, I just like to think in the end point. And I, I... So I think the end point is the model uses things the way a human does, and then maybe also more.

    2. SGSarah Guo37:15

       Oh, well, Erik, thanks so much for the very wide-ranging and interesting conversation. Thanks for joining us on No Priors.

    3. ESEric Steinberger37:20

       Thank you. (instrumental music plays) And, and yeah, I mean, first and foremost, thank you again for supporting Magic and, uh, thank you for giving me the opportunity to speak here.

    4. EGElad Gil37:27

       Great to see you. Find us on Twitter @nopriorspod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.

Episode duration: 37:46