Skip to content
Dwarkesh PodcastDwarkesh Podcast

@Asianometry & Dylan Patel — How the semiconductor industry actually works

Dylan Patel runs Semianalysis, the leading publication and research firm on AI hardware: https://www.semianalysis.com/. Jon Y runs @Asianometry, the world’s best YouTube channel on semiconductors and business history. 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒 * Transcript: https://www.dwarkeshpatel.com/p/dylan-jon * Apple Podcasts: https://podcasts.apple.com/us/podcast/dylan-patel-jon-asianometry-how-the-semiconductor/id1516093381?i=1000671564456 * Spotify: https://open.spotify.com/episode/6q1XODE2L5bqqBwe7434S7?si=seXQ6K_LQZeAV6776H6MhQ * Me on Twitter: https://twitter.com/dwarkesh_sp 𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒 * Jane Street is looking to hire their next generation of leaders. Their deep learning team is looking for FPGA programmers, CUDA programmers, and ML researchers. To learn more about their full time roles, internship, tech podcast, and upcoming Kaggle competition, go here: https://jane-st.co/dwarkesh * Stripe builds financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue. Learn more here: https://stripe.com/ If you’re interested in advertising on the podcast: https://www.dwarkeshpatel.com/p/advertise 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 00:00:00 – Xi’s path to AGI 00:05:05 – Liang Mong Song 00:09:10 – How semiconductors get better 00:12:01 – China can centralize compute 00:19:35 – Export controls & sanctions 00:33:36 – Huawei’s intense culture 00:39:36 – Why the semiconductor industry is so stratified 00:41:43 – N2 should not exist 00:46:38 – Taiwan invasion hypothetical 00:50:06 – Mind-boggling complexity of semiconductors 00:59:58 – Chip architecture design 01:05:21 – Architectures lead to different AI models? China vs. US 01:10:57 – Being head of compute at an AI lab 01:17:09 – Scaling costs and power demand 01:37:50 – Are we financing an AI bubble? 01:51:05 – Starting Asianometry and SemiAnalysis 02:06:55 – Opportunities in the semiconductor stack

Jon Y (Asianometry)guestDylan PatelguestDwarkesh Patelhost
Oct 2, 20242h 10mWatch on YouTube ↗

EVERY SPOKEN WORD

  1. 0:00 – 5:05

    Xi’s path to AGI

    1. J(

      Leong Mong-sung is a nut. He's like, "We will make Samsung into this monster." He does not care about people, he does not care about business. He wants to take it to the limit, the only thing.

    2. DP

      There's no fucking way you can pay for the scale of clusters that are being planned to be built next year for OpenAI unless they raise, like, $50 to $100 billion.

    3. J(

      (laughs) Hold on, hold on. We've already lost John.

    4. DP

      (laughs)

    5. J(

      We've already accepted GPT-5 will be good.

    6. DP

      But yeah, y- you got it, you know? (laughs)

    7. DP

      You got it. You got it. It's like, bro, like, life is so much more fun when you just, like, are delusionally, like, you know?

    8. J(

      We're just ripping bong hits, are we?

    9. DP

      We're not even close to the dot-com bubble. Why wouldn't this one be bigger? We're gonna rip, baby.

    10. DP

      (laughs)

    11. J(

      You could-

    12. DP

      Rip that bong, baby. (laughs)

    13. J(

      You could raise AI for another two decades.

    14. DP

      If you are Xi Jinping and scale pilled, you must now centralize the compute resources, right? They could have a bigger model than any of the labs next year.

    15. DP

      Today, I'm chatting with Dylan Patel, who runs SemiAnalysis, and John, who runs the Asianometry YouTube channel.

    16. DP

      Does he have a last name?

    17. J(

      No, I do not.

    18. DP

      (laughs)

    19. J(

      No, I'm just kidding. John Y.

    20. DP

      That's right, is it?

    21. J(

      John Y.

    22. DP

      Wait, wh- wh- why is it only one letter?

    23. J(

      Because Y is the best letter.

    24. DP

      (laughs)

    25. DP

      (laughs) Why is your face covered?

    26. DP

      (laughs)

    27. J(

      Why not?

    28. DP

      (laughs) No, seriously, why is it covered? (laughs)

    29. J(

      Because I'm afraid of looking at myself get older and fatter over the years.

    30. DP

      (laughs)

  2. 5:05 – 9:10

    Liang Mong Song

    1. J(

      And I think, like, back in the 2000s prior to TSMC, before SMIC got big, it was actually much more kinda open, more flat. I think after that, there was like, after Leong Mong-sung and after all the Samsung issues and after all the, the SMIC's rise, when they're u- literally saw-

    2. DP

      I think, I think you should tell that story, actually. The, the, the TSMC guy that went to Samsung and SMIC and all that. I think you should tell that story.

    3. J(

      There are two stories. There's a guy, he ran a semiconductor company in Taiwan called Worldwide Semiconductor. And this guy, Richard Chang, was very religious. I mean, all the TSMC people are pretty religious. But, like, he in particular was very fervent and he wanted to bring religion to China. So after he sold his company to TSMC, huge coup for TSMC, he worked there for about eight or nine months and he was like, "All right, I'll go to China." Because back then there was... The relations between China and m- uh, Taiwan were much more different. And so he goes over there. Shanghai says, "We'll give you a bunch of money." And then-Richard Chang basically recruits half of, like, a whole bunch, it's like a conga line of, like, Taiwanese that go live-

    4. DP

      (laughs)

    5. J(

      Just, like, they- th- they get on the plane and they fly over. And generally, that's actually a lot of, like, a lot of, like, acceleration points within China's semiconductor industry, it's from talent flowing from Taiwan. And then the second thing was, like, My- Liang Mong-Song. Liang Mong-Song was a, is a nut. And I've met him-

    6. DP

      (laughs)

    7. J(

      I've not met him.

    8. DP

      Oh. (laughs)

    9. J(

      I've met people who work with him, and they say he is a nut. He is a, he is probably on the spectrum, and he's, he does not care about people, he does not care about business, he does not care about anything. He wants to take it to the limit, the only thing. That's the only thing he cares about. He worked from TSMC, literal genius, 300 patents or whatever, 285. Goes, works all the way to, like, the top, top tier, and then one day, uh, he decides, he loses out on some sort of power game within TSMC and gets demoted.

    10. DP

      And he was, like, head of R&D, right, or something? Before the-

    11. J(

      He was, like, one of the top R&D, he was, like, second or third place and he-

    12. DP

      And it was for the head of R&D position, basically.

    13. J(

      Correct. More of the head of R&D position. He's like, "I can't deal with this." And he goes to Samsung, and he steals a whole bunch of talent from TSMC, literally, again, conga line, goes and just emails people and say, "We will pay..." At some point, some of these people were getting paid more than the Samsung chairman, which, uh, not really comparable, but, like, you know what I mean. So they're going-

    14. DP

      Isn't the Samsung chairman usually, like, like, part of the family that owns Samsung?

    15. J(

      Correctamundo.

    16. DP

      Okay, yeah, so it's, like, kinda irrelevant. Yeah, fair enough.

    17. J(

      So it's a bit, but he goes over there, and he's like, "Well, I'm, like, we will make Samsung into this monster. We forget everything. Forget all of the stuff you've been trying to do at, like, incremental."

    18. DP

      Uh-huh.

    19. J(

      "Toss that out. We are going to the leading edge, and that is it." They go to the leading edge. The guys, like-

    20. DP

      They win Apple's business.

    21. J(

      They win Apple's business. They win it back from TSMC. Or did they win it back from TSMC? They win-

    22. DP

      They had a portion of the-

    23. J(

      They had a big portion of it. And then TSMC, Morris Chang is, like, at this time was running the company, and he's like, "I'm not letting this happen." 'Cause that guy, toxic to work for as well, but also goddamn brilliant, and also, like, very good at motivating people. He's like, "We will work literally day or night." Sets up what is called the nightingale army, where you have, they split a bunch of people and they say, "You are working R&D night shift. There is no rest at the TSMC fab. You will go in. There is, as you go in, there'll be a day shift going out." They called it the, it, uh, it's like you're burning your liver. 'Cause in Taiwan, they said, like, if you get old, like, as you work, you, you're roo- sacrificing your liver. They called it the liver buster. So they basically did this nightingale armory for, like, a year, two years. They finished FinFET. They, they basically just blow away Samsung. And at the same time, they sue Liang Mong-Song directly for stealing seek- trade secrets. Samsung s- basically separates from Liang Mong-Song, and Liang Mong-Song goes to SMIC.

    24. DP

      And, and, and so Samsung, like, at one point was better than TSMC.

    25. J(

      Mm-hmm.

    26. DP

      And then, yeah, he goes to SMIC, and SMIC is now better than ... Well, or not better, but they caught up rapidly as well after.

    27. J(

      Very rapid. That guy's a genius. That's a guy who's a genius. I mean, I, I, I don't even, I don't even know what to say about him. He's, like, 78 and he's, like, beyond brilliant, does not care about

  3. 9:10 – 12:01

    How semiconductors get better

    1. J(

      people.

    2. DP

      So-

    3. J(

      Like, what, yeah, what- what is research to make the next process node look like? C- is it just a matter of, like, a hundred researchers go in, they do, like, the next N plus one, then the next morning, the next, uh, a hundred researchers go in? It's experiments.

    4. DP

      Okay, interesting.

    5. J(

      They have a recipe, and they're, what they'll do, every recipe, a TSMC recipe, uh, is the culmination of a long, long years of, like, research, right? It's highly secret. And the idea is that you're, what you're gonna do is that you go, you look at one particular part of it and you say, "Experiment, run an experiment. Is it better? Is it not? Is it better or not?" Kind of a thing like that.

    6. DP

      Yeah, you're basically, it's, it's, it's, it's multivariable problem that each, every single tool, sequentially you're processing the whole thing. You, you turn up knobs up and down on every single tool. You can increase the pressure on this one specific deposition tool, or-

    7. J(

      And, and what are you trying to measure? Is it, like, does it increase the yield? Or, like, what do- what is it that-

    8. DP

      It's not, it's yield, it's performance, it's power. It's not just a one, you know, it's not just better or worse, right? It's a multivariable search space.

    9. J(

      And what do these people know such that they can do this? Is it they understand the chemistry and physics?

    10. DP

      So it's a lot of intuition, but yeah, it's, it's PhDs in chemistry, PhDs in physics, PhDs in, uh, EE.

    11. J(

      Brilliant geniuses people, and they all just-

    12. DP

      And they don't even know about, like, the end chip a lot of times.

    13. J(

      Mm-hmm.

    14. DP

      It's like, "Oh, I am an etch engineer, and all I focus on is how hydrogen fluoride f- etches this," right? "And that's all I know. And, like, if I do it at different pressures, if I do it at different temperatures, if I do it with a slightly different recipe of chemicals, it changes everything."

    15. J(

      I remember, like, uh, someone told me this when I was speaking, like, "How did America lose the ability to do this sort of thing? Like etch in hydrofluoric and acid, all of that." I told them, like, the, he told me basically it was like, it's, it's very apprentice, master-apprentice.

    16. DP

      Mm-hmm.

    17. J(

      Like, you know, in Star Wars Sith, there's only one, right? Master-apprentice, master-apprentice. In, it used to be that there is a master, there's apprentice, and they pass on this secret knowledge. This guy knows nothing but etch, nothing but etch. Over time, the apprentices stop coming. And then in the end, the, the apprentices moved to Taiwan. And that's the same way it's still run. Like, uh, you have the NTU and NTHU, uh, Tsing Hua University, National Tsing Hua University. There's a bunch of masters, they teach apprentices, and they just pass this secret, sacred knowledge down. Mm-hmm. Who are the most AGI-pilled people in the supply chain? Is there anybody that's in, like, the hardware supply chain?

    18. DP

      I g- I gotta, I gotta have my phone call with Colette right now.

    19. J(

      Okay, go for it. (laughs)

    20. DP

      (laughs) Sorry, sorry.

    21. J(

      Can, can, could we mention on the podcast that NVIDIA has got a guy who's calling Dylan for the (laughs) for, to update him on the earnings call?

    22. DP

      Well, it's not this, not exactly that, but ...

    23. J(

      Go for it. Go for it.

    24. DP

      Yeah.

    25. J(

      So, Dylan is back from his call with Jensen Huang. (laughs)

    26. DP

      It was not with Jensen, Jesus.

    27. J(

      What did they tell you, huh? What did they tell you about next year's earnings? (laughs)

    28. DP

      No, it was just color around, like, uh, Hopper, Blackwell-

    29. J(

      Uh-huh.

    30. DP

      ... and, like, margins. It's, like, quite boring stuff-

  4. 12:01 – 19:35

    China can centralize compute

    1. DP

      there.

    2. DP

      All right, all right. We covered the chips themselves. How, how do they get, like, the, the 10 gigawatt, uh, data center up? What else do they need?

    3. DP

      So I, I think there is a true, like, question of how decentralized do you go versus centralized, right? And if you look in the US, right, uh, as far as, like, labs and such, uh, the, you know, OpenAI, xAI, you know, uh, Anthropic, and then Microsoft having their own effort, Anthropic having their own efforts despite having their partner, and then Meta. Um, and, you know, you go down the list, it's like there's a... quite a decentralization of... And then all the startups, like, interesting startups that are out there doing stuff.

    4. DP

      Yep.

    5. DP

      There's quite a decentralization of efforts. Uh, today in China, it is still quite decentralized, right? It's not like Alibaba, Baidu, you are the champions, right? You have, like, DeepSeek. Like, who the hell are you, does government even support you, like, doing amazing stuff, right? If you are S- Xi Jinping and scale pilled-

    6. DP

      Interesting.

    7. DP

      ... you must now centralize the compute resources, right?

    8. DP

      Yeah, yeah.

    9. DP

      Because you have, you have sanctions on how many Nvidia GPUs you can get in now. They're still north of a million a year, right, even post, uh, October last year's sanctions. We still have more than a million H20s, um, and, and other Hopper GPUs getting in through, you know, other means, but legally, like, the H20s. And then on top of that, you have, um, you have your domestic chips, right? But those, that's less than a million chips. So then when you look at it, it's like, Oh, well, we're still talking about a million chips. The scale of data centers people are training on today/over the next six months is 100,000 GPUs, right?

    10. DP

      Yeah.

    11. DP

      Uh, OpenAI, xAI, right? These are, like, quite well-documented, um, and others. But in China, they have no individual system of that scale yet, right? So then the question is like, how do we get there? Um, you know, no, no company has had the centralization push to have a cluster that large and train on it yet, at least publicly, like, well known. And the best models seem to be from a company that has got like 10,000 GPUs, right, or 16,000 GPUs, right?

    12. DP

      Yeah.

    13. DP

      So it's not, it's not quite a, quite as centralized as the US companies are. And the US companies are quite decentralized. If you're Xi Jinping and you're scale pilled, do you just say, "XYZ Company is now in charge-"

    14. DP

      Yeah.

    15. DP

      ... and every GPU goes to one place? And then, then you don't have the same issues as the US, right? In the US we have a big problem with, like, being able to build big enough data centers, being able to build substations and transformers and all this-

    16. DP

      Yeah.

    17. DP

      ... that are large enough in a dense area. China has no issue with that at all because their supply chain adds, like, as much power as, like, half of Europe every year, right? Like, or some th- some absurd statistics, right? Um, so they're building transformer substations, they're building new power plants constantly. Um, so they have no problem with, like, getting power density. And you go look at, like, Bitcoin mining, right? Um, around the Three Gorges Dam, at one point at least, there was like 10 gigawatts of, like, Bitcoin mining estimated, right? Um, which, you know, we're talking about, you know, gigawatt data centers are coming over, you know, '26, '27 in the U- or '26ish in the US, or '27, right?

    18. DP

      Yeah.

    19. DP

      You know, sort of this is an absurd scale relatively, right? We don't have gigawatt data centers, you know, ready, but like, China could just build it in six months, I think, around the Three Gorges Dam or many other places, right? Because they have, they have the ability to do the substations, they have the, they have the power generation capabilities. Everything can be, like, done like a flip of a switch, but they haven't done it yet. And then they can centralize the chips like crazy, right? Now, oh, all million chips that Nvidia is shipping in Q3 and Q4, the H20, um, let's just put them all in this one data center. They just haven't had that centralization effort, right?

    20. J(

      Well, you can argue that, like, the more you centralize it, the more you start building this monstrous thing within the industry, you start getting attention to it, and then suddenly, you know, lo and behold, you have a little bit of a, you have a little worm in there suddenly while you're doing your big training run.

    21. DP

      Ah.

    22. J(

      Oh, this GPU off. Oh, this GPU. Oh, no. Oh, no. Oh, no.

    23. DP

      (laughs)

    24. DP

      (laughs) I don't know if it's like that easy to hack.

    25. DP

      Wait, was that a Chinese accent, by the way?

    26. J(

      (laughs)

    27. DP

      Just to be clear, John is, is East Asian. He's Chinese.

    28. J(

      I am of East Asian descent.

    29. DP

      Mm-hmm.

    30. DP

      Half Taiw- Taiwanese, half Chinese.

  5. 19:35 – 33:36

    Export controls & sanctions

    1. DP

    2. DP

      To the extent that you could just take out a five-gigawatt aluminum refining center and, like, build a giant data center there, then I guess the way to control Chinese AI has to be the chips, because like everything else they, they... So like, uh, how do you like... Just, like, walk me through h- h- how many chips do they have now? How many will they have in the future? What will the r- like how many is that in comparison to US and the rest of the world?

    3. DP

      Yeah, so, so in the world, I mean, the world we live in is they are not restricted at all in like the physical infrastructure side of things in terms of power, data centers, et cetera, because their supply chain is built for that, right?

    4. DP

      Mm-hmm.

    5. DP

      And, and it's pretty easy to pivot that. Whereas the US adds so little power each year, and Europe loses power every year. The, the Western sort of industry for, uh, power is non-existent in comparison, right? But on the flip side is, uh, quote unquote "Western," including Taiwan, manufacture- chip manufacturing is way, way, way, way larger than China's, especially on leading edge where China theoretically has, you know, depending on the way you look at it, either zero or a very small percentage share, right? And so there, um, you have, you have wafer... You have, you have equipment, wafer manufacturing, and then you have, uh, advanced packaging capacity, right? And where the US can control China, right? So advanced packaging capacity is kind of a shot, because the vast majority, the, the largest advanced packaging company in the world was Hong Kong headquartered. They just moved to Singapore, but like, that's effectively like, you know, in a realm where the US can't sanction it, right? Um, a majority of these other companies are in similar places, right? So advanced packaging capacity is very hard, right? Advanced packaging is useful for stacking memory, stacking chips on, uh, COAS, right? Things like that. Then, then the step down is wafer fabrication. Uh, there's immense capability to restrict China there. Uh, and despite the US making some sanctions, China in the most recent quarters was like 48% of ASML's revenue, right? So, you know, and, and, and like 45% of like Applied Materials, and you just go down the list. So it's like, obviously it's not being controlled that effectively. Um, but it could be on the equipment side of things. The chip side of things is actually being controlled quite effectively, I think, right? Like yes, there is like shipping GPUs through Singapore and Malaysia and, and other countries in Asia to China, but you know, the amount you can smuggle is quite small, and then the sanctions have limited the chip performance to a point where it's like, you know, this is actually kind of fair. Uh, but there is a problem with how everything is restricted, right? Um, 'cause you want to be able to restrict China from building their own domestic chip manufacturing industry that is better than what we ship them. Um, you want to prevent them from having chips that are better than what we have.

    6. DP

      Yeah.

    7. DP

      And then... Or, and then you want to prevent them from having AIs better. The ultimate goal being-

    8. DP

      Yeah.

    9. DP

      ... you know, and if you read the restrictions, like very clear, it's about AI.

    10. DP

      Yeah, yeah.

    11. DP

      Um, even in 2022, which is amazing, like at least the Commerce Department was kind of AI-pilled. It was like, is, is you want to restrict them from having AIs worse than us, right? So starting on the right end, it's like, okay, well, if you want to restrict them from having better AIs than us, you have to restrict chips. Okay. If you want to restrict them from having chips, uh, you have to let them have at least some level of chip that the West also that is good, better than what they can build internally. Um, but currently, the, the restrictions are flipped the other way, right? They can build better chips in China, then, uh, we restrict them in terms of chips that NVIDIA or AMD or an Intel can sell to China. Um, and so there's sort of a problem there in terms of the equipment that is shipped can be used to build chips that are better than what the Western companies can actually ship them.

    12. DP

      Uh, John, Dylan seems to think the export controls are k- kind of a failure. Do you, do you, do you agree with him or...?

    13. J(

      That is a very interesting question, because I think it's like-

    14. DP

      Why, thank you (laughs) .

    15. J(

      Like, what do you...

    16. DP

      Darkish, you're so good.

    17. J(

      Yeah, Darkish, you're the best. I think it's, I think failure is a tough word to say, because I think it's like, what are we trying to achieve, right? Like, and say they're talking about AI, right?

    18. DP

      Yeah.

    19. J(

      When you do sanctions like that, it's you need, like, such deep knowledge of the technologies.

    20. DP

      You know, just taking lithography, right? If your goal is to restrict China from building chips and you just like boil it down to like, "Hey, lithography is 30% of making a chip, so are 25%. Cool, let's, let's sanction lithography." Okay, where do we draw the line? Okay, let me ask, let me ask? Let me figure out what, where the line is, and if I'm a bureaucrat or if I'm a lawyer at the Commerce Department or what have you, well, obviously I'm gonna go talk to ASML.

    21. DP

      Yeah.

    22. DP

      And ASML is gonna tell me, "This is the line." Because they know like, hey, well, th- you know, this, this, this is-

    23. DP

      Yeah.

    24. DP

      You know, there's like some blending over.

    25. J(

      There's like... They're, they're like looking at like, "What's going to cost us the most money," right?

    26. DP

      And then they all constantly say like, "If you restrict us, then China will have their own industry," right? And, and the way I like to look at it is like...... chip manufacturing is like, like 3D chess or like, you know, a, a massive jigsaw puzzle in that if you take away one piece, China can be like, "Oh, yeah, that's the piece. Let's put it in," right? And currently this, i- export, say, uh, restrictions, year by year by year they keep updating them, ever since like 2018 or so, '19, right, when Trump started, and now Biden's, you know, accelerated them. They've been like... They haven't just like take a bat to the table and like brick it, right? Like, it's like, let's take one jigsaw puzzle out, walk away. Oh, shit. Let's take two more out. Oh, shit. Right? Like, you know, it's like instead if they like... They, you either have to go kind of like full bat to the fricking like table/wall or, or chill out, right? Like, and like, you know, let them, let them do whatever they want.

    27. DP

      Yeah.

    28. DP

      'Cause the alternative is everything is focused on this thing and they make that. And then now when you take out another two pieces, it's like, "Well, I have my domestic industry for this. I can also now make a domestic industry for these." Like, you go deeper into the tech tree or what have you.

    29. DP

      Yeah.

    30. DP

      It's a very... It's art, right? In the sense that there are technologies out there that can compensate. Like, if you believe... The belief that lithography is a linchpin within the system is, it's not exactly true, right? At some point, if you keep pulling, keep pulling a thread, other things will start developing to kind of close that loop. And like, I think it's a, it's, uh, it is... That's why I say it's an art, right? I don't think you can stop Chinese semiconductor industry- the semiconductor industry from progressing. I think that's basically impossible. So the question is, the Chinese government believes in the primacy of semiconductor manufacturing. They u- they've believed it for a long time, but now they really believe it, right? To, to some extent, these sanctions have made China believe in the importance of the semiconductor industry-

  6. 33:36 – 39:36

    Huawei’s intense culture

    1. DP

      five nanometer here," right?

    2. DP

      And are they doing it because they believe in AI or because they wanna make Huawei phones?

    3. DP

      Uh, you know, Huawei was the largest TSMC customer for, like, a few quarters actually-

    4. DP

      Mm-hmm. Yeah.

    5. DP

      ... before they got sanctioned. Uh, Huawei makes most of the telecom equipment in the world, right? Uh, you know, phones, of course, modems, but of course accelerators, networking equipment. You know, you go down the whole, like, video surveillance chips, right? Like, you kinda, like, go through the whole gambit.

    6. DP

      Yeah.

    7. DP

      A lot of that could use seven and five nanometer.

    8. DP

      So-

    9. J(

      Do you think the dominance of Huawei is actually a bad thing for the rest of the Chinese tech industry?

    10. DP

      I think Huawei is so fucking cracked that, like, it's, it's hard to say that, right? Like, Huawei outcompetes Western firms regularly with two hands tied behind their back. Like, you know, like, what, what the hell is Nokia and, like, uh, Sony Ericsson? Like, trash, right? Like, compared to Huawei. And Huawei is not allowed to ship, sell to, like, European companies or American companies, and they don't have TSMC, and yet they still destroy them, right?

    11. DP

      Mm-hmm.

    12. DP

      Um, and, and same applies to, like, the new phone, right? It's like, oh, it's, like, as good as, like, a year old Qualcomm phone on a process node that's equivalent to, like, four years old, right, or three years old. So it's like, wait, so they actually out-engineered us with a worse process node, you know? So it's like, oh, wow. Okay. Like, you know, Huawei is, Huawei is, like, crazy cracked.

    13. J(

      Where do you think that culture comes from?

    14. DP

      The military 'cause it's the PLA.

    15. J(

      It is the, we, we... It is generally seen as an arm of the PLA. But, like, how do you square that with the fact that sometimes the PLA seems to mess stuff up?

    16. DP

      Oh, like filling water in rockets?

    17. DP

      Yeah. (laughs)

    18. J(

      I don't know if that was true. I'm denying, I'm not denying-

    19. DP

      So there, so there is, there is, like, that, like, like, like, crazy conspir- not conspiracy. It was like, you, you don't know what the hell to believe in China, especially as a not-Chinese person, but, like-

    20. J(

      Like, nobody knows... Even Chinese people don't know what's going on in China.

    21. DP

      There's, like, you know, like, all sorts of stuff. Like, "Oh, they're filling water in their rockets. Clearly they're, like, incompetent." It's like, look, if I'm the Chinese military, I want the Western world to, like, believe I'm completely incompetent because one day I can just, like, destroy the fuck out of everything, right? With all these hypersonic missiles and all this shit, right? Like drones and... Like, no, no, no, no. Uh, we're filling water in our missiles. These are all fake. We don't actually have 100,000 missiles that we manufacture in a facility that's, like, super hyper advanced and Raytheon is stupid as shit because they can't make, you know, missiles nearly as fast, right? Like, I think, like, that's also, like, a flip side is, like, how much false propaganda is there, right? 'Cause there's a lot of, like, "No, SMIC could never. SMIC could never. They had... They don't have the best tools," blah, blah, blah. And then it's like, motherfucker, they just shipped 60 million phones last year with this chip that performs only one year worse than, like, what Qualcomm has. It's like proof is in the pudding, right? Like, you know, there's, there's a lot of, like, cope, if you will.

    22. J(

      I just wonder where it comes from. I do really do just wonder where that culture comes from. Like, there's something crazy about them, where they're kinda like everything they touch they seem to succeed in. And, like, I, I kinda wonder why.

    23. DP

      They're making cars. I wonder if it's going on there. (laughs)

    24. J(

      I think, like, if, like, supposedly, like, if we kind of imagine, like, historically, like, do you think they're getting something from somewhere?

    25. DP

      What do you mean?

    26. DP

      Espionage, you mean?

    27. J(

      Yeah, like-

    28. DP

      Well, obviously.

    29. J(

      Like, like, East Germany and the Soviet industry was basically, was just a, it was like a conveyor belt of, like, secrets coming in. And they're just, used that to run everything. But the Soviets were never good at it.

    30. DP

      Yeah.

  7. 39:36 – 41:43

    Why the semiconductor industry is so stratified

    1. DP

      uh-

    2. DP

      Why isn't there more vertical integration in the semiconductor industry? Well, like, why are there, like, this subcomponent requires this other subcomponent from this other company, which requires a subcomponent from another company? Like, why, why is more of it not done in-house?

    3. DP

      The way to look at it today is it's super, super stratified. And every industry has anywhere from one to three competitors. And pretty much the most competitive it gets is like 70% share, 25% share, 5% share.

    4. DP

      Mm.

    5. DP

      Um, in any layer of, like, manufacturing chips, anything, anything, chemicals, different types of chips. But it used to, used to be vertically integrated, like, and-

    6. J(

      Well, in the very beginning it was integrated, right? And-

    7. DP

      Wh- why did that stop?

    8. J(

      What happened was, you know, the funniest thing was that like, you know, you had companies that used to do it all in the one, and then suddenly sometimes a guy would be like, "I hate this. I think I know, I know th- how to do better." Spins off, does his own thing, starts his company. Goes back to his old company, says, "I can sell you a product-"

    9. DP

      Mm.

    10. J(

      "... that's better." Right? And that's the beginning of what we call the semiconductor manufacturing in- equipment industry. Like basically-

    11. DP

      Like in the '70s, right?

    12. J(

      ... it was spinoff.

    13. DP

      Like everyone made their own equipment in the '60s.

    14. J(

      '60s and '70s. Like they spin off all these people. And then what happened was that the companies that accepted, you know, these outside products and equipment got better stuff. They did better. Like, you can talk about a whole bunch ... Like, there are companies that were totally vertically integrated in semiconductor manufacturing for decades. And they are n- they're still good, but they're nowhere near competitive.

    15. DP

      One thing I'm confused about is, like, the actual foundries themselves. There's, like, fewer and fewer of them-

    16. J(

      Yeah.

    17. DP

      ... every year, right? So there's, like, more, maybe more companies overall, but like the, like the final people, like, who, uh, make the m- m- make the wafers, there's less and less. Uh, and then i- I- it's interesting in a way similar to like the AI foundation models where, um, you need to use like the revenues from like a previous model in order ... Or like the, your, um, market share to, like, fund the next round of ever more expensive development.

    18. J(

      When TSMC launched the foundry industry, right, and when they started, there was a whole wave of, like, Asian companies that funded semiconductor foundries-

    19. DP

      Mm-hmm.

    20. J(

      ... of their own. You had Malaysia with Siltera. You have Singapore with Chartered. You had, uh, there was one, there's Worldwide, there's Worldwide Semiconductor where I talked about earlier. There's one from Hong Kong.

    21. DP

      Bunch in Japan.

    22. J(

      Bunch in Japan. Like, they

  8. 41:43 – 46:38

    N2 should not exist

    1. J(

      all sort of did this thing, right? And I think the thing was that when you go into leading edge when the thing is that, like, it got harder and harder, which means that you had to aggregate more demand from all the customers to fund the next node, right? So technically, in the sense that what this kind of do is aggregating all this money-

    2. DP

      Yeah.

    3. J(

      ... all this profit, to kind of fund this next node to the point where now, like, there's no room in the market for an N2 and, or N3. Like, there, technically you could argue that ... economically, you can make an argument that, like, N2 is a monstrosity that doesn't make sense economically and would, should not exist in some ways without the immense single concentrated spend of, like, five players in the market.

    4. DP

      I'm sorry to, like, completely derail you, but, like, there's this video where it's like, uh, there's this unholy concoction of meat slurry.

    5. J(

      Yes.

    6. DP

      (laughs)

    7. DP

      What? (laughs)

    8. J(

      (laughs) Sorry, there's like a video that's like, "Ham is disgusting. It's an unholy concoction of, like-

    9. DP

      (laughs)

    10. J(

      ... meat with no bones or collagen."

    11. DP

      Uh-huh.

    12. J(

      And like, I don't know, like two... He was like, the way he was describing two-nanometer is kind of like that, right?

    13. DP

      It's like the guy who pumps his right arm so much, and he's like mo- super muscular. The human body was not meant to be so muscular.

    14. J(

      (laughs)

    15. DP

      Like- Wh- wh- what, what's the point? Like why, why is two nanometer not justified?

    16. J(

      I'm not saying N2 as like, N2 specifically, but say N2 as a concept. The next node should technically... Like right now, there is a p- there will come a point where economically, the next node will not be possible, like at all, right?

    17. DP

      Unless, unless more, you know, technologies spawn. Like AI now makes, you know-

    18. J(

      Yeah. AI, yeah.

    19. DP

      ... one nanometer or whatever, you know-

    20. J(

      Yeah. There was a long period of time-

    21. DP

      ... where 16A-

    22. J(

      Yeah, yeah.

    23. DP

      ... viable, right? So, so like right before AI-

    24. J(

      Ra- and, and makes it viable in what's, uh, as in like, uh, it makes it-

    25. DP

      Like money. Money.

    26. J(

      ... worth it or like make-

    27. DP

      So every, every two years you get a shrink, right?

    28. J(

      Yeah.

    29. DP

      Uh, like clockwork, Moore's Law. Um, and then five nanometer happened, it took three years. Holy shit. And then three nanometer happened, it took three... Or no, sorry, is it three nanometer or five? Uh, took three years. Holy shit. Like is Moore's Law dead, right? Like 'cause TSMC didn't... And then what did Apple do? Even on the third year of three... of, of, uh, uh... or sorry, when three nanometer finally launched, they still only... Apple only moved half of the iPhone volume to three nanometer. So this is like... Now that they had a fourth year of I- of five nanometer for a big chunk of iPhones, right? And it's like, oh, is the mobile industry petering out? Then you look at two nanometer, and it's like gonna be a similar, like very difficult thing for the, for the industry to pay for this, right? Apple, of course, they have... you know, because they get to make the phone, they have so much profit, they can funnel into the more and more expensive chips. But finally, like that was, that was really running out, right? It was two... H- how economically viable is two nanometer just for one player, TSMC? You know, ignore Intel, ignore Samsung. Just in, you know, because in... Samsung is paying for it with memory, not with their actual profit. And then Intel is paying it from it from their former CPU monopoly, um-

    30. J(

      Private equity money-

  9. 46:38 – 50:06

    Taiwan invasion hypothetical

    1. DP

    2. J(

      (laughs)

    3. DP

      Okay. The, uh, uh, suppose Taiwan is invaded or Taiwan has an earthquake, nothing is shipped out of Taiwan in the... from now on. What happens next? The rest of the world, how would it feel its impact a day in, a week in, a month in, a year in?

    4. J(

      I mean, it's, it's a terrible thing. It's a terrible thing to talk about. I think it's like... Can you just say it's all terrible? Everything's terrible. Because it's not just like leading edge. Leading edge, people were focused on leading edge, but there's a lot of trailing edge stuff that, like, people depend on every day. I mean, we all worry about AI. The reality is you're not gonna get your fridge, you're not gonna get your cars, you're not gonna get everything. It's terrible. And then there's the human part of it, right?

    5. DP

      Yeah.

    6. J(

      It's all terrible. Can we... Like it's, it's depressing.

    7. DP

      I think that-

    8. J(

      And I live there. Yeah.

    9. DP

      I think day one, market crashes a lot, right? You gotta think about like... I think, I think the big, like big six, six mo-... Six biggest companies, magnificent seven, whatever the heck it's called are like 60, 75% of the S&P 500, and their entire business relies on chips, right? Google, Microsoft, Apple, Nvidia. Uh, you know, you go down the list, right? They're, they're all... Meta, right? They all entirely rely on AI. Um, you, and you would have a tech reset, like extremely insane tech reset, by the way, right? Like, so market would crash a week, a day in, a couple weeks in, right? Like, people are preparing now. People are like, "Oh, shit." Like, "Let's start building fabs. Fuck all the environmental stuff." Like war is probably happening.

    10. J(

      Yeah, yeah.

    11. DP

      Um, but, uh, but like the supply chain is trying to like figure out what the hell to do to refix it. But six months in, the supply of chips for making new cars, gone or sequestered to make military shit, right? Um, you can no longer make cars. Um, and we don't even know how to make non-semiconductor in- induced cars, right? Like this unholy concoction with all these like chips, right? Uh, you- you, uh-

    12. J(

      Car's like 40% chips now.

    13. DP

      Yeah. (laughs)

    14. J(

      It's like it's just chips on, in the tires.

    15. DP

      (laughs)

    16. DP

      There's like, there's like 2000 plus chips. Really, like every Tesla door handle has like four chips in it. It's like, what the fuck?

    17. DP

      (laughs)

    18. DP

      Like why? Um, like, like but like it's like, it's like shitty like microcontrollers and stuff-

    19. J(

      Yeah.

    20. DP

      ... but like there's like 2000 plus chips even in an, in, in an ICE vehicle, like internal combustion engine vehicle, right? And every engine has dozens of dozens of chips, right? Um, anyways-This all shuts down because, uh, not all of the production. There's some in Europe, there's some in the US, there's some in Japan, there's some in Singapore-

    21. J(

      Yeah, they're gonna, they're gonna bring in a guy to work in, on Saturday until 4:00 (laughs) .

    22. DP

      Well, yeah, yeah. I mean, yeah (laughs) . So, so you have like, TSMC always builds new fabs. That old fab, they, like, tweak production up a little bit more and more, and th- new designs move to the next, next, next node. And, and old stuff fills in the old nodes, right? So, you know, ever since TSMC has been the most important player, and not just TSMC, there's UMC there, there's PSMC there, there's a number of other companies there. Taiwan's share of, like, total manufacturing has grown every single process node. So in, like, 130 nanometer, there's a lot, and including, like, many chips from, like, Texas Instruments or Analog Devices or, like, NXP, like, all these companies. 100% of it is manufactured in Taiwan, right? By, you know, either PS- TSMC or UMC or whatever. But then you, like, step forward and forward and forward, right, like 28 nanometer. Like, 80% of the world's production of 28 nanometers is in Taiwan. Oh, fuck, right? Like, you know, and everything in 28 nanometers, like, what's made on 28 nanometer today? Tons of microcontrollers and stuff, but also, like, every display driver IC. Like, cool, like even if I can make my Mac chip, I can't make the chip that drives the display. Like, you know, you just go down the list, like, everything, no fridges, no, no automobiles, no, no weed whackers because that shit has... My toothbrush has fucking Bluetooth in it, right? Like, why? I don't know. But like, you know, there's, like, so many things that, like, just like, poof, we're tech

  10. 50:06 – 59:58

    Mind-boggling complexity of semiconductors

    1. DP

      reset.

    2. DP

      We were supposed to do this i- interview, like, many months ago, and then I, like, kept, like, delaying 'cause I'm like, "Ah, I don't understand any of this shit." (laughs) But like, it is like a very difficult thing to understand where I feel like with AI, it's like, I, it's not that, like-

    3. DP

      No, you've just spent time. You've spent-

    4. DP

      Sure, sure.

    5. DP

      ... the time to-

    6. DP

      But like, I also feel like it's, like, less compli- Uh, it, it, it feels like it's the kind of thing where, like, in an amateur kind of way, you can, like, you know, pick up what's going on in the field.

    7. DP

      Yes, es-

    8. DP

      In this field, like, I, I, the thing that cares about is, like how, how does one learn the layers of the stack? Because the layers of the stack are like, there's not just the papers online. You can't just, like, look up the, the tutorial on how the transformer works or whatever. It's like-

    9. DP

      Yes. I mean, like-

    10. DP

      It's like many layers of really difficult shit.

    11. DP

      There are, like, 18 year olds who are just cracked at AI, right?

    12. DP

      Yeah.

    13. DP

      Already, right? And like, there's high school dropouts that get, like, jobs at OpenAI. This existed in the past, right? Pat Gelsinger, current CEO of Intel, went straight to work. He was, he, like, grew up in the Amish area of Pennsylvania, and he went straight to work at Intel, right? Because he's just cracked, right? That is not possible in semiconductors today. You can't even get, like, a job at, like, a tool company without, like, a, at least, like, a freaking master's in chemistry, right? And probably a PhD, right? Like, like, of the, like, 75,000 TSMC workers, it's like 50,000 have a PhD or something insane, right? It's like, okay, this is like, there's like some, there's like a next level amount of like, how specialized everything's gotten. Whereas today, like, you can take like, you know, Sholto, you know, he, he, when did he start working on AI? Not that long ago. Right?

    14. J(

      Not to say anything bad about Sholto.

    15. DP

      (laughs)

    16. DP

      No, no, no, he's c- but he's cracked, he's like omega cracked-

    17. DP

      Yeah, yeah.

    18. DP

      ... at like what he does. What he does, you could pick him up and drop him into another part of the AI lay- stack. First of all, he understands it already. And then second of all, he could probably become cracked at that too, right? Um, whereas that is not the case in semiconductors, right? You ca- you, one, you, like, specialize like crazy. Two, you can't just pick it up. Um, you know, like Sholto, I think, what did he say? He, like, just started, like-

    19. DP

      Uh, he was a consultant in McKinsey, and at, like, night, he would, like, read papers about robotics-

    20. DP

      Right.

    21. DP

      ... and like, run experiments and whatever.

    22. DP

      Yeah, and then, and then, like, he, like, was like, like people noticed, it was like, "Who the hell is this guy?"

    23. DP

      Yeah.

    24. DP

      "And why is he posting this?" Like-

    25. DP

      Yeah.

    26. DP

      ... I thought everyone who knew about this was at Google already, right?

    27. DP

      Yeah, yeah, yeah.

    28. DP

      And it's like, "Come to Google," (laughs) right? That can't happen in semiconductors, right? Like, it's just not, like, conducively, like, it's not possible, right? One, arXiv is, like, a free thing. Um, the paper publishing industry is, like, ou- abhorrent everywhere else. And you just, like, cannot download IEEE papers or, like, SPIE papers or, like, other organizations. And then two, at least up until, like, late 2022 or really early 2023 in the case of Google, right? I think, what, the PaLM inference paper, up until the PaLM inference paper, before that, all the good, best stuff was just posted on the internet. After that, you know, it's kind of a little bit clamping down by the labs, but there's also still all these other companies making innovations in the public. That, and, and like, what is state of the art is public. That is not the case in semiconductors.

    29. J(

      Semiconductors have been shut down since 1960s, 1970s, basically. I mean, like, it's kinda crazy how little information has been formally transmitted from one, one country to another. Like, the last time you could really think of this was like 19, maybe the Samsung era, right?

    30. DP

      So then how do you guys keep up with it?

  11. 59:58 – 1:05:21

    Chip architecture design

    1. DP

      I think, um, you know, it's first, it's important to state that semiconductor manufacturing and design is the largest search base of any problem that h- humans do because it is the most complicated, uh-

    2. DP

      Mm-hmm.

    3. DP

      ... industry that anything that humans do. And so, you know, when you think about it, right, there's, there's, uh, 1E10, 1E11, right? Uh, 100 billion transistors-

    4. DP

      Yeah.

    5. DP

      ... uh, on, on leading edge chips, right? Blackwell has, uh, two 20 billion transistors or something like that. So what is... And those are just on-off switches. And then think about every permutation of putting those together, contact ground, et cetera, drain source, blah, blah, blah, with wires, right? There's 15 metal layers, right? Connecting every single transistor in every possible arrangement. This is a search base that is literally almost infinite, right? You could like... The search base is much larger than any other search base that humans know of.

    6. DP

      And the nature of the search, like what are you trying to optimize over?

    7. DP

      Well, useful compute, right? What is, you know, if your- if the, if the goal is optimize intelligence per picajoule, right? Um, and- and intelligence is some nebulous nature of like the, what the model architecture is.

    8. DP

      Yeah, yeah.

    9. DP

      Uh, but, and then, and then picajoule is like a unit of energy, right?

    10. DP

      Yeah.

    11. DP

      How do you optimize that? So there's humongous innovations possible in architecture, right? Because vast majority of the power on a H100 does not go to compute. And there are more efficient, like-... uh, compute our, you know, ALUs, arithmetical unit like designs, right? But even then, the vast majority of the power doesn't go there, right? The vast majority of the power goes to moving data around, right? And then when you look at what is the movement of data? It's either networking or memory. You know, you have, you have a humongous amount of movement relative to compute, and a hu- humongous amount of power consumption relative to compute. And so the, so how can you minimize that data movement, m- and then maximize the compute? There are 100X gains from architecture. Even if we, like, literally stopped shrinking, I think we could have 100X gains from architectural advancements.

    12. DP

      Over what time period?

    13. DP

      Um, the, the, the question is how much can we advance the architecture, right?

    14. DP

      Yeah.

    15. DP

      The, the challenge, the other challenge is like, the number of people designing chips has not necessarily grown in a long time, right? Um, yeah, like, company to company it shifts, but like, within like the semiconductor industry in the US, and the US makes, you know, designs the vast majority of leading edge chips, the number of people designing chips has not grown much.

    16. DP

      Mm-hmm.

    17. DP

      Um, what has happened is the output per individual has soared because of EDA, electronic design assistance tooling, right? Now, this is all still, like, classical tooling. There's just a little bit of inkling of AI in there yet, right? What, what happens when we bring this in, is the question. And, and how you can solve this search base somehow, uh, with humans and AI working together to optimize this so it's not most of the power is mo- data movement, and then the er- logic, the, the compute is actually very small, to flip side, um, the compute is... First of all, compute can get like 100X more efficient just with, like, design changes, and then you could minimize that data movement massively, right? So, you can get a humongous gain in efficiency just from architecture itself. And then process node helps you innovate that there, right? And, uh, power delivery helps you innovate that. Uh, system design, chip-to-chip networking helps you innovate that, right? Like, memory technologies, there's so much innovation there, and there's so many different vectors of innovation that people are pursuing-

    18. DP

      Mm-hmm.

    19. DP

      ... simultaneously, uh, to where, like, Nvidia gen to gen to gen will do more than 2X performance per dollar. Uh, I, I think that's very clear. And then, like, hyperscalers are probably gonna try and shoot above that, but we'll see if they can execute.

    20. DP

      The, there's, like, uh, two narratives you can tell here of how this happens. One is that these AI c- companies who are training the foundation models, who understand the trade-offs of, like, how much is the marginal increase in compute versus memory worth to them, and what trade-offs do they want between different kinds of memory, they understand this. And so therefore, the accelerators they build, uh, they can make these sort of trade-offs in a way that's like most optimal or, and also design, like, the architecture of the, the model itself in a way that, uh, uh, uh, reflects, like, w- what are the hardware trade-offs. Another is NVIDIA, because it has, like... M- I, I, I don't know how this works, but presumably they have some sort of, like, know-how, like, they're accumulating all this, like, uh, knowledge about how to better design this architecture and like also better search tools for so on. Um, who has basically, like, better moat here in terms of... W- will NVIDIA keep getting better at design, getting this 100X improvement? Or will it be like OpenAI and Microsoft and, uh, Amazon and Anthropic who are designing their accelerators, who will keep getting better at, like, designing the accelerator?

    21. DP

      I, I think that there's a few vectors to go here, right? One is, you mentioned, and I think it's important to note, is that hardware has a huge influence on the model architecture that's optimal. And so it's not a one-way street that better chip equals, you know... The m- the optimal model for Google to run on TPUs, given a given amount of-

    22. DP

      Mm-hmm.

    23. DP

      ... dollars, a given amount of, uh, compute, is different architecturally than what it is for OpenAI with NVIDIA g- stuff, right? It is, like, absolutely different. And then, like, even down to, like, networking decisions that different companies do and data center design decisions that people do, the optimal, like, if you were to say, you know, X amount of compute of TPU versus GPU, compute optimally what is the best thing, you will d- diverge in what the architecture is. And I think that's important to know, right?

    24. DP

      Wait,

  12. 1:05:21 – 1:10:57

    Architectures lead to different AI models? China vs. US

    1. DP

      can, can I ask about that real quick? Uh, the, um... So earlier we were talking about how China has the, uh, uh, H20s or, uh, B20s, um-

    2. DP

      Yeah.

    3. DP

      Uh, and there, there's like much less compute per memory bandwidth and, like, the amount of memory, right? Does that mean that Chinese models will actually have like very different architecture and characteristics than American models in the future?

    4. DP

      So, so you can take this to like a very, like, large conclu- like, leap, and it's like all, you know, neuromorphic computing or whatever is, like, the optimal path, and that looks very different than, like, what a transformer does, right? Um, or you could take it to, like, a simple thing, which is like the level of sparsity, eh, th- like f- well, coarse-grained sparsity, i.e. like, experts and all this sort of stuff.

    5. DP

      Yeah.

    6. DP

      Um, the arrangement of, w- like, what exactly the attention mechanism is, because there are a lot of tweaks.

    7. DP

      Yeah.

    8. DP

      Uh, it's not just like pure transformer attention, right? Or like, hey, D mod, like how wide versus tall the model is, right? That's like very important. Like, D mod versus, you know, number of layers, right? Um, these are all, like, things that, like, would be different, like... And I, and like, I know they're different between, like, say-

    9. DP

      Yeah, yeah.

    10. DP

      ... a Google and an OpenAI in what is optimal.

    11. DP

      Yeah.

    12. DP

      But what really s- it really starts to get like, hey, if you are limited on a number of different things, like, uh, like China invests humongously in compute and memory. Um, you know, which is, like, basically the memory cell is directly coupled... Or is the, uh, the compute cell, right? So these are, like, things that, like, China's investing hugely, and you go to conferences like, oh, there's 20 papers from Chinese companies/universities about compute and memory. Or like, you know, hey, like, because the flop limitation is here, maybe NVIDIA pumps up the on-chip memory and, like, changes the architecture, because they s- they still stand to benefit tens of billions of dollars by selling chips to China, right? Today, it's just like neutered American chips, right? Uh, uh, neutered chips that go to the US, but like, it'll start to diverge more and more architecturally, because they'd be stupid not to make chips for China, right? Um-And Huawei, obviously again, like has like their constraints, right? Like where are they limited on memory? Oh, they have a sh- lot of networking capabilities, and they could move to like certain optical, like, networking technologies directly onto the chip much sooner than we could, right? Because that is what's optimal for them within their search base of solutions, right? Because this whole area is like blocked off. Right?

    13. NA

      It's kind of really interesting to see, to think about like the development of how Chinese AI models will differ b- from American AI models because of the, 'cause of these changes or these constraints.

    14. DP

      And it applies to use cases, it applies to data, right? Like American models are very important about like, "Let me learn from you," right? "Let me be able to use you directly as a random consumer," right? That is not the case for a Chinese model, I assume, right? Uh, because there's probably very different use cases for them. Uh, China's crushes the West at video and image recognition, right? Uh, at ICML, like Albert Gu at, you know, of Cartesia, like state-space models, like every single Chinese person was like, "Can I take a selfie with you?" Man was harassed. In the US, like you see Albert, and he's like, it's awesome, he invented state-space models, but it's not like state-space models are like, like here. But that's because state-space models potentially have like a huge advantage in like video and image and audio, which is like stuff that China does more of and that is further along and has better capabilities in, right?

Episode duration: 2:10:53

Install uListen for AI-powered chat & search across the full episode — Get Full Transcript

Transcript of episode pE3KKUKXcTM

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

Add to Chrome