The First Dedicated YC GPU Cluster - With Together AI

1. 0:00 – 0:26

   YC and Together AI Are Partnering on a GPU Cluster

   1. AGAnkit Gupta0:00

      [upbeat music] Hey everyone, I'm here with Vipul, CEO and co-founder of Together AI, and we've got some exciting news. YC and Together are partnering to bring online the first dedicated YC GPU cluster. This gives our portfolio of AI native startups easier access to the compute they need to build and scale. Vipul, thanks for being here.

   2. VPVipul Ved Prakash0:21

      So great to be here, Ankit.

   3. AGAnkit Gupta0:23

      Vipul, why don't you start by telling us a bit about Together AI? What exactly do you guys

2. 0:26 – 1:22

   What Together AI Does

   1. AGAnkit Gupta0:26

      do?

   2. VPVipul Ved Prakash0:26

      Yeah. So, you know, we started the company almost exactly four years ago, and as, as you know, my co-founders are professors and academics, and we were quite convinced at the time that generative AI was going to be a very foundational technology. And I think we also had a sense that because of the capital structure required, we may see large concentration. And, and, and, and we really wanted to create a platform for innovation, for, you know, the sort of a participatory side of Together. So what we've built at Together AI is, uh, really a cloud service that's designed for the whole life cycle of generative AI, which is everything from building models to, uh, post-training open models, to serving them at scale.

   3. AGAnkit Gupta1:09

      Nice. And could you tell us a little bit about, um, how companies actually work with you? You know, YC, we are, um, we acquired the first GPU cluster for YC itself to share to our companies. Tell us a little bit about the types of companies that have worked with Together

3. 1:22 – 1:58

   From Research Labs to Cursor: Together's 8,000 Customers

   1. AGAnkit Gupta1:22

      AI in the past.

   2. VPVipul Ved Prakash1:22

      Yeah. You know, we-- our, our focus is really on AI native startups, and we have over 8,000 customers today, and this range from, you know, small research groups who are sort of still figuring out what they want to do and experimenting, and there's just such an empirical feel that we enable them with, uh, access to compute or access to inference services, all the way to some of the biggest names like Cursor and Cognition and ElevenLabs. And these companies are training models. They are-- And, and we are serving their model at scale for them.

4. 1:58 – 3:24

   The Landscape of AI Native Startups at YC

   1. AGAnkit Gupta1:58

      Yeah, one of the really incredible things from my perspective over the last year has been all the different types of what we would call AI native startups that have been built, right? You have companies that are training their own models from scratch, where they're now able to use this GPU cluster to get access to compute-

   2. VPVipul Ved Prakash2:12

      Right

   3. AGAnkit Gupta2:12

      ... especially in this environment. You have AI native services companies that are often building on top of models, and increasingly building on top of open source models hosted on places like Together. And then you have kind of everything in between, from data providers to application layer companies. So there's this whole landscape of companies in the YC portfolio, and they all have kind of very heterogeneous compute needs.

   4. VPVipul Ved Prakash2:35

      Right.

   5. AGAnkit Gupta2:35

      And so I think for us, having a partner where there's different ways to work with you guys has been a big win for a lot of those companies.

   6. VPVipul Ved Prakash2:41

      Yeah. And, and I, I think that flexibility is, uh, particularly important because, uh, y- y- you're trying to be efficient with your capital, uh, and, and being able to deploy it behind the workloads you want, and you want really rapid sort of iteration. I think we have sort of really helped create a platform that, uh, allows companies to, you know, to be able to do all these things, uh, efficiently. Building a company in a today's AI native world is, is, is quite different-

   7. AGAnkit Gupta3:13

      Yeah

   8. VPVipul Ved Prakash3:13

      ... uh, I imagine from the days you started your company.

   9. AGAnkit Gupta3:16

      Yeah.

   10. VPVipul Ved Prakash3:16

       Uh, w- what are you seeing? What, what do you think are the main challenges, uh, that, uh, you know, YC startups are experiencing?

5. 3:24 – 4:56

   How Building an AI Company Has Changed Since 2018

   1. AGAnkit Gupta3:24

      Yeah. I think there's a bunch of different ones all at once. It's a very exciting time at YC. I think one of the, the really cool things is way more things look like software companies now than ever before.

   2. VPVipul Ved Prakash3:33

      Right.

   3. AGAnkit Gupta3:33

      Like, when I ran my company, we were a AI native pharma company. This was back in 2018.

   4. VPVipul Ved Prakash3:37

      Right.

   5. AGAnkit Gupta3:37

      At the time, saying that you were a software first pharma company was sort of an absurd concept. Um, now I think it is actually very common to think of industries in which things you don't necessarily think of as software companies, whether it's insurance or healthcare or pharma or whatever else-

   6. VPVipul Ved Prakash3:49

      Right

   7. AGAnkit Gupta3:49

      ... are software companies for the first time, um, which expands just, like, the, the pie of things that people might work on, um, in Y Combinator. Um, at the same time, it means there's whole new challenges, like, there's whole new industries in which people are facing bottlenecks that they didn't face before.

   8. VPVipul Ved Prakash4:04

      Right.

   9. AGAnkit Gupta4:04

      So on compute, right?

   10. VPVipul Ved Prakash4:05

       Yeah.

   11. AGAnkit Gupta4:05

       One of the things that's, um, very different, you know, um, eight years ago when I was running my company, we could just, like, go to AW- AWS and get 1,000, you know, uh, instances-

   12. VPVipul Ved Prakash4:14

       Yeah

   13. AGAnkit Gupta4:14

       ... with GPUs. And yeah, they were, like, maybe expensive or whatever.

   14. VPVipul Ved Prakash4:17

       Right.

   15. AGAnkit Gupta4:17

       But it wasn't that hard to get them. Like, you could get them, and you could get them on spot instances, and we didn't have to do long reservations and whatnot. Um, I think now what we see is as people's compute needs, needs go up, actually just having access to capacity is a really big problem, let alone great pricing.

   16. VPVipul Ved Prakash4:31

       Right.

   17. AGAnkit Gupta4:31

       And I think, you know, for us with this partnership, being able to secure capacity, secure it at a really great price, have great support also available, and make it so startups don't have to commit to two years, they only have to commit to a few weeks-

   18. VPVipul Ved Prakash4:43

       Right

   19. AGAnkit Gupta4:43

       ... makes it so that for startups that have these heterogeneous compute needs-

   20. VPVipul Ved Prakash4:47

       Yeah

   21. AGAnkit Gupta4:47

       ... you can kind of solve for that. Um-

   22. VPVipul Ved Prakash4:49

       Right. Right

   23. AGAnkit Gupta4:49

       And so it kind of solves for this bottleneck that can help them access all of these weird new markets that are now for the first time software problems.

6. 4:56 – 5:29

   Why the Cost of Compute Keeps Going Up

   1. VPVipul Ved Prakash4:56

      Yeah. And, and do you see this becoming a differentiator from the perspective of the fund? It, it's, uh, you know, it is becoming harder and harder, and I, I think I, I feel as models are getting better, right, the, the, uh, value of tokens that are being produced is, is increasing, which is sort of has an impact on the value of FLOPS and, and the cost of compute, and you may have this curve of, you know, ever-expanding cost of compute. So-

   2. AGAnkit Gupta5:24

      Yeah

   3. VPVipul Ved Prakash5:25

      ... so finding ways of enabling companies seems

7. 5:29 – 7:07

   YC as the Biggest Seed Funder of Research Companies

   1. VPVipul Ved Prakash5:29

      important.

   2. AGAnkit Gupta5:30

      A- absolutely, and especially because, um, you know, YC has become probably the biggest, um, seed funder of research style companies. You know-

   3. VPVipul Ved Prakash5:37

      Right

   4. AGAnkit Gupta5:37

      ... our, our probably very first real machine learning seed investment was literally OpenAI.

   5. VPVipul Ved Prakash5:42

      Right. [laughs]

   6. AGAnkit Gupta5:42

      Uh, it was-- OpenAI was birthed as YC Research, um, when Sam was president here. Um, and then since then, we've funded all sorts of companies developing their own models in a bunch of different verticals. There's companies like Deepgram making voice AI models. There's companies that are then distributing those and building their own companies like VAPI, for example, or Retail or YC companies. Um, there's a new one called miso Labs, just launched a few weeks ago-

   7. VPVipul Ved Prakash6:02

      Yeah

   8. AGAnkit Gupta6:02

      ... that has an incredible new emotive voice model. Um, in other domains, like in language, you have companies building new edge device models or- Different variants of model architectures that aren't the ones that are currently already being scaled up, and then you have this whole long chain of other, uh, modalities in which people are doing research, whether it's in biology or in healthcare or kind of you name it. And so I, I think what's really exciting right now, and actually I didn't even say anything about hard tech, where there's actually tons of interesting robotics companies that we're, we're funding these days. And for a lot of these, um, they are research bets. They are not companies where we expect them to commercialize in significant ways immediately. They're ones where they have to solve a hard technical problem, and access to compute ends up being a big bottleneck for a lot of them. And so-

   9. VPVipul Ved Prakash6:44

      Right

   10. AGAnkit Gupta6:44

       ... yeah, we, we very much see, um, securing access to compute as a really important thing to attract the best founders to-

   11. VPVipul Ved Prakash6:51

       Yeah

   12. AGAnkit Gupta6:52

       ... do YC, and then also to enable them to succeed, um, both during the program and afterwards. Um, like, what's great is companies can get on this cluster or, you know, on our future ones basically during the batch, and then continue to grow-

   13. VPVipul Ved Prakash7:03

       Right

   14. AGAnkit Gupta7:03

       ... um, with whatever their compute provider is for a very long time.

8. 7:07 – 8:47

   Why YC Chose Together AI

   1. VPVipul Ved Prakash7:07

      Yeah. Yeah, and we, we are definitely seeing that. It's been pretty exciting to, uh, you know, track the journey of some of these companies that-

   2. AGAnkit Gupta7:13

      Yeah [laughs]

   3. VPVipul Ved Prakash7:13

      ... that we've onboarded. What was it about Together that, uh, uh, led you to do this, uh, part- partnership with us?

   4. SPSpeaker7:19

      YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com/apply. It's never too early, and filling out the app will level up your idea. Okay, back to the video.

   5. AGAnkit Gupta7:33

      There was a few things. Um, one is we, um, really prioritize speed. I think being able to work with a partner who is excited to get a cluster up really fast, and then also build it in a way that works with startups with lots of different needs. You know, we have some companies who are just looking for a single node-

   6. VPVipul Ved Prakash7:47

      Right

   7. AGAnkit Gupta7:47

      ... and they just want access now, but they're in the scaling mode where you could totally see them becoming a lot bigger. There's others who are buying 256 GPUs on day one and looking to scale to several thousand very soon.

   8. VPVipul Ved Prakash7:58

      Right.

   9. AGAnkit Gupta7:58

      And being able to offer a product that actually supports all of those use cases was a big ask. And then the second was having a great engineering and research team that could help actually enable the best practices in these companies. You know, a lot of the founders, they have really great engineering backgrounds. They have trained models in other places where there's been managed compute available to them. They haven't necessarily had to manage their own compute cluster. And so having a partner that, um, can help with just showing the ropes so that, um, yeah, of course, there's a lot that's managed by Together directly. There's some parts that the pro- that the founders themselves need to figure out, but they can make it so it's not a zero to one right away.

   10. VPVipul Ved Prakash8:32

       Right.

   11. AGAnkit Gupta8:32

       And there's a lot of best practices to encode. I think that was a big part of it. And then I will say, like, I just lastly, personally, I've actually followed a lot of the research from Together for many years. Like, I'm friends with Suh, and-

   12. VPVipul Ved Prakash8:42

       Right

   13. AGAnkit Gupta8:42

       ... I've, you know, seen some of the work that they've put out in research papers, um, in the past. And so actually maybe

9. 8:47 – 10:43

   Flash Attention, Mamba, and the Science of Production AI

   1. AGAnkit Gupta8:47

      something, be curious if you want to share a little bit more about, like how do you guys think about all of your research efforts, you know, work like Flash Attention, for example-

   2. VPVipul Ved Prakash8:54

      Yeah

   3. AGAnkit Gupta8:54

      ... um, and all the other work you guys do, and how that fits into your product offering. Like, how do you balance those two things, and how do companies benefit from that?

   4. VPVipul Ved Prakash9:01

      When we founded Together, we felt that there was going to be quite a bit of research and effort that's going to go into making models better. So, you know, uh, uh, uh, uh, reducing loss and building better and better models, and there was this entire space, uh, of, uh, production AI, which it's a science, and it was, uh, understudied, and it's really kind of, uh, you know, linear algebra executing in these novel accelerators. It's a new form of computer science, and we felt it was going to be very important because this is, uh, you know, a, a very, very scaled endeavor, and, uh, you are deploying large amounts of capital. Uh, the, the cost of training and the cost of producing a token is very high, and you want to make it really efficient. So a lot of our research has focused on, uh, AI systems from, you know, uh, uh, approaches to, uh, doing attention, and that continues to be a really sort of rich field of, uh, as, as now, uh, you know, attention is, uh, uh, going after longer and longer context-

   5. AGAnkit Gupta10:12

      Right

   6. VPVipul Ved Prakash10:12

      ... to, you know, uh, architectures. We also did work on the Mamba architecture, which is now making its way into, uh, uh, you know, long context attention, like in the Nemotron model.

   7. AGAnkit Gupta10:23

      Yeah, I was gonna say the lo- the latest Nemotron models also seem to use it, right?

   8. VPVipul Ved Prakash10:26

      Yeah, and compilers and, uh... So, so this, this is a very, uh, in- interesting and productive area of research for us and for the companies that work with us because we can accelerate their workloads, make them, and generally achieve the best unit economics possible on

10. 10:43 – 12:39

    Compute Planning Advice for Early Stage Companies

    1. VPVipul Ved Prakash10:43

       these workloads and, and that's really having, uh, uh... It's, it's very useful to early startups, but then also the scaled ones who care a lot about the unit economics of their, uh, applications. So-

    2. AGAnkit Gupta10:56

       Yeah. I, I was gonna ask you on, on that latter point, like, you know, when you think about a lot of the research companies you've worked with, you've seen a lot of them scale, um, on Together's platform. What should early stage companies who are starting to use the Together cluster or a different compute provider think about now to prepare for what's coming, both in terms of the compute shortage-

    3. VPVipul Ved Prakash11:14

       Right

    4. AGAnkit Gupta11:14

       ... uh, and how to sort of set themselves up for success as their company starts to work and get product market fit?

    5. VPVipul Ved Prakash11:19

       Yeah. You know, there's so much infrastructure now, right? Relative to even a couple of years ago, from really frontier class open weights models are, are accessible. There is so much machinery-

    6. AGAnkit Gupta11:32

       Okay

    7. VPVipul Ved Prakash11:32

       ... and tooling for, for, you know, training models, post-training them. There's, uh, rich source of data sets. So I think you can be incredibly efficient, uh, in, in, in terms of kind of using this machinery and, uh, uh, deploying it. And I, I, you know, we do see many of the, uh, YC startups we are working with, uh, are very proficient with the, the current set of, uh, uh, uh, tools that exist. I think the, the compute planning is definitely something that it's complex, right? You, you... When you are a seed stage company, you have a certain amount of capital and your, uh, uh, sort of perspective of what you're going to do with it is very different From, you know, once you get to a, a next level of scale. And, uh, so I th- I think part of what we are trying to do is build products and build price and package them in ways that allow companies to be able to scale up to the, to the next stage. And, and I think some of the collaborations that we are doing are going to be sort of essential in, you know, helping companies

11. 12:39 – 13:31

    When Your Compute Bill Is Bigger Than Your Cash Balance

    1. VPVipul Ved Prakash12:39

       underwrite. And I think this is where, uh, companies are very different from software companies-

    2. AGAnkit Gupta12:44

       Yeah

    3. VPVipul Ved Prakash12:44

       ... 'cause you, you really have to think about this is going to be the biggest expense in some ways for development and, and product.

    4. AGAnkit Gupta12:52

       Yeah, I mean, I think a big challenge we, we see, and I think we saw before having... What really, uh, motivated setting up this cluster was a lot of companies just to be like, just to put into numbers, you know, the upfront they would need to pay in order to secure the capacity-

    5. VPVipul Ved Prakash13:05

       Right

    6. AGAnkit Gupta13:05

       ... for their next two years of compute was greater than their current cash balance.

    7. VPVipul Ved Prakash13:09

       Right. [laughs]

    8. AGAnkit Gupta13:09

       And so you just could not do that-

    9. VPVipul Ved Prakash13:11

       Yeah

    10. AGAnkit Gupta13:11

        ... unless you go and raise, and basically-

    11. VPVipul Ved Prakash13:12

        Right

    12. AGAnkit Gupta13:12

        ... put all of your money into a compute partner.

    13. VPVipul Ved Prakash13:14

        Yeah.

    14. AGAnkit Gupta13:14

        And so a big thing we look to solve with this is, you know, use some of YC's scale so that we can secure access to that compute so a cos- a com- an individual startup doesn't have to make a 24-month bet. They can make a few month bet instead, which is-

    15. VPVipul Ved Prakash13:28

        Right

    16. AGAnkit Gupta13:28

        ... much more doable with access to capital dollars and the way they

12. 13:31 – 14:24

    How YC Companies Are Using the Cluster Today

    1. AGAnkit Gupta13:31

       currently flow. And so they can kind of continuously scale and not have to lock up all their cash basically, and make it so, you know, they raise at more effective terms for themselves.

    2. VPVipul Ved Prakash13:39

       Yeah. Yeah, absolutely. What, what is the kind of feedback, um, that you're hearing from, from these companies?

    3. AGAnkit Gupta13:45

       Oh, it's been great. I mean, I think, well, what's, what's very cool is, you know, there's lots of different companies who have different needs, you know, some with training clusters versus inference clusters. I think they've really loved working with your guys' support team to build the right kind of workflows to work really well and understand the kind of realistic underlying, uh, sort of compute setups that actually have really good performance and availability. Um, and then we've made it so that they can sort of book in advance, and now companies can plan out their compute several months in advance, which has been very helpful for them, where, um, some of them might know that they need, uh, to do a big training run three months from now, and they can choose to buy a relatively small amount of compute for a while, and then a much larger amount.

    4. VPVipul Ved Prakash14:19

       Yeah.

    5. AGAnkit Gupta14:20

       And that flexibility is great for them. And, you know, across the portfolio of

13. 14:24 – 15:47

    What's Next for the Partnership

    1. AGAnkit Gupta14:24

       companies, there's still enough to service all of the demand at all times, so the cluster is at 100% utilization. But for any individual company who doesn't have, you know, a fixed utilization, it's really helpful to have this sort of-

    2. VPVipul Ved Prakash14:35

       Right

    3. AGAnkit Gupta14:35

       ... lever they can pull and plan out in advance.

    4. VPVipul Ved Prakash14:37

       Yeah. Yeah, I'm, uh, really looking forward to expanding this. I think it's working. It's, it's, it's, uh, it's been a very creative way of solving this problem, and I think it's working really well and, and, uh, I think this is going to become essential to startups who are thinking about creating AI products.

    5. AGAnkit Gupta14:54

       Yeah, I think you're gonna be hearing from me quite a lot about how we can make this thing even bigger. I mean, I think, I think in general, you know, at YC we think of ourselves as sort of a founders union.

    6. VPVipul Ved Prakash15:02

       Yeah.

    7. AGAnkit Gupta15:02

       You know, I think a lot of how we, uh, what we, what we push to do is regardless of how someone's building technology, whether it's a research company or an application company or whatever else, is to give the shared resources to allow the founder to have a lot of the leverage that a much bigger company might have, whether that's in terms of raising capital and having things like demo data support that, or a GPU cluster in the way that a bigger company might have, or access to certain advisors or whatever else.

    8. VPVipul Ved Prakash15:27

       Right.

    9. AGAnkit Gupta15:27

       And, um, yeah, I'm excited to bring compute as one of the levers we can pull to kind of enable that for people. I'm, I'm really excited for where this is gonna go.

    10. VPVipul Ved Prakash15:34

        Awesome.

    11. AGAnkit Gupta15:35

        Cool. Thanks so much for coming. This is gonna be, uh, awesome. I'm excited to keep working with you guys.

    12. VPVipul Ved Prakash15:38

        Thanks for having me. [upbeat music]

Episode duration: 15:47