How a reasoning model cracked an 80-year-old math problem — the OpenAI Podcast Ep. 20

1. 0:00 – 0:44

   Intro

   1. AMAndrew Mayne0:00

      Hello, I'm Andrew Mayne, and welcome to the OpenAI Podcast. On today's episode, we're speaking with Alexander Wei, Hongxing Wu, and Lijie Chen from the reasoning research team behind a recent math breakthrough from an OpenAI model. They'll tell us the story behind the discovery and what stood out to them about the reaction.

   2. LCLijie Chen0:18

      Everyone had a hard time sleeping because it's so [laughs] so exciting, yeah.

   3. HWHongxun Wu0:22

      Okay, this model is something that's really amazing.

   4. LCLijie Chen0:25

      I mean, this is something that can be published in the best journal of maths.

   5. AWAlexander Wei0:30

      Maybe this is the one in 100 times where it's too good to be true, but it's, it's actually true.

   6. AMAndrew Mayne0:37

      Lijie, tell me what you work on.

   7. LCLijie Chen0:39

      Oh, w- I work on reasoning with Alex.

   8. AMAndrew Mayne0:42

      Okay. How did you find your way into reasoning?

2. 0:44 – 6:35

   AI and the International Math Olympiad and International Olympiad of Informatics

   1. LCLijie Chen0:44

      Last summer, Alex had this breakthrough in, like, IOI and IMO.

   2. AMAndrew Mayne0:48

      Mm-hmm.

   3. LCLijie Chen0:48

      You know, I used to be a parti- participant in IOI.

   4. AMAndrew Mayne0:52

      Okay.

   5. LCLijie Chen0:52

      And, uh, that was like, oh, that's crazy, you know, a model can already win medals, uh, um, gold medals. At that time, I was a profes- assistant professor at UC Berkeley, but then I'm thinking, like, maybe I should try to rethink my, [laughs] my career.

   6. AMAndrew Mayne1:07

      [laughs]

   7. LCLijie Chen1:07

      It seems like making the model smarter will have, maybe have some bigger impact on the, on the world. And that, and then I just kind of had a conversation with Alex, like, back in n- last October, and then I got super excited about this thing, and eventually I just joined OpenAI.

   8. AMAndrew Mayne1:24

      We hear IOI and IMO come up a lot. Alex, you wanna unpack those for everybody?

   9. AWAlexander Wei1:28

      So IMO and IOI are, are these two competitions for high schoolers. Uh, they stand for the International Math Olympiad and In- International Olympiad of Informatics respectively, and these are just devilishly hard math problems. Uh, you get two sessions for each of these exams that are, like, four and a half to five hours, and you just have to do three problems. And so for a long time, uh, these were sort of an implicit, like, grand challenge in AI. Like, when would we be able to get models that could perform as best, as well as the best humans on these exams?

   10. AMAndrew Mayne2:06

       That was a pretty interesting starting point, I think, for measuring the success of the model, and we're here to talk about how far things have gone since then, which is pretty incredible. But how did you find your way into reasoning?

   11. AWAlexander Wei2:16

       So I did my PhD in ML.

   12. AMAndrew Mayne2:19

       Mm.

   13. AWAlexander Wei2:19

       Um, and towards the end of my PhD, I got excited about this idea of spending more compute at inference time-

   14. AMAndrew Mayne2:26

       Mm

   15. AWAlexander Wei2:27

       ... to solve, uh, you know, harder and harder reasoning problems. At the time, I was playing with, uh, GPT 3.5 Turbo in the API.

   16. AMAndrew Mayne2:35

       Mm-hmm.

   17. AWAlexander Wei2:36

       And, uh, I, I didn't really get any interesting results, but there was this team at OpenAI that was, seemed to be doing something pretty similar, and so I got super excited about it, and, you know, was lucky enou- enough to be able to join.

   18. AMAndrew Mayne2:48

       So probably the simplest way to describe that is, like, letting the inference time is basically letting the model think longer about it.

   19. AWAlexander Wei2:55

       Yes, that's right. So basically, um, before this era of test-time compute, models sort of answered immediately without, like, right off the cuff-

   20. AMAndrew Mayne3:06

       Mm

   21. AWAlexander Wei3:06

       ... without thinking. And what inference-time compute, test-time compute does is you now g- let the model, give the model a chance to think and improve its answer and, like, try different things before having to finally output something.

   22. AMAndrew Mayne3:19

       Mm.

   23. AWAlexander Wei3:19

       That obviously just h- helps make the model smarter, let- lets them do things that they wouldn't otherwise be able to do instantly.

   24. AMAndrew Mayne3:26

       When you started to work on reasoning, did you have an idea of where you wanted to see this go? Like, what your expectations were? Were you looking at it purely from, "Hey, this is very cool from an academic point of view," or did you have some sort of other vision?

   25. AWAlexander Wei3:39

       Uh, I think for me, the draw of reasoning when I first got excited about it was that this was something that, you know, models just obviously can't do right now.

   26. AMAndrew Mayne3:48

       Mm-hmm.

   27. AWAlexander Wei3:49

       Um, so this was like end of 2023, start of 2024. Models were, you know, struggling with grade school math problems.

   28. AMAndrew Mayne3:57

       Mm-hmm.

   29. AWAlexander Wei3:57

       And so at that time it was just like, can we just get the, these models to do something reasonable on m- math at all, let alone, like, have them be like, you know, like, you know, much, much better than I am at it? Uh, I remember when I s- the first, my first day at OpenAI, uh, Nolan Brown asked me, uh, you know, when I thought models would get IMO gold. Uh, that, that was, uh, just a benchmark we talked about. I think at the time, a lot of people, even within research, thought that, like, you know, uh, IMO gold was out of reach this year, but maybe, like, 2026.

   30. AMAndrew Mayne4:34

       Mm.
3. 6:35 – 8:33

   An OpenAI model disproves the Erdős unit distance conjecture

   1. AMAndrew Mayne6:35

      You had a model that was able to basically disprove one of the Erdős conjectures. Would you, could you explain that just a little bit?

   2. AWAlexander Wei6:42

      Yeah, so our models last week, uh, they were able to produce a proof of the u- or a disprove rather, of the unit distance conjecture due to Erdős, and this was an 80-year-old open problem in the field of, field of, uh, combinatorial geometry, where basically the question concerns, uh, if you have endpoints, let's say on a piece of paper-

   3. AMAndrew Mayne7:06

      Mm-hmm

   4. AWAlexander Wei7:07

      ... how many of them can be one inch apart, uh, exactly? And h- how, how many pairs can be one inch apart exactly? And how does this number grow asymptotically with the number of points on the piece of paper?

   5. AMAndrew Mayne7:22

      This wasn't a, a trivial problem. When Erdős put this together, the idea was to say that it could, you know, like, I think ideally was had to be only done on a plane or something like this, but there was, you know, the idea that maybe there was no better way. And this has been out there because it's a very interesting problem, and the fact that a model solved this is pretty profound, and also this model was a general-purpose model, correct?

   6. AWAlexander Wei7:45

      Yes, that's right. So Erdős' original conjecture was essentially that, uh, the optimal cons- the optimal solution to, uh, having as many, uh, distance one points on the plane was to arrange them in a square grid.

   7. AMAndrew Mayne8:00

      Mm-hmm.

   8. AWAlexander Wei8:01

      And what the model proved was that the square grid was not actually, uh, close to optimal at all, and that you can do much better, uh, with a different construction using a lot of, uh, high-powered number theory.

   9. AMAndrew Mayne8:16

      Hongxun, how did you, how did you choose these problems?

   10. HWHongxun Wu8:19

       I, I guess we didn't really choose the problem. We, what happened was, uh, we want to test the upper bound of our model's capability.

   11. AMAndrew Mayne8:26

       Mm-hmm.

   12. HWHongxun Wu8:27

       So we just used a selected subset of Erdős problems and to test the capability of the model.

4. 8:33 – 11:04

   Running the model and checking the proof

   1. AMAndrew Mayne8:33

      I would love to know, one, who is the one that hit Enter and asked the model the question?

   2. LCLijie Chen8:39

      I guess both of us, like I and Hongxun.

   3. AMAndrew Mayne8:41

      You guys at the same time, like pressed? [laughs]

   4. LCLijie Chen8:44

      [laughs] Yeah, maybe.

   5. HWHongxun Wu8:46

      I, I think what happened was actually we were testing, like, two side different internal models.

   6. AMAndrew Mayne8:51

      Mm-hmm.

   7. HWHongxun Wu8:51

      And we both saw, like, some correct solutions.

   8. AMAndrew Mayne8:55

      Mm-hmm.

   9. HWHongxun Wu8:56

      Uh, it was really, really exciting for us.

   10. AMAndrew Mayne8:58

       How did you know that it worked?

   11. LCLijie Chen8:59

       Of course, you first ask the model to check it.

   12. AMAndrew Mayne9:01

       Okay. [laughs]

   13. LCLijie Chen9:02

       Uh, but of course, you know, models sometimes they are not reliable, so like-

   14. AMAndrew Mayne9:05

       [laughs] I got it. It's good. Don't worry about it.

   15. LCLijie Chen9:06

       [laughs] Yeah, so then we just, uh, after we check it with the model, it seems plausible, then we just ask a bunch of, you know, other, our mathematics friends in the company, you know, um, Mattab and, uh, Makselki.

   16. AMAndrew Mayne9:18

       Mm-hmm.

   17. LCLijie Chen9:18

       And at first they were like, "Oh, there's no way this can be true."

   18. AMAndrew Mayne9:21

       [laughs]

   19. LCLijie Chen9:21

       "This is a major open problem." And, uh, but after, you know, just they think about it for five days, they couldn't figure out any mistake, then they become more convinced, then eventually they are like, "A- actually this may be correct." Yeah, then, like, everyone had a hard time sleeping because it's so, [laughs] so exciting, yeah.

   20. AMAndrew Mayne9:39

       What was the conversation like when you started getting, you know, people saying that this was accurate?

   21. HWHongxun Wu9:44

       For me, I was not that surprised because I guess when Mattab first say... Okay, first, what happened was first Mattab say, "This is definitely wrong."

   22. AMAndrew Mayne9:52

       Okay. [laughs]

   23. HWHongxun Wu9:53

       But I actually knew that he probably just spent like five minute, 10 minute-

   24. AMAndrew Mayne9:56

       Right

   25. HWHongxun Wu9:56

       ... looking at it. [laughs] So, like, in my heart, I don't really believe that. Um, but later he told me it's 50%. I was thinking, "Okay, if we extrapolate the trend, then maybe next night it will be 8- 100%." [laughs] Um, so yeah, I guess it, it's a little bit dreamlike, um, but also was, like, it feels a bit, a little bit natural that, um, that this model would do something amazing. Later it, it just become more and more, um, more and more real that, um, this might actually be correct. This might actually be a big deal, the first time the model can publish something that get into like, uh, top math journals.

   26. AMAndrew Mayne10:37

       Mm-hmm.

   27. HWHongxun Wu10:38

       It's, like, we knew this day is gonna come but never knew that it's gonna become, it's gonna become reality so fast. It's like living a dream.

   28. LCLijie Chen10:46

       I mean, this is something, like, can be published in, like, the best journal of maths. It is way beyond like a, you know, IMO level. [laughs]

   29. AMAndrew Mayne10:55

       Mm-hmm.

   30. LCLijie Chen10:56

       So I, I only expect something to happen at some time, but at some, at some point, but maybe not this, just not this May, yeah.

5. 11:04 – 15:55

   Why general models matter for discovery

   1. AMAndrew Mayne11:04

      One of the things I think that we, we've seen emphasized at OpenAI is that OpenAI doesn't try to train to specific benchmarks and stuff, that OpenAI tries to build really good general overall models, and I think sometimes people say like, "Well, we just try to build a generally smart model, and we find these things a lot of way, and when it comes to reasoning, it's the same thing. Something that's really good reasoning overall, you find these capabilities." Does that ring true for you or?

   2. AWAlexander Wei11:27

      Yeah, I think, yeah. I think for, for this model in particular, I think it's one that, um, I think all of us have also just used, um, like in lieu of, uh, the current model, uh, current model in, uh, Codex, and it, it, it works quite well as just a general purpose model. Having the capabilities to do this, uh, Erdős, uh, unit distance result, I think people will be able to do this at home-

   3. AMAndrew Mayne11:54

      [laughs]

   4. AWAlexander Wei11:55

      ... in the, uh, in the near future.

   5. AMAndrew Mayne11:57

      It's been exciting to see people react to this and pay attention to this. Uh-We went from just a very short period of time ago where people said models weren't good at math, and now models are doing this. What have been some of the more fun things you've seen online or reactions from people?

   6. HWHongxun Wu12:13

      Ever since we announced the results, my friend, uh, Ying, he says, start to, uh, asking me to try to- try out their pro- open problems.

   7. AMAndrew Mayne12:21

      Mm.

   8. HWHongxun Wu12:22

      Uh, including my advisor gave me like two, three open problems to try out. Um, I think the reaction was, um, very positive.

   9. AWAlexander Wei12:30

      I think people really get a sense that the frontier of AI today can really, uh, come up with, you know, research output that I think many human mathematicians would be proud to achieve.

   10. AMAndrew Mayne12:42

       Mm.

   11. AWAlexander Wei12:43

       Um, and I think it's really great that we're able to communicate this like, you know, that this is the frontier of progress to the rest of the world. I've seen people like, you know, m- make these, uh, make these like designs of trying to like sketch out, um, like the model's construction and if you plot it on a grid, it's actually like this very like pretty like symmetric geometric design.

   12. LCLijie Chen13:07

       Yeah, I guess we are thinking maybe try to make one of the design, you know, put them in a frame and, uh, put them on a desk or something. [laughs]

   13. AWAlexander Wei13:16

       [laughs]

   14. LCLijie Chen13:16

       Kind of to celebrate this like, you know, moment.

   15. AMAndrew Mayne13:19

       Yeah, I think it's gonna be fun when we start seeing it, things like tiling problems and other stuff where we can actually just look at the, the artifacts we need. So we've been hearing more about Erdős problems lately, and some seemed like they weren't as challenging to solve as perhaps as people thought. They just needed some attention, yet this one seems to be a little bit more complicated. Where would you rank this?

   16. LCLijie Chen13:39

       Oh, yes. I think he proposed like a thousand question or more, right? So like he, you know, Er- Erdős problem is just collection of all the problems he has asked.

   17. AMAndrew Mayne13:48

       Mm.

   18. LCLijie Chen13:48

       You know, some problem he has offered some money for, for solution.

   19. AMAndrew Mayne13:52

       Mm.

   20. LCLijie Chen13:52

       Some problem he could just, you know, note it. And this, this problem he, you know, he offered I think $500-

   21. AMAndrew Mayne13:58

       Mm

   22. LCLijie Chen13:59

       ... uh, which is for last century, so you know, it's, it was a little bit... Yeah. And also like this is one of the central question in this field of, uh, discrete geometry, and, you know, and this is, has been, you know, heavily discussed by mathematician in like many discrete geometry papers. And so it's kind of one of the question people have thought about a lot and really want to see the answer. So I would say this is more like a major open problem in a concrete field of mathematics instead of some just like, you know, um, many other Erdős question which may be just some, you know, um, something like Erdős ask after lunch or something. [laughs]

   23. AMAndrew Mayne14:40

       So how do you collect that $500? Did it, did it, did it, did it disappear when he passed away? [laughs]

   24. LCLijie Chen14:45

       Uh, I think there's a special agency for that-

   25. AMAndrew Mayne14:48

       Oh

   26. LCLijie Chen14:48

       ... but, uh, you usually people just frame the, the check.

   27. AMAndrew Mayne14:51

       Yeah.

   28. LCLijie Chen14:51

       Yeah, so maybe we'll just frame the check in Sam's office. I don't know. [laughs]

   29. AMAndrew Mayne14:55

       Yeah. [laughs] How do you feel this proves that reasoning is effective?

   30. LCLijie Chen15:00

       Well, I think the biggest, uh, proof is that if you look at the plot, the, in the official blog-
6. 15:55 – 18:25

   Creativity, tools, and how the proof worked

   1. AWAlexander Wei15:55

      what can it do?

   2. AMAndrew Mayne15:56

      When you go through the proof and you look at what it came up with, were there things that surprised you, things that you would describe as creative?

   3. AWAlexander Wei16:01

      So for some context, like the proof is like well above my own mathematical pay grade.

   4. AMAndrew Mayne16:06

      Mm-hmm.

   5. AWAlexander Wei16:07

      Um, but like just at, at a high level, um, my understanding was that, you know, this, this idea of, uh, taking class field theory and applying it to, uh, problems in, in combinatorial geometry hadn't really been done before, though this was, though there were like, you know, though some people like knew that there was, there could be this bridge between these two fields. Being able to do that and execute it requires like to, first of all, to make the connection requires, uh, quite a bit of like insight and creative, creativity, and then to execute the proof is also like, you know, a very like delicate, careful affair that very few people would be able to do.

   6. HWHongxun Wu16:48

      I think the most surprising thing for me is you tell model do something-

   7. AMAndrew Mayne16:52

      Mm

   8. HWHongxun Wu16:52

      ... and you, you went to have a lunch, and when you come back, you see that it actually does much better than you thought.

   9. AMAndrew Mayne17:00

      Hmm.

   10. HWHongxun Wu17:00

       And at that moment you feel like, okay, this model is something that's really amazing.

   11. AMAndrew Mayne17:05

       So going back to GPT-3.5 Turbo and working with that, and looking at a model that was doing automatically instant sort of, uh, inference and fre- figuring these things out, to now a model that's able to do incredible mathematical proofs, is it using tools? Is it using Lean? Is it using some other things like that, or is this doing purely inside the model?

   12. LCLijie Chen17:28

       So for this particular case, the model basically is like Codex. It can code-

   13. AMAndrew Mayne17:32

       Mm

   14. LCLijie Chen17:33

       ... it can, um, look at the website and, and find information.

   15. AMAndrew Mayne17:37

       Mm-hmm.

   16. LCLijie Chen17:37

       Yeah, so it's, it's basically a general ChatGPT setup.

   17. AMAndrew Mayne17:40

       Okay.

   18. LCLijie Chen17:40

       ChatGPT can also write Python and execute them, but I don't think the mo- the, the model write any Lean. Yeah.

   19. AMAndrew Mayne17:46

       Mm.

   20. HWHongxun Wu17:47

       I think Lijie has a story about the Cambridge, uh, dictionary. [laughs]

   21. LCLijie Chen17:50

       Oh, okay.

   22. HWHongxun Wu17:51

       Yeah.

   23. LCLijie Chen17:51

       So okay, the first thing the model do when it gets the website is to check what unit means-

   24. AMAndrew Mayne17:57

       [laughs]

   25. LCLijie Chen17:57

       ... in the Cambridge dictionary. It's a little bit [laughs] ridiculous. Yeah.

   26. AMAndrew Mayne18:01

       So it like looked up the word unit?

   27. LCLijie Chen18:03

       Yeah. It wants to make sure it has the absolute correct u- understanding of what is unit.

   28. AMAndrew Mayne18:08

       Have you seen it do other things like that where you're seeing like, oh, it's trying to ground itself to make sure it understands the question?

   29. HWHongxun Wu18:14

       And definitely. A lot of time in the model answer it will actually, uh, explain the definition again-

   30. AMAndrew Mayne18:19

       Yeah
7. 18:25 – 22:31

   Why AI should feel empowering for mathematicians

   1. AMAndrew Mayne18:25

      people who are very knowledgeable about computer science, people who, uh, know a lot about mathematics, is it intimidating to all of a sudden see this happen?

   2. HWHongxun Wu18:34

      I think it should not be intimidating.

   3. AMAndrew Mayne18:36

      Mm.

   4. HWHongxun Wu18:36

      I just think it should be empowering instead.

   5. AMAndrew Mayne18:38

      Okay.

   6. HWHongxun Wu18:38

      After the proof actually come out, uh, like mathematician has improved, first improved the bound they proved.

   7. AMAndrew Mayne18:44

      Mm-hmm.

   8. HWHongxun Wu18:45

      Uh, and second, they use the intuition, the motivation of the construction to, um, knock down other open problems as well.

   9. AMAndrew Mayne18:53

      Mm-hmm.

   10. HWHongxun Wu18:54

       So I think the trend is gonna continue. Um, like model can make good breakthrough on some very hard questions we don't know how to solve. But then how to digest that idea, how to, uh, use that method for other good things, uh, I think human still, uh, has a role in this.

   11. AMAndrew Mayne19:13

       So what do you think the role of somebody working in mathematics is going to be like five years from now?

   12. LCLijie Chen19:18

       I think there'll be a lot of AI and the human collaboration.

   13. AMAndrew Mayne19:21

       Mm-hmm.

   14. LCLijie Chen19:21

       Yeah, because AI... And now, you know, AI they know a lot, right? They can connect distant ideas, but human can also think for longer. Like currently it seems AI cannot build a new theory for maths-

   15. AMAndrew Mayne19:33

       Mm-hmm

   16. LCLijie Chen19:33

       ... for example. But, uh, I, I guess human, once they have the help of AI, they can just grab all the ideas from distant field of maths. I think they can empower human way more. Yeah.

   17. AMAndrew Mayne19:43

       Do you see this working into other fields? Are we gonna see discoveries in physics?

   18. AWAlexander Wei19:48

       So I, I can't speak for, uh, physics, but um, I mean, I guess, like we're, we're all researchers in AI, and I think definitely for me, like my day-to-day looks completely different than when I s- first started, uh, doing res- research in this field. Um, I think like so much of my work is now done by coding agents. Um, like I can, I can just like do so much more, and I think that, that's been a sort of like magical feeling that like with AI you, you can... You, you're really starting now to f- feel like you can use AI to build AI faster.

   19. AMAndrew Mayne20:26

       How much has AI changed the way you do these sorts of things?

   20. HWHongxun Wu20:29

       I think changed completely. Like even when I just joined, uh, half a year ago, uh, I was hand-coding the codes.

   21. AMAndrew Mayne20:37

       Mm-hmm.

   22. HWHongxun Wu20:37

       Uh, looking up the Slack channels for like directions, uh, but now the default is just ask Codex.

   23. AMAndrew Mayne20:45

       Mm.

   24. HWHongxun Wu20:45

       And I ask Codex do a lot of things and, um, then I just go to lunch, I just go to, uh, you know, talk to people.

   25. AMAndrew Mayne20:53

       Mm-hmm.

   26. HWHongxun Wu20:54

       Um, the, the work completely changes.

   27. AMAndrew Mayne20:56

       And now you use Codex on your phone, and you can check on it.

   28. HWHongxun Wu20:59

       Yeah.

   29. AMAndrew Mayne20:59

       It's, uh, it's interesting how much more I want to do things now that you have this sort of tool that can work all the time and do stuff. Lijie, how do you explain this to your friends who are sort of trying to understand what this means and how it's gonna impact other fields?

   30. LCLijie Chen21:14

       So I mean, I have some math- mathematician friends, and I have some, you know, friends in, in other fields. Yeah. So I think the way I wo- I would tell them is that I feel like, you know, some, some, some maybe are afraid that, you know, AI will replace them. You know, AI will just replace mathematician. Yeah, but I think it's really about, you know, empowering like every theoretical researcher. Yeah, because you know, AI really have this advantage of knowing so many stuff and connect things. Currently it seems like the problem hard for human may not be hard for AI.
8. 22:31 – 27:24

   Advice for researchers using AI

   1. AMAndrew Mayne22:31

      I was a researcher, how would I get started? What advice would you have to say, "Okay, try this first"? We'll start with you, Hongxun.

   2. HWHongxun Wu22:37

      Get GPT Pro subscription.

   3. AMAndrew Mayne22:39

      [laughs]

   4. LCLijie Chen22:40

      Yeah, of course. Of course.

   5. HWHongxun Wu22:40

      It's really, really much better than, uh, thinking without Pro, uh, and because it think longer.

   6. AMAndrew Mayne22:46

      Yeah.

   7. HWHongxun Wu22:46

      Uh, and try to, um, ask the boldest question you can ask.

   8. AMAndrew Mayne22:52

      Hmm.

   9. HWHongxun Wu22:52

      I had experience that sometimes I try to decompose a problem into smaller problem and ask the model, and turns out that it was not as good as just directly ask the question-

   10. AMAndrew Mayne23:03

       Hmm

   11. HWHongxun Wu23:03

       ... because my decomposition was not the best way. [laughs]

   12. AMAndrew Mayne23:05

       Why do you think that was?

   13. HWHongxun Wu23:07

       I think because as human we have all kinds of priors on-

   14. AMAndrew Mayne23:10

       Mm-hmm

   15. HWHongxun Wu23:11

       ... how problem should be solved, and they are very helpful in reducing the thinking time. Um, but, uh, very often the prior are wrong and there are blind spots.

   16. AMAndrew Mayne23:20

       Mm-hmm.

   17. HWHongxun Wu23:21

       And AI models, they sometimes just can, you know, surprise us with-

   18. AMAndrew Mayne23:26

       Mm

   19. HWHongxun Wu23:26

       ... uh, discovering these hidden things.

   20. AMAndrew Mayne23:28

       When I spoke to Alex Olchawska, he talked about how kind of treating it like a graduate student.

   21. HWHongxun Wu23:33

       Mm-hmm.

   22. AMAndrew Mayne23:33

       You know, not, not talking down too low, but not too high, but at the right level so you could just understand that it knew the terms and it worked for you. Alex, how about for you? What advice would you give somebody who's a researcher who's, wants to try to figure out how to be more effective with this?

   23. AWAlexander Wei23:46

       Yeah, I think a lot of it is actually like, I think these days learning to trust the model-

   24. AMAndrew Mayne23:52

       Mm-hmm

   25. AWAlexander Wei23:52

       ... and like figuring out like, you know, how far you can go in trusting the model, and also learning like, you know, what's beyond what the model can do. Because if you don't have a sense of that, you don't like, you know, maximally-

   26. AMAndrew Mayne24:04

       Mm

   27. AWAlexander Wei24:04

       ... uh, use, uh, the full capabilities of the model. I think Lijie has taught me a lot about how to use these tools better. I, I, I feel like I'm, I'm like sort of a dinosaur in some respects-In terms of a- adoption. 'Cause I think I, I started, like, working at OpenAI well before these tools existed, and so I think I have a lot of old bad habits-

   28. AMAndrew Mayne24:25

       Hmm

   29. AWAlexander Wei24:25

       ... where I, I don't trust the models enough. I still think it's, like, the models of six months ago or something.

   30. AMAndrew Mayne24:31

       That's an interesting paradigm. Okay. So Lijie, what advice would you give?
9. 27:24 – 37:30

   What comes next for math and AI research

   1. AMAndrew Mayne27:24

      how long before there are no more unsolved Erdős problems?

   2. LCLijie Chen27:26

      Some of them are very, very hard.

   3. AMAndrew Mayne27:28

      Yeah.

   4. LCLijie Chen27:29

      Yeah, so yeah, I don't know.

   5. AMAndrew Mayne27:31

      Do you foresee us, maybe Alex, needing to come up with a new category of problems? [laughs]

   6. AWAlexander Wei27:35

      I think probably, like, the hardest problems on that list, I think that list includes, like, the Collatz conjecture.

   7. AMAndrew Mayne27:40

      Mm-hmm.

   8. AWAlexander Wei27:40

      Like, these are problems that feel like very, very far out of reach of, like, the mathematical technology of today-

   9. AMAndrew Mayne27:46

      Mm-hmm

   10. AWAlexander Wei27:47

       ... even though many of them are, like, quite simple to state.

   11. AMAndrew Mayne27:49

       So we'll still have some more things to work on and continuously move things through. That's good to know. It's exciting though too to start to think about what happens when you do start applying this to other areas, in physics, in astronomy, and start looking at data sets and stuff, and what kind of discoveries are gonna be in store. Do you have any particular area that you're hoping to see?

   12. HWHongxun Wu28:09

       Oh, I hope they just solve P versus NP. [laughs]

   13. AMAndrew Mayne28:12

       Okay. How about you, Alex?

   14. AWAlexander Wei28:15

       I think the next milestone in my head is really, like, AI that can, like, do AI research.

   15. AMAndrew Mayne28:20

       Mm-hmm.

   16. AWAlexander Wei28:21

       Um, I think there are so many, like, u- unsolved problems here. We're, in a, in a sense, like, in many ways, like, limited by, like, the, you know, all the limitations of just, you know, our own intelligences. I, I'm optimistic about, like, you know, just having AI broadly available as a technology because there's just, like, so much more demand for intelligence in the world that, you know, like, humans can supply.

   17. AMAndrew Mayne28:49

       Lijie?

   18. LCLijie Chen28:49

       Oh, I wanted to say P versus NP too, but Hongxun said it.

   19. AWAlexander Wei28:53

       Yeah. [laughs]

   20. LCLijie Chen28:53

       Uh, so I guess beyond that, like, one concrete thing I'm very interested in is, like, you know, like, currently it seems AI is trying to combine ideas from different fields and of, of course, in a very novel and, uh, you know, sophisticated way. But, like, can, can AI actually generate completely new ideas from scratch? I mean, that's something, like, we haven't really seen concretely in AI-

   21. AMAndrew Mayne29:15

       Mm-hmm

   22. LCLijie Chen29:16

       ... and, uh, that's something I maybe want to see next happening, and that can be very cool.

   23. AMAndrew Mayne29:21

       Have you seen traces of that yet?

   24. LCLijie Chen29:22

       I think so. Like, um, you know, even in this, like, Erdős problem, I mean, I think some... if you look at, like, the chain of thought, which is like 125 pages, I think some, some of the thoughts are pretty creative-

   25. AMAndrew Mayne29:34

       Mm

   26. LCLijie Chen29:34

       ... although they didn't work out. Yeah. [laughs] I mean, the, the final idea is more, like, combining all the stuff, but some of... it has some creative thoughts.

   27. AMAndrew Mayne29:42

       Well, it is interesting. You know, early on, arguments were like these models weren't creative, but you could give it two ideas that had never been connected before and say, "What's the relationship?" And that would be something very, very new and felt like something different, and I feel like we'll probably be seeing more of that. Do you see us coming up with new forms of mathematics?

   28. HWHongxun Wu30:00

       Um, I think that actually be, will be a further way down the line-

   29. AMAndrew Mayne30:04

       Mm-hmm

   30. HWHongxun Wu30:04

       ... um, because-
10. 37:30 – 41:16

    Cryptography, quantum computing, and the future

    1. AMAndrew Mayne37:30

       Do you foresee things applying to like cryptography and there's also some debate too about do these models get so good that we kind of surpass even where quantum computing goes, which sounds kind of crazy.

    2. LCLijie Chen37:45

       Yeah. I think cryptography is really a im- im- important topic these days because, you know, the foundation of cryptography is, is really about some problems like factoring.

    3. AMAndrew Mayne37:54

       Mm.

    4. LCLijie Chen37:54

       It's hard to solve by, by computers, right? But basically we only have conjecture. There's no mathematical proof of this fact.

    5. AMAndrew Mayne38:02

       Mm.

    6. LCLijie Chen38:02

       And suppose the model gets really good at the, at, at, you know, algorithms, maybe they will prove, you know, some of the cryptography conjecture and saying, "Okay, those, those, those protocol, they're actually secure. We don't have to conjecture them to be secure." Or maybe they'll have s- they'll find some loophole, and that's also very important. Like, you know, w- I think we need to make sure, you know, the, the foundation of all, of all security is good, so the model can stress test the, the foundation of the cryptography to make sure like we have better security.

    7. AMAndrew Mayne38:33

       What about quantum computing?

    8. LCLijie Chen38:36

       Uh, I think that's a very different territory, right?

    9. AMAndrew Mayne38:39

       Mm.

    10. LCLijie Chen38:39

        Like quantum computing like... Okay, actually, I used to study quantum computing. Like the, my first paper is, uh, is on quantum advantage [laughs]

    11. AMAndrew Mayne38:47

        Mm.

    12. LCLijie Chen38:47

        Which shows like for some tasks quantum computer can do better than classical, um, com- computers. But so far I think the models, I mean, they are just classical computers. I mean, they are, they do what human can do.

    13. AMAndrew Mayne38:59

        Mm.

    14. LCLijie Chen39:00

        I mean, maybe a bit better. The quantum computer they, they can sometimes do like more fancy stuff-

    15. AMAndrew Mayne39:05

        Mm

    16. LCLijie Chen39:05

        ... like simulating some quantum, um, effect in chemistry, which we probably not su- okay, I'm not a, an expert on, on that-

    17. AMAndrew Mayne39:13

        Mm

    18. LCLijie Chen39:13

        ... but, uh, that might not... It's unclear. Like it is just two different paradigms.

    19. AMAndrew Mayne39:17

        Mm.

    20. LCLijie Chen39:18

        So yes, I'm not super sure how they like, uh, compare to each other.

    21. HWHongxun Wu39:20

        But I think AI's gonna greatly accelerate the s- the pace that we develop quantum computers.

    22. AMAndrew Mayne39:25

        Mm.

    23. HWHongxun Wu39:26

        Uh, like in recent, just in these years, uh, there's improvement in like error correcting.

    24. AMAndrew Mayne39:32

        Mm.

    25. HWHongxun Wu39:33

        Uh, like you have error correcting code- quantum error correcting codes that, uh, only uses like simpler type of operations.

    26. AMAndrew Mayne39:40

        Mm.

    27. HWHongxun Wu39:40

        And that really speed up the like physical implementation. So, uh, I expect more of these to come from like, uh, collaboration with AI, uh, that, uh, AI can propose new like quantum error correction algorithms-

    28. AMAndrew Mayne39:55

        Mm

    29. HWHongxun Wu39:55

        ... and then we can develop the quantum computers much faster.

    30. LCLijie Chen39:58

        Once you ask the model to solve a question, you can of, of course follow up with, you know, "How did you solve it? Uh, can you explain this part of the proof to me?" And, uh, then the model will patiently try to t- teach you how, how everything goes ab- line by line. Yeah. So it's like, it's actually not just, you know, one, one-shot problem solving. You can ask it a lot of question to, you know, to learn the how the proof works, and I, I really like that.

Episode duration: 41:17