No Priors Ep. 67 | With Voyage AI Co-Founder and CEO

Welcome to No Priors. Today, we're talking to Tenggu Ma, assistant professor of computer science at Stanford, and the co-founder and CEO of Voyage. Voyage trains state-of-the-art components for next generation retrieval systems, including embeddings models and re-rankers. We're really excited to talk about his research and the RAG debate today. Uh, welcome, Tenggu.

Yeah. Thanks so much. Thanks for having me here. Looking forward to the debate.

Yeah. W- why don't we start with, um, just a- a little bit of an overview of y- like, your research agenda to date, because I- I think uniquely, it covers, um, a broad range of fields, uh, within and around deep learning from, like, theory to RL to embeddings and optimizers. So can you talk a little bit about, um, sort of how you picked the directions you have?

Yeah. So, um, I think most of the papers I wrote, uh, have some theoretical thinking in it. I guess maybe that's the commonality. And, um, um, be- besides that, I think I worked on, uh, quite a few topics, as you mentioned, ranging from the theoretical understanding, mathematical proofs of deep learning systems, um, to, um, all the way to practical large language models, reinforcement learning, um, deep reinforcement learning. And these days, recently, I think what we are working on, um, more centralized to, um, efficiency of training the large language models and reasoning, improving the reasoning tasks for large language models. So I- my vision is that in the future, uh, the efficiency is very important because, um, we are running out of data and compute, so we have to either use the data much better and use the compute much better. Um, and also reasoning tasks seems to be a pretty, uh, important direction. And, um, um, uh, and also in some sense, kind of like a- a risky direction in the sense that we don't know exactly how, um, how fast we can, uh, solve those challenging reasoning questions yet.

Mm-hmm. Um, can you, uh, mention a few of the, like, key papers or work that, uh, you know, you or students in your lab have done, just so our listeners can look them up?

In the very, uh, early days, I think I worked on some of this matrix completion, optimization for matrix completion. That's like 10 years ago. Um, and then I move on to, uh, embedding models, like sentence embeddings, vector embeddings. One of the papers we wrote is a ve- actually simple paper where we average the word embeddings to get sentence embeddings, and then we did some, uh, of these transformations using PCA to make the performance much better. Um, that was even before transformer came out. Um, and then I think I move on to, uh, transformers, large language models, um, and contrastive learning, which is the new way of training the embedding models. Um, uh, especially the- the, um- um, the direction started with some of the papers on, uh, using contrastive learning for images, and we work on improving those and understanding why contrastive learning can work. Uh, and recently, we work on, uh, optimizers for large language models. For example, one of the paper, uh, we wrote last year was Sofia, uh, which we found, where we found that we have a new tran- optimizer which can improve the training efficiency by 2X for pre-training.