ChartNet: Training Vision-Language Models to Understand Charts

Ankit Gupta 0:00 [upbeat music]

Ankit Gupta 0:07 We're here at YCML with Jovana, a PhD candidate at MIT. She's gonna tell us a little bit about her work that she presented at CVPR this year. Jovana, tell us a little about ChartNet.

Jovana Kondic 0:16 Thank you so much, Ankit. Um, so this is a foundation models data generation pipeline and dataset, uh, where we enable the next generation of vision-language models to do robust chart understanding.

Ankit Gupta 0:28 Right.

Jovana Kondic 0:28 And this work was, uh, done jointly with MIT, IBM Watson AI Labs, and IBM Research. Um, so what makes chart understanding difficult for VLMs? Well, unlike natural images, um, the models have to understand the text, and they also have to engage in some kind of numerical reasoning.

Ankit Gupta 0:46 So this is a chart, like a bar graph, for example, or something to that effect?

Jovana Kondic 0:49 A bar graph is the simplest one-

Ankit Gupta 0:50 Okay

Jovana Kondic 0:50 ... but they can go really wild. You know, you can imagine a crazy scatter plot, a violin plot, any, any-

Ankit Gupta 0:56 Figure one in a nature paper, basically-

Jovana Kondic 0:58 Exactly

Ankit Gupta 0:58 ... type of thing. Yeah.

Jovana Kondic 0:58 Exactly. So we're, we're, um, trying to, uh, deliver kind of the most comprehensive, largest to date, most diverse, yet open source, uh, and fully synthetic, actually, dataset-

Ankit Gupta 1:10 Okay

Jovana Kondic 1:11 ... um, that can meet the demands of foundation models in terms of the size, in terms of the modality coverage. So besides the charts being presented as images, we also have the supporting metadata in the form of code, um, data tables, natural language summaries, and so on. Um, and what enables us to generate this dataset is leveraging the insight that charts are typically generated programmatically. So you have some kind of a plotting code that, uh, you, you know, give it your data, it renders a chart, and this,

Jovana Kondic 1:42 um, facilitates our data generation pipeline design. So it's a two-stage pipeline. Um, we start with a seed set of images that we use a VLM to translate into the plotting, the roughly corresponding plotting code, and then we perform iterative code augmentation with an LLM. Um, and this enables us to end up with these image and code pairs where we could use the underlying plotting code to generate, um, rounded supporting metadata in the form of data table summaries and so on.

Ankit Gupta 2:12 Okay.

Jovana Kondic 2:13 And we also have a quality filtering unit that ensures that our synthetic dataset meets certain quality.

Ankit Gupta 2:19 So you basically take an existing chart, and then you're generating code that makes the chart, and then a new prompt that tweaks that to make a new data point.

Jovana Kondic 2:27 Yeah. So let me actually show you. In the, in the next slide, it exactly visualizes what you were asking about. So we start with a very simple line chart.

Jovana Kondic 2:37 Um, we roughly translate it into code, and then these are the iterative augmentations. These all happen in the code space.

Ankit Gupta 2:43 Okay.

Jovana Kondic 2:43 I just rendered them so that you, you know, we can actually easily visualize what's going on under the hood.

Ankit Gupta 2:48 So you're actually... Okay. So you're, you're iterating in code space and then just, like, rendering out what chart it produces.

Jovana Kondic 2:53 Exactly.

Ankit Gupta 2:54 Cool.

Jovana Kondic 2:54 You just run the code, and it, and it executes, and it renders the image. That's exactly right. And so the resulting dataset is a million-scale dataset. We try to cover all the chart types that we could think of and different plotting libraries for diversity coverage also. Um, each sample is a tuple of plotting code, image, data table, natural language summary, and question and answer traces with, uh, reasoning. And then besides the kind of core synthetic dataset, we also provide additional subsets in the

Jovana Kondic 3:24 form of real-world curated data, human-annotated data, grounding, and [audio cuts out] data. And then the question is, well, how well does this work in practice, which we all [laughs] ultimately care about. So we fine-tune a range of open source models from under a billion parameter scale to seven billion

Jovana Kondic 3:39 parameters, and we actually find that across all the model families, we observe gains across all the different chart understanding tasks. And most excitingly, we enable a two billion parameter open source model to outperform, um, orders of magnitude larger GPT-4o,

Jovana Kondic 3:55 and we also find that the, uh, gains translates to real-world public benchmarks, even though this is all based on the training of the synthetic dataset.

Ankit Gupta 4:04 Makes sense. Very cool.

Jovana Kondic 4:05 And I'm also really excited to say that this dataset was used by IBM Research in their, uh, next generation development of vision-language models. So as ChartNet was, um, kind of a part of their core training mixture. Um, and on average, across, um, our own and public benchmarks on chart and table extraction, we found that the model trained on ChartNet outperformed leading peer open source models and then also, um, Claude Opus on these dedicated tasks. So it's like a four billion parameter versus we don't know how many parameters, but presumably more.

Ankit Gupta 4:39 Probably, yeah.

Jovana Kondic 4:40 Um-

Ankit Gupta 4:40 And, and that's using, um, presumably both the code that you're generating and those resultant chart images for the VLA model?

Jovana Kondic 4:48 Exactly. So the key gain, which is really kind of the take-home message that I wanna share here, is that, you know, these capabilities that we commonly associate with these massive frontier models, we could instead gain by training on a carefully curated dataset that has this multimodal coverage. So we have images, we have plotting code, and then we also have the supporting metadata in the form of text, data values, and so on.

Ankit Gupta 5:15 Cool. Nice. And so how can someone access this? Is it on Hugging Face?

Jovana Kondic 5:18 Yeah. So it's, uh, fully open source. Um, here are the QR codes that point to the CVPR paper and the dataset. We've had really good, um, community adoption so far, so we've had 50,000 downloads in the past few months it's been, since it's been released. It's actively curated. We keep expanding it. So yeah, we'd love to hear any thoughts in terms of where to improve, what to do next.

Ankit Gupta 5:36 Cool. Thanks so much. Appreciate it.

Jovana Kondic 5:38 Thank you so much.

Ankit Gupta 5:38 Thank you.

Jovana Kondic 5:38 Appreciate it.

Ankit Gupta 5:38 Thanks.