ChartNet: Training Vision-Language Models to Understand Charts

1. 0:00 – 0:07

   Intro

   1. AGAnkit Gupta0:00

      [upbeat music]

2. 0:07 – 0:28

   ChartNet overview: a synthetic, open-source dataset for chart understanding

   1. AGAnkit Gupta0:07

      We're here at YCML with Jovana, a PhD candidate at MIT. She's gonna tell us a little bit about her work that she presented at CVPR this year. Jovana, tell us a little about ChartNet.

   2. JKJovana Kondic0:16

      Thank you so much, Ankit. Um, so this is a foundation models data generation pipeline and dataset, uh, where we enable the next generation of vision-language models to do robust chart understanding.

   3. AGAnkit Gupta0:28

      Right.

3. 0:28 – 0:58

   Why charts are hard for VLMs: text parsing + numerical reasoning

   1. JKJovana Kondic0:28

      And this work was, uh, done jointly with MIT, IBM Watson AI Labs, and IBM Research. Um, so what makes chart understanding difficult for VLMs? Well, unlike natural images, um, the models have to understand the text, and they also have to engage in some kind of numerical reasoning.

   2. AGAnkit Gupta0:46

      So this is a chart, like a bar graph, for example, or something to that effect?

   3. JKJovana Kondic0:49

      A bar graph is the simplest one-

   4. AGAnkit Gupta0:50

      Okay

   5. JKJovana Kondic0:50

      ... but they can go really wild. You know, you can imagine a crazy scatter plot, a violin plot, any, any-

   6. AGAnkit Gupta0:56

      Figure one in a nature paper, basically-

   7. JKJovana Kondic0:58

      Exactly

   8. AGAnkit Gupta0:58

      ... type of thing. Yeah.

4. 0:58 – 1:11

   Design goals: comprehensive chart diversity plus rich multimodal metadata

   1. JKJovana Kondic0:58

      Exactly. So we're, we're, um, trying to, uh, deliver kind of the most comprehensive, largest to date, most diverse, yet open source, uh, and fully synthetic, actually, dataset-

   2. AGAnkit Gupta1:10

      Okay

5. 1:11 – 1:42

   Key insight: charts are programmatically generated

   1. JKJovana Kondic1:11

      ... um, that can meet the demands of foundation models in terms of the size, in terms of the modality coverage. So besides the charts being presented as images, we also have the supporting metadata in the form of code, um, data tables, natural language summaries, and so on. Um, and what enables us to generate this dataset is leveraging the insight that charts are typically generated programmatically. So you have some kind of a plotting code that, uh, you, you know, give it your data, it renders a chart, and this,

6. 1:42 – 2:19

   Two-stage pipeline: image-to-code seeding, then LLM-driven code augmentation

   1. JKJovana Kondic1:42

      um, facilitates our data generation pipeline design. So it's a two-stage pipeline. Um, we start with a seed set of images that we use a VLM to translate into the plotting, the roughly corresponding plotting code, and then we perform iterative code augmentation with an LLM. Um, and this enables us to end up with these image and code pairs where we could use the underlying plotting code to generate, um, rounded supporting metadata in the form of data table summaries and so on.

   2. AGAnkit Gupta2:12

      Okay.

   3. JKJovana Kondic2:13

      And we also have a quality filtering unit that ensures that our synthetic dataset meets certain quality.

7. 2:19 – 2:36

   Iterating in code space: rendering variations from a base chart

   1. AGAnkit Gupta2:19

      So you basically take an existing chart, and then you're generating code that makes the chart, and then a new prompt that tweaks that to make a new data point.

   2. JKJovana Kondic2:27

      Yeah. So let me actually show you. In the, in the next slide, it exactly visualizes what you were asking about. So we start with a very simple line chart.

8. 2:36 – 2:54

   Quality filtering: keeping synthetic data reliable

   1. JKJovana Kondic2:37

      Um, we roughly translate it into code, and then these are the iterative augmentations. These all happen in the code space.

   2. AGAnkit Gupta2:43

      Okay.

   3. JKJovana Kondic2:43

      I just rendered them so that you, you know, we can actually easily visualize what's going on under the hood.

   4. AGAnkit Gupta2:48

      So you're actually... Okay. So you're, you're iterating in code space and then just, like, rendering out what chart it produces.

   5. JKJovana Kondic2:53

      Exactly.

   6. AGAnkit Gupta2:54

      Cool.

9. 2:54 – 3:24

   What each sample contains: code, image, tables, summaries, and Q&A reasoning traces

   1. JKJovana Kondic2:54

      You just run the code, and it, and it executes, and it renders the image. That's exactly right. And so the resulting dataset is a million-scale dataset. We try to cover all the chart types that we could think of and different plotting libraries for diversity coverage also. Um, each sample is a tuple of plotting code, image, data table, natural language summary, and question and answer traces with, uh, reasoning. And then besides the kind of core synthetic dataset, we also provide additional subsets in the

10. 3:24 – 3:39

    Additional subsets: real-world curated data, human annotations, and grounding

    1. JKJovana Kondic3:24

       form of real-world curated data, human-annotated data, grounding, and [audio cuts out] data. And then the question is, well, how well does this work in practice, which we all [laughs] ultimately care about. So we fine-tune a range of open source models from under a billion parameter scale to seven billion

11. 3:39 – 3:55

    Fine-tuning results: consistent gains across model families and tasks

    1. JKJovana Kondic3:39

       parameters, and we actually find that across all the model families, we observe gains across all the different chart understanding tasks. And most excitingly, we enable a two billion parameter open source model to outperform, um, orders of magnitude larger GPT-4o,

12. 3:55 – 4:05

    Efficiency highlight: small open models outperform larger frontier models on chart tasks

    1. JKJovana Kondic3:55

       and we also find that the, uh, gains translates to real-world public benchmarks, even though this is all based on the training of the synthetic dataset.

    2. AGAnkit Gupta4:04

       Makes sense. Very cool.

13. 4:05 – 5:15

    Industry adoption: IBM uses ChartNet in VLM training mixture; strong benchmark performance

    1. JKJovana Kondic4:05

       And I'm also really excited to say that this dataset was used by IBM Research in their, uh, next generation development of vision-language models. So as ChartNet was, um, kind of a part of their core training mixture. Um, and on average, across, um, our own and public benchmarks on chart and table extraction, we found that the model trained on ChartNet outperformed leading peer open source models and then also, um, Claude Opus on these dedicated tasks. So it's like a four billion parameter versus we don't know how many parameters, but presumably more.

    2. AGAnkit Gupta4:39

       Probably, yeah.

    3. JKJovana Kondic4:40

       Um-

    4. AGAnkit Gupta4:40

       And, and that's using, um, presumably both the code that you're generating and those resultant chart images for the VLA model?

    5. JKJovana Kondic4:48

       Exactly. So the key gain, which is really kind of the take-home message that I wanna share here, is that, you know, these capabilities that we commonly associate with these massive frontier models, we could instead gain by training on a carefully curated dataset that has this multimodal coverage. So we have images, we have plotting code, and then we also have the supporting metadata in the form of text, data values, and so on.

14. 5:15 – 5:39

    Access and community: open-source release, Hugging Face availability, ongoing curation

    1. AGAnkit Gupta5:15

       Cool. Nice. And so how can someone access this? Is it on Hugging Face?

    2. JKJovana Kondic5:18

       Yeah. So it's, uh, fully open source. Um, here are the QR codes that point to the CVPR paper and the dataset. We've had really good, um, community adoption so far, so we've had 50,000 downloads in the past few months it's been, since it's been released. It's actively curated. We keep expanding it. So yeah, we'd love to hear any thoughts in terms of where to improve, what to do next.

    3. AGAnkit Gupta5:36

       Cool. Thanks so much. Appreciate it.

    4. JKJovana Kondic5:38

       Thank you so much.

    5. AGAnkit Gupta5:38

       Thank you.

    6. JKJovana Kondic5:38

       Appreciate it.

    7. AGAnkit Gupta5:38

       Thanks.

Episode duration: 5:39