YC Root AccessChartNet: Training Vision-Language Models to Understand Charts
At a glance
WHAT IT’S REALLY ABOUT
ChartNet generates massive synthetic chart data to boost VLM reasoning
- ChartNet targets chart understanding, which is hard for vision-language models because it requires accurate text reading plus numerical reasoning beyond typical natural-image perception.
- The dataset is generated via a two-stage pipeline: a VLM converts seed chart images into approximate plotting code, then an LLM iteratively augments the code to create diverse new chart instances.
- Each data point is multimodal—plotting code, rendered chart image, data table, natural-language summary, and QA traces with reasoning—enabling richer supervision than image-only training.
- A quality-filtering component is used to keep synthetic samples reliable, while additional subsets include real-world curated and human-annotated data for grounding.
- Fine-tuning open-source models (sub-1B to 7B) on ChartNet improves performance across chart tasks, with reports of smaller models outperforming much larger frontier systems on chart-specific benchmarks and transferring to public real-world evaluations.
IDEAS WORTH REMEMBERING
5 ideasChart understanding demands capabilities VLMs often lack by default.
Unlike natural images, charts require OCR-like text recognition and structured numerical reasoning (e.g., reading values, comparing trends), so general vision pretraining is insufficient.
Programmatic chart creation enables scalable, controllable synthetic supervision.
Because charts are typically rendered from plotting code, you can generate consistent image–code–data alignments at scale and systematically vary styles, chart types, and layouts.
Generating variations in code space is a powerful form of augmentation.
Instead of perturbing pixels, ChartNet edits plotting code with an LLM, then re-renders; this produces semantically meaningful diversity (new axes, legends, chart types, data ranges) while preserving correctness.
Multimodal training signals can substitute for sheer model size on specialized tasks.
The project’s core claim is that carefully curated multimodal coverage (image + code + tables + text + reasoning traces) can unlock “frontier-like” chart skills in smaller open models.
Quality filtering is essential for synthetic datasets to be useful.
Even with code-driven generation, translation and augmentation can introduce mismatches; a dedicated filtering stage helps keep pairs coherent and reduces training on noisy supervision.
WORDS WORTH SAVING
5 quotesso this is a foundation models data generation pipeline and dataset, uh, where we enable the next generation of vision-language models to do robust chart understanding.
— Jovana Kondic
unlike natural images, um, the models have to understand the text, and they also have to engage in some kind of numerical reasoning.
— Jovana Kondic
we're, we're, um, trying to, uh, deliver kind of the most comprehensive, largest to date, most diverse, yet open source, uh, and fully synthetic, actually, dataset-
— Jovana Kondic
what enables us to generate this dataset is leveraging the insight that charts are typically generated programmatically.
— Jovana Kondic
the key gain, which is really kind of the take-home message that I wanna share here, is that, you know, these capabilities that we commonly associate with these massive frontier models, we could instead gain by training on a carefully curated dataset that has this multimodal coverage.
— Jovana Kondic
High quality AI-generated summary created from speaker-labeled transcript.
