The world of data engineering is changing rather quickly.
I’ve spent the better part of my 14-year career as a software engineer working with data, and one problem I’ve always found particularly interesting is extracting structured information from unstructured documents.
The problem statement is fairly straightforward: Given a document containing a mixture of text, tables, and images, how accurately can we extract meaningful, structured data for downstream processing?
It’s a surprisingly common problem across industries.
Take healthcare, for example. There’s an enormous amount of information buried in clinical publications, medical records, physician notes, and research reports. Before we can analyze any of it, we first need to extract and structure the information.
Traditionally, this involved a combination of OCR, NLP, domain-specific ML models, and a fair amount of custom engineering. Services like AWS Textract and Amazon Comprehend Medical helped simplify parts of this process, but building reliable extraction pipelines still required significant effort. Despite the best models and approaches, extracting information from images has been a very hard problem to solve, more so than the problem of extracting information from tables.
And then came Generative AI.
Multimodal LLMs have changed how we approach this problem. Give a model a text extractable document, specify the output schema, and ask it to extract the information.
What previously required multiple processing stages can, at least in principle, be accomplished with a single model call! It is absolutely fascinating that a complex task that usually involves humans can be automated to a great extent. I don’t believe one can ever take humans out of the loop, because anyone who uses LLMs cannot guarantee that the data is truly free of hallucinations, but what I’m excited about is that we can get most of the way there.
But there’s one aspect of document extraction that particularly interests me: What happens when the information we’re interested in isn’t written as text but represented visually?
Think of a scientific publication, most of the meaty information from countless experiments and runs are usually summarized in a few plots and charts that are meant for a human to consume and contextualize. Some examples are line graphs, scatter plots, box plots, or survival curves.
These visualizations are intuitive for humans. We can look at a graph and quickly understand trends, relationships, and approximate values. But extracting the underlying numerical data programmatically is a different problem altogether.
That got me thinking.
How good are modern multimodal LLMs at actually reading numerical data from charts?
And more importantly, could I put together a reasonable experiment to answer that question without spending more than $20?
I decided to build a small Python-based pipeline to test this.
The idea is simple.
Rather than manually collecting charts from research publications and creating ground-truth datasets, why not generate the charts myself?
I used Plotly to generate synthetic charts using known data. That gave me two things: the chart image to pass to an LLM and the underlying numerical data to evaluate its response against.
No manual annotation required.
The pipeline has four main components:
I created 30 different chart types, ranging from simple bar and line charts to more complex visualizations such as violin plots, 3D surface plots, stacked bars, and multi-series charts.
Each chart has a corresponding Pydantic schema representing the expected structured data.
The generated chart image is sent to an LLM along with instructions to extract its data into the specified Pydantic model.
I used Pydantic AI and its structured output capabilities for this. One thing I particularly liked was being able to switch between different model providers without rewriting the extraction logic.
Since I generated the charts, I already know what the correct numbers should be.
That makes it possible to compare the extracted data against the ground truth and calculate evaluation metrics.
I used Prefect to orchestrate the pipeline, with caching to avoid making unnecessary LLM calls when the chart and extraction schema hadn’t changed.
This was particularly important for keeping the experiment inexpensive. I wanted to be able to iterate on the evaluation logic without repeatedly paying for identical extractions.
Nothing particularly groundbreaking architecturally, but that’s also the point. The experiment uses fairly standard Python tooling to explore an interesting problem.
Here is a collage of all the different Plots/Charts that I generated/tested as part of the experiment.
One thing I quickly realized was that evaluating numerical extraction isn’t quite as straightforward as evaluating a classification model.
Suppose a chart contains a value of 42.0, and the LLM extracts 41.8.
Is that correct? Incorrect? Close enough?
The answer depends on what you’re trying to accomplish.
I ended up looking at two groups of metrics.
Extraction metrics: Precision, recall, and F1 measure how successfully the model identifies and returns the expected data points. I also calculated an accuracy metric.
Numerical error metrics: For matched values, I calculated RMSE (Root Mean Squared Error), MAPE (Mean Absolute Percentage Error), and Pearson correlation.
These capture different aspects of extraction quality. Finding the expected data points is not the same thing as extracting their numerical values precisely.
For some chart structures, I used heuristic evaluation. For more ambiguous cases, I also used LLM-based evaluation. This distinction becomes important when interpreting the results.
Repo Link: https://github.com/vik1124/Pexel I ran the experiment against 10 models across OpenAI, Anthropic, and an open-weight model accessed through OpenRouter.
All charts were evaluated at high resolution.
Here’s how they performed:
A few things stood out to me.
For several chart types — including line charts, pie charts, scatter plots, heatmaps, and waterfall charts — all evaluated models achieved an F1 score of 1.0. That’s impressive, even allowing for the fact that these were synthetic charts with a known structure. What is even more impressive is that for some of the chartypes likes Histograms and line charts the presence or absence of labels did not affect the overall performance across most models
It suggests that, at least for relatively straightforward visualizations, multimodal LLMs can do a very good job of recovering structured information.
But the story changes once the charts become more complicated.
Here are the five chart types with the lowest average F1 scores across models:
That’s quite a difference from the near-perfect results on simpler charts.
This makes intuitive sense. Reading a clearly labeled bar chart is a different task from reconstructing information represented in a violin plot or a three-dimensional surface.
It also highlights something important: The type of visualization matters just as much as the model you’re using.
There was also an interesting failure involving an area chart with Wind and Solar data series.
Three models received an F1 score of zero since they swapped the results of Wind and Solar data despite the chart having a legen.. Other models extracted the same chart successfully.
A model can potentially recover plausible numerical values and still associate them with the wrong series.
For downstream data analysis, that’s arguably more dangerous than a model simply failing to extract anything. I also looked at how precisely the models extracted numerical values using regression metrics.
Across the evaluations where matched numerical values were available, Pearson correlation was remarkably high for most models.
The model-level averages ranged from approximately 0.987 to nearly 1.0.
The best average MAPE in this experiment was approximately 0.09%, achieved by GPT-6 Astra, while Qwen3.8 27B had an average MAPE of approximately 1.10%.
Those are interesting results, but there’s a catch.
Regression metrics could only be calculated for 286 of the 300 evaluations. The remaining cases didn’t have enough matched values to calculate the metrics.
A model that completely misses a series may therefore have no regression error recorded for that missing data.
And Pearson correlation, by itself, doesn’t tell us whether the model extracted every data point or reproduced the absolute values correctly.
That’s why I think it’s useful to evaluate both extraction completeness and numerical fidelity rather than relying on one metric.
This was another interesting observation.
GPT-5.5 achieved the highest average F1 score at 0.951, but its average extraction latency was approximately 35.7 seconds.
Claude Opus 5 achieved an F1 of 0.912, with an average latency of just 11.2 seconds.
That’s roughly three times faster in this particular setup, with a difference of about four percentage points in F1.
Claude Sonnet 5 was similarly fast, although its extraction scores were lower.
Of course, API latency depends on provider infrastructure, model configuration, network conditions, and other factors. I’m not treating these results as universal performance benchmarks.
Still, it’s a useful reminder that choosing an LLM for a data pipeline is rarely just about maximizing accuracy.
Latency, cost, and error tolerance all matter.
One of my original constraints was keeping this experiment cheap.
I didn’t want to spend hundreds of dollars generating repeated results for a hobby project.
Based on the recorded token usage, the pricing assumptions in my notebook produced an estimated cost of approximately $4.98 across all 300 extraction calls.
That estimate uses October 2026 model pricing and an assumed 70% input / 30% output token split. It isn’t an exact API billing total, and it doesn’t include every possible expense associated with developing or evaluating the pipeline.
The individual model costs varied quite a bit.
For example, extracting data from all 30 chart types using GPT-6 Luna had an estimated cost of approximately $0.015, with an average F1 of 0.908. GPT-5.5 achieved the highest F1 of 0.951 at an estimated cost of $0.81 for the same 30 charts.
The absolute amounts are small because this was a small experiment. But the differences become relevant when thinking about extraction pipelines that might need to process thousands or millions of documents.
One additional point of note is that different models use a variable number of tokens to represent an image. As is clear from the plot above, Anthropic models use more tokens per plot compared to the Open AI ones, though the difference isn’t a lot, however Qwen which is a significantly smaller model uses twice the number of tokens to represent the image, which skews the cost calculation here since it turns out be more expensive to use in comparison to Luna.
Of course, it is a very different story if you end up having Qwen 3.8 27b parameter model running locally on your machine which will definitely make the smaller model much more attractive to use despite the slight ding in metrics
Before drawing too many conclusions, it’s worth acknowledging the limitations.
This was a hobby experiment, not an exhaustive benchmark. However I am tempted to turn it into one.
I tested one generated example per chart type against each model. I didn’t run multiple randomized trials, calculate confidence intervals, or systematically explore how changes in image resolution, prompt structure, and data density affect the results.
The charts were synthetically generated, which also means they don’t necessarily reflect the complexity of real-world scientific publications.
The aggregate F1 scores are useful for exploring the results of this experiment, but I wouldn’t use them to declare one model universally better than another at chart extraction. And I’m certainly not claiming that a multimodal LLM can reliably reconstruct all the numerical data from arbitrary scientific figures.
There’s a significant gap between a successful extraction in a controlled experiment and a production-grade data extraction pipeline.
I think there’s potential to expand this work.
A few directions I’d be interested in exploring are evaluating multiple randomly generated charts per type, testing how extraction accuracy changes with image resolution and data density, and comparing labeled versus unlabeled plots.
I’d also like to investigate the variance in results that happens when the same plot, prompt and LLM model results in changing values and associated metrics.
But I’m deliberately trying not to turn a small experiment into an enormous framework that would also cost a lot of money to run.
I do want to note that I would be curious to determine how LLM based extraction compares to a Human. As I’d imagine a human would struggle with the same plots that the LLM had issues with. I can’t imagine a human being able to accurately determine a value from an Area or Bubble chart with ease.
For now, I’m happy with what I learned. One of the things I find exciting about the current AI ecosystem is how accessible experimentation has become.
A few years ago, building a pipeline capable of extracting structured numerical data from arbitrary chart images would likely have involved computer vision models, custom OCR logic, extensive training data, and a substantial amount of engineering.
Today, I can put together an experiment using Plotly, Pydantic AI, Prefect, and a collection of multimodal LLM APIs.
That doesn’t mean the problem is solved.
It means the barrier to experimenting with solutions has become dramatically lower.
As a data engineer, I find that particularly interesting because extraction is only the first step. We still need to validate the extracted information, handle errors, maintain data lineage, and make sure the resulting structured data is trustworthy enough for downstream use.
Making unstructured data accessible is getting easier. Making it reliable is still an engineering problem.
And that’s probably the part I’m most interested in exploring next.
P.S — An interesting takeway from this blog for those interested in protecting their data would be to use non-standard visualizations like Area or Violin plots if you want to make sure that a bot doesn’t scrape your hard work away ! Note that this blog post does not use esoteric plots to present the results :)
Tools used: Python, Plotly, Pydantic, Pydantic AI, Prefect, Pandas, and multimodal LLM APIs.
Repo: https://github.com/vik1124/Pexel
Rethinking Data Extraction in the Age of AI was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.