# Give to AI Formulas from Charts

> Source: <https://github.com/BorisYamp/plotparse>
> Published: 2026-09-13 09:52:22+00:00

Finds charts in PDF files and recovers the analytical formula of every curve on them.

No neural networks anywhere: the whole pipeline is deterministic, reproducible and explainable — every number in the output can be traced back to a specific geometric feature of the page.

``` bash
$ analyze_pdf paper.pdf

Page 1 — source: vector PDF graphics
  Chart detected, confidence 0.96.
  X axis: "X", linear scale, range 0…10, 6 ticks, calibration R² 1.0000
  Y axis: "Y", linear scale, range 0…50, 6 ticks, calibration R² 1.0000
  Series 1 "linear A" (line, blue, 200 points), X ∈ [0; 10], Y ∈ [0.9868; 20.99]
     FORMULA: y = 2·x + 0.9868
     model "linear", R² = 1.00000, RMSE = 5.774e-13, 2 params
  Series 2 "quad B" (line, red, 200 points), X ∈ [0; 10], Y ∈ [-0.01318; 49.99]
     FORMULA: y = 0.5·x^2 + 0.0002635·x - 0.009103
     model "parabola", R² = 1.00000, RMSE = 0.002764, 3 params
  Series 3 "sine C" (line, green, 200 points), X ∈ [0; 10], Y ∈ [16.99; 32.98]
     FORMULA: y = 7.993·sin(0.8·x - 0.0007325) + 24.99
     model "sine", R² = 1.00000, RMSE = 0.007426, 4 params
```

Text extracted *from the PDF* (axis titles, curve labels) is of course reproduced in
whatever language the document uses.

1. **Decides whether the page contains a chart at all** — weighted score over: two long
perpendicular lines, short tick strokes touching them, numeric labels along the axes that
fall on a straight line under regression, grid lines, and a polyline with many nodes
inside the axes box. The decisive feature is the*linearity of the labels* : for random
text the regression R² is low, for a real axis it is ≈ 1.
2. **Calibrates the axes** — pixel → value regression with iterative worst-point rejection.
A logarithmic-scale hypothesis is tested separately (same regression over`log10(value)` );
this matters more than it sounds, because a straight line on a semi-log axis*is* an
exponential, and without detecting the scale the formula comes out meaningless.
3. **Extracts every curve** and converts it to data coordinates.
4. **Fits a formula** — 11 models, winner picked by parsimony/AICc rather than by max R².

Two independent front-ends feed step 3, chosen automatically:

- **Vector** (`src/vector.cpp` ) — the main path. In a PDF a chart is stored as paths and
text, so curve coordinates are read out of the file exactly, with no computer vision.
Accuracy: fractions of a percent.
- **Raster** (`src/raster.cpp` ) — for scans and embedded images. Axes are found by
morphological opening with a long kernel, labels are read with Tesseract, the curve is
isolated by saturation/hue (for black curves: dark pixels minus long straight lines,
i.e. minus grid and frame), then a per-column median gives the trace. Measured accuracy
on the test scan: ≈ 0.3 % of the range.

When several curves share a chart, each series gets its own label:

- **legend** — if a short coloured swatch sits immediately left of a text run, the label is
assigned to the series*of that colour* , not to the geometrically nearest curve (a legend
usually sits in a corner, so "nearest curve" would hand every entry to whichever curve
happens to pass by it);
- **label next to the curve** — otherwise the nearest series is taken, within 15 % of the
shorter side of the plot box.

Matching is one-to-one and greedy by increasing cost. Text runs already consumed as axis
numbers, axis titles or the chart title are excluded from the candidates. Vector branch
only — see *Limitations*.

11 models: polynomials of degree 1–5, exponential, power, logarithm, sine, logistic,
Gaussian, hyperbola, square root. Each gets a meaningful initial guess (log-linearisation
for exponential and power, FFT peak plus mean-level crossing count for the sine, half-maximum
position for the logistic) — with `p0 = {1,1,1}` almost nothing converges.

The winner is **not** the maximum R². By R² a high-degree polynomial always wins, because it
eats the noise and the discretisation error. The rules, in order:

1. if several models reach R² ≥ 0.9999 — the one with fewer parameters wins;
2. otherwise, among models whose RSS is no worse than 1.6× the best — again fewest parameters;
3. inside that group — by AICc.

On synthetic data (11 dependency types × 2 noise levels) this rule scores 22/22.

Dependencies (Ubuntu 24.04):

```
apt-get install cmake ninja-build pkg-config \
  libmupdf-dev mupdf-tools libeigen3-dev libceres-dev \
  libgflags-dev libgoogle-glog-dev \
  libfreetype-dev libjpeg-dev libjbig2dec0-dev libopenjp2-7-dev \
  libharfbuzz-dev libgumbo-dev libmujs-dev \
  libopencv-dev libtesseract-dev tesseract-ocr tesseract-ocr-rus
```

`tesseract-ocr-rus` is only needed to read Cyrillic axis titles; everything else works
without it.

```
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
```

Produces `build/analyze_pdf`.

```
./build/analyze_pdf chart.pdf                 # human-readable report
./build/analyze_pdf chart.pdf --json          # machine-readable
./build/analyze_pdf scan.pdf --csv out/       # also dump curve points as CSV
./build/analyze_pdf chart.pdf --force raster  # force the CV path
./build/analyze_pdf chart.pdf --force vector  # force the vector path
./build/analyze_pdf scan.pdf --dpi 300        # render resolution for the raster path
```

Tesseract prints its own diagnostics to stderr; stdout stays clean, so `--json` can be piped
directly into a parser.

| Path | Role | Libraries | 
|---|---|---|
| `src/calib.cpp` | number parsing, axis calibration, log scale, minus-sign recovery | Eigen | 
| `src/pdf_backend.cpp` | MuPDF wrapper: paths, text runs, page rendering | MuPDF | 
| `src/vector.cpp` | axes, ticks, series, labels, "is this a chart" score | — | 
| `src/raster.cpp` | CV + OCR path | OpenCV, Tesseract | 
| `src/fit.cpp` | model library, initial guesses, parsimony/AICc selection | Eigen, Ceres | 
| `src/report.cpp` | report text, vector→raster fallback orchestration | — | 
| `src/main.cpp` | CLI | — | 
| `python-reference/` | the original Python implementation this was ported from (docs in Russian) | — | 

`pdf_backend.hpp` and `raster.hpp` are the only places that know about MuPDF and
OpenCV/Tesseract respectively; the rest of the code works with their plain structs
(`PageContent`, `RawPath`, `TextSpan`, `PdfDocument::Raster`).

`reference/` holds the fixture PDFs plus two recorded outputs:

- `expected_cpp.txt` — what this implementation prints on all nine fixtures. Regenerate and
diff it to catch regressions.
- `expected.txt` — the original Python implementation's output, in Russian. Kept for
provenance; useful for comparing*numbers* , not text.

```
for f in exp sin logy scatter_parabola power_en no_chart raster_exp multi_text multi_legend; do
  echo "########## $f.pdf"; ./build/analyze_pdf reference/$f.pdf 2>/dev/null; echo
done > /tmp/out.txt
diff /tmp/out.txt reference/expected_cpp.txt && echo "no regressions"
```

| File | Ground truth | Expected result | 
|---|---|---|
| `exp.pdf` | `y = 2e^{0.5x} − 1` | exponential, R² = 1.0 | 
| `sin.pdf` | `y = 4sin(1.3x + 0.4) + 2` | sine, R² ≈ 1.0 | 
| `logy.pdf` | `y = 5e^{0.8x}` , log Y axis | **Y axis = log** , exponential | 
| `scatter_parabola.pdf` | `y = 3x² − 2x + 7` , noise σ=2 | parabola, R² ≈ 0.994 | 
| `power_en.pdf` | `y = 1.7x^{2.3}` | power (NOT a polynomial) | 
| `raster_exp.pdf` | same as exp, but rasterised | raster path, R² ≈ 0.99999 | 
| `no_chart.pdf` | text and a table | no chart detected, score 0 | 
| `multi_text.pdf` | 3 curves labelled `A` /`B` /`C` next to each | 3 series, labels attached | 
| `multi_legend.pdf` | same 3 curves, labelled by legend | 3 series, labels via swatch colour | 

- **Curve labels are vector-only.** The raster path does not look for them yet — that needs
OCR over the whole plot area rather than the narrow strips next to the axes, and it would
keep catching the grid and the curves themselves.
- **Same-coloured overlapping curves are not separated** — they merge into one series.
- **Closed and parametric curves** (circle, hysteresis loop) are detected, but a`y(x)` formula is meaningless for them; the report flags the X-ambiguity.
- **Bar and pie charts** are recognised as "a chart", but the dependency model does not
apply to them.
- **Complex functions outside the 11-model library** (sums of harmonics, damped oscillation,
piecewise definitions) are not recognised as such — the tool still reports the best of the
11, just with a lower R². There is no explicit "I don't know this shape" signal beyond
that R².
- **Extrapolation past the plotted range is unreliable** — the model was only fitted inside
the visible window.
- **Cyrillic in labels.** matplotlib writes PDFs with Type3 fonts that carry no ToUnicode
map, so the text layer returns garbage for Cyrillic. Handled by re-reading the title with
OCR off a page render, which needs`tesseract-ocr-rus` .
- **Lost minus sign.** The same Type3 fonts often drop the minus glyph, so an axis`−4 −2 0 2 4` extracts as`4 2 0 2 4` . Handled by testing "first/last k labels are
negative" hypotheses and keeping the best R².

Both cost real debugging time and are not obvious from the MuPDF docs.

1. **Do not flip the page coordinates yourself.**`fz_bound_page` /`fz_run_page` already
hand you a page space whose origin is top-left with y growing downwards — unlike the raw
coordinates inside`fz_path` , which`fz_path_walker` sees*before* the`ctm` is applied.
The transform you pass should therefore be a pure shift,`fz_make_matrix(1,0,0,1,-x0,-y0)` ; adding a flip mirrors the whole page.
2. **Merge `fill_path` + `stroke_path` for the same path.** The PDF operator`B` (fill*and* stroke) reaches an`fz_device` as two separate callbacks with the same`fz_path*` and the same`ctm` . PyMuPDF's`get_drawings()` reports this as a single object
(`type: "fs"` , with`fill` and`color` together). Without merging them, the axes frame is
counted twice and the grid-line/tick statistics in the detection score come out inflated.

- **Solver** — Ceres instead of`scipy.optimize.curve_fit` . The winning model's formula and
R² match the reference byte-for-byte almost everywhere; 2nd/3rd place in the "alternatives"
list occasionally differs, because on deliberately bad models (a Gaussian fitted over a
sine) Ceres converges to a different local optimum than scipy's LM. This never changed the
winner in testing.
- **`--force vector` really means vector-only.** In the original,`force` was only branched
on for`"raster"` ;`"vector"` did not disable the automatic raster fallback, which
contradicted its own CLI help.
- **Per-curve labels** — new, the original identified series only by index and colour.
- **The report is in English** (the original printed Russian). The translation was verified
by hashing every numeric token in the output before and after: identical, so only wording
changed. As a side effect the report can no longer be byte-compared against the Python
reference — hence the separate`reference/expected_cpp.txt` baseline.

AGPL-3.0-or-later — see [LICENSE](/BorisYamp/plotparse/blob/main/LICENSE).

This is dictated by the dependency on **MuPDF**, which is AGPL (or a paid commercial licence
from Artifex). Everything else here — Eigen (MPL2), Ceres (BSD), OpenCV (Apache-2.0),
Tesseract (Apache-2.0) — is compatible with a more permissive licence. If you need one,
replace the MuPDF backend with PDFium (BSD): the PDF-specific code is confined to
`src/pdf_backend.cpp` behind the interface in `include/plotparse/pdf_backend.hpp`.