# 8 Chinese AI models on the same Pi coding-agent task: what we measured

> Source: <https://dev.to/vancine-fan/8-chinese-ai-models-on-the-same-pi-coding-agent-task-what-we-measured-oin>
> Published: 2026-08-28 16:14:44+00:00

Most model comparisons try to answer a question that is too broad: “Which model is best?” We wanted a smaller, reproducible question instead:

What happens when eight current Chinese AI models receive the same coding task through the same agent?

We used Pi as the coding agent and ran one isolated JavaScript task across these models:

Each run started from its own copy of the fixture. The test directory was kept unchanged, the work directory was checked for unexpected files, and raw run evidence was stored separately from the task workspace. The same Pi provider configuration and task contract were used for every model.

This is deliberately a narrow test. It does not measure architecture work, long-horizon debugging, frontend judgment, or performance on a real production repository.

All eight models completed the task successfully. Across the complete run we recorded:

The public page includes the model-by-model table, runtime and token measurements, the Pi configuration, methodology notes, and a downloadable JSON file:

[View the full benchmark and data](https://vancine.com/coding-agent-benchmark?utm_source=devto&utm_medium=community&utm_campaign=pi_benchmark_launch)

A small benchmark cannot tell you which model is generally better. It can still answer useful operational questions:

For this task, the answer to the first question was yes for all eight models. The differences are in the detailed run data, not a winner label.

The total above is the audited amount recorded for these eight runs, not a forecast for arbitrary coding work. Agent cost depends heavily on task length, retries, context growth, and tool behavior. A real repository can be much more expensive than this small fixture.

The benchmark page includes the Pi setup and downloadable structured results. If you repeat it, keep the task, tests, agent version, model IDs, timeout, and evidence rules fixed. Otherwise you are comparing different experiments.

Disclosure: I operate Vancine, the OpenAI-compatible API used for these runs. The page is published as product evidence, and the result should not be read as a general model ranking.

I would especially value feedback on the harness and on what the next coding-agent task should test.