# PT-BR typed-decision benchmark: 12 external models and an open evaluation/submission guide

> Source: <https://discuss.huggingface.co/t/pt-br-typed-decision-benchmark-12-external-models-and-an-open-evaluation-submission-guide/191316#post_1>
> Published: 2026-11-01 17:48:18+00:00

We maintain the **PT-BR typed-decisions benchmark** and have published a completed comparison of **12 external models on 5,196 test items across seven Brazilian Portuguese tasks**. Pinned checkpoint revisions, per-task scores and full reports are linked from the leaderboard.

Each model chooses a typed answer from supplied alternatives. The headline metric is mean balanced accuracy across seven tasks, with shuffled option order and bootstrap confidence intervals. Native decision models use their native runtimes; general LLMs in the completed round use constrained-letter readout. Runtime and preprocessing settings are documented. Execution times on different hardware are not a controlled speed comparison.

This measures the published tasks, not general Portuguese chat quality or performance in every production workflow.

The **[evaluation and submission guide (English / português)](https://github.com/felhen-ai/saracura/blob/main/research/evaluation-guide.md)** contains the commands and an isolated CPU example.

`research/llm_bridge.py`. Arbitrary architectures and hosted APIs are not automatically supported.`research/validate_submission.py`. A qualifying report has all seven tasks, 5,196 items, zero request failures and passes validation. The validator recomputes the scores; it does not prove which checkpoint produced the answers.
No dataset preparation, manual labeling, model training or private-track evaluation is required. This is a command-line workflow, and participants supply their own compute. The evaluator and validator use Python’s standard library; model serving needs its own ML packages.

Use either the **dataset’s Community discussion** or the **GitHub submission form**, both linked in the guide. Include:

Do not attach credentials or private data. Submissions are reviewed before inclusion; nothing is listed automatically. Report validation and independent reproduction are separate steps.

We also develop Saracura. Its two checkpoints were fine-tuned on the benchmark tasks, so we exclude them from the main ranking and display their results separately. All 14 completed reports are preserved. For the other models, the absence of identified task-specific tuning does not establish that their upstream training data excluded the tasks.

Feedback on the protocol, runtime configurations or participation guide is welcome. A concrete correction is already useful; there is no need to rerun a model just to point out a configuration issue.