PT-BR typed-decision benchmark: 12 external models and an open evaluation/submission guide

The maintainers of the PT-BR typed-decisions benchmark published a completed comparison of 12 external models on 5,196 test items across seven Brazilian Portuguese tasks, scored by mean balanced accuracy with shuffled option order and bootstrap confidence intervals. The benchmark's evaluation and submission guide, built on the research/llm_bridge.py and research/validate_submission.py scripts using Python's standard library, requires a qualifying report to cover all seven tasks and 5,196 items with zero request failures, and the validator recomputes scores without proving which checkpoint produced the answers. The maintainers excluded their own Saracura checkpoints from the main ranking because both were fine-tuned on the benchmark tasks, displaying their results separately among 14 preserved completed reports.

We maintain the PT-BR typed-decisions benchmark and have published a completed comparison of 12 external models on 5,196 test items across seven Brazilian Portuguese tasks . Pinned checkpoint revisions, per-task scores and full reports are linked from the leaderboard. Each model chooses a typed answer from supplied alternatives. The headline metric is mean balanced accuracy across seven tasks, with shuffled option order and bootstrap confidence intervals. Native decision models use their native runtimes; general LLMs in the completed round use constrained-letter readout. Runtime and preprocessing settings are documented. Execution times on different hardware are not a controlled speed comparison. This measures the published tasks, not general Portuguese chat quality or performance in every production workflow. The evaluation and submission guide English / português https://github.com/felhen-ai/saracura/blob/main/research/evaluation-guide.md contains the commands and an isolated CPU example. research/llm bridge.py . Arbitrary architectures and hosted APIs are not automatically supported. research/validate submission.py . A qualifying report has all seven tasks, 5,196 items, zero request failures and passes validation. The validator recomputes the scores; it does not prove which checkpoint produced the answers. No dataset preparation, manual labeling, model training or private-track evaluation is required. This is a command-line workflow, and participants supply their own compute. The evaluator and validator use Python’s standard library; model serving needs its own ML packages. Use either the dataset’s Community discussion or the GitHub submission form , both linked in the guide. Include: Do not attach credentials or private data. Submissions are reviewed before inclusion; nothing is listed automatically. Report validation and independent reproduction are separate steps. We also develop Saracura. Its two checkpoints were fine-tuned on the benchmark tasks, so we exclude them from the main ranking and display their results separately. All 14 completed reports are preserved. For the other models, the absence of identified task-specific tuning does not establish that their upstream training data excluded the tasks. Feedback on the protocol, runtime configurations or participation guide is welcome. A concrete correction is already useful; there is no need to rerun a model just to point out a configuration issue.