{"slug": "pt-br-typed-decision-benchmark-12-external-models-and-an-open-evaluation-guide", "title": "PT-BR typed-decision benchmark: 12 external models and an open evaluation/submission guide", "summary": "The maintainers of the PT-BR typed-decisions benchmark published a completed comparison of 12 external models on 5,196 test items across seven Brazilian Portuguese tasks, scored by mean balanced accuracy with shuffled option order and bootstrap confidence intervals. The benchmark's evaluation and submission guide, built on the research/llm_bridge.py and research/validate_submission.py scripts using Python's standard library, requires a qualifying report to cover all seven tasks and 5,196 items with zero request failures, and the validator recomputes scores without proving which checkpoint produced the answers. The maintainers excluded their own Saracura checkpoints from the main ranking because both were fine-tuned on the benchmark tasks, displaying their results separately among 14 preserved completed reports.", "body_md": "We maintain the **PT-BR typed-decisions benchmark** and have published a completed comparison of **12 external models on 5,196 test items across seven Brazilian Portuguese tasks**. Pinned checkpoint revisions, per-task scores and full reports are linked from the leaderboard.\n\nEach model chooses a typed answer from supplied alternatives. The headline metric is mean balanced accuracy across seven tasks, with shuffled option order and bootstrap confidence intervals. Native decision models use their native runtimes; general LLMs in the completed round use constrained-letter readout. Runtime and preprocessing settings are documented. Execution times on different hardware are not a controlled speed comparison.\n\nThis measures the published tasks, not general Portuguese chat quality or performance in every production workflow.\n\nThe **[evaluation and submission guide (English / português)](https://github.com/felhen-ai/saracura/blob/main/research/evaluation-guide.md)** contains the commands and an isolated CPU example.\n\n`research/llm_bridge.py`. Arbitrary architectures and hosted APIs are not automatically supported.`research/validate_submission.py`. A qualifying report has all seven tasks, 5,196 items, zero request failures and passes validation. The validator recomputes the scores; it does not prove which checkpoint produced the answers.\nNo dataset preparation, manual labeling, model training or private-track evaluation is required. This is a command-line workflow, and participants supply their own compute. The evaluator and validator use Python’s standard library; model serving needs its own ML packages.\n\nUse either the **dataset’s Community discussion** or the **GitHub submission form**, both linked in the guide. Include:\n\nDo not attach credentials or private data. Submissions are reviewed before inclusion; nothing is listed automatically. Report validation and independent reproduction are separate steps.\n\nWe also develop Saracura. Its two checkpoints were fine-tuned on the benchmark tasks, so we exclude them from the main ranking and display their results separately. All 14 completed reports are preserved. For the other models, the absence of identified task-specific tuning does not establish that their upstream training data excluded the tasks.\n\nFeedback on the protocol, runtime configurations or participation guide is welcome. A concrete correction is already useful; there is no need to rerun a model just to point out a configuration issue.", "url": "https://wpnews.pro/news/pt-br-typed-decision-benchmark-12-external-models-and-an-open-evaluation-guide", "canonical_source": "https://discuss.huggingface.co/t/pt-br-typed-decision-benchmark-12-external-models-and-an-open-evaluation-submission-guide/191316#post_1", "published_at": "2026-11-01 17:48:18+00:00", "updated_at": "2026-11-01 17:52:02.627789+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "natural-language-processing"], "entities": ["PT-BR typed-decisions benchmark", "Saracura", "research/llm_bridge.py", "research/validate_submission.py", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/pt-br-typed-decision-benchmark-12-external-models-and-an-open-evaluation-guide", "markdown": "https://wpnews.pro/news/pt-br-typed-decision-benchmark-12-external-models-and-an-open-evaluation-guide.md", "text": "https://wpnews.pro/news/pt-br-typed-decision-benchmark-12-external-models-and-an-open-evaluation-guide.txt", "jsonld": "https://wpnews.pro/news/pt-br-typed-decision-benchmark-12-external-models-and-an-open-evaluation-guide.jsonld"}}