{"slug": "request-to-register-dedeprogames-phone-tool-bench-as-an-official-benchmark-on", "title": "Request to register DedeProGames/phone-tool-bench as an official benchmark on the Hub", "summary": "A developer posting as DedeProGames has asked Hugging Face to register DedeProGames/phone-tool-bench as an official benchmark on the Hub, submitting a held-out phone tool-calling benchmark of 520 items spanning English, Portuguese, Spanish and German over the full 108-tool phone catalog from LiquidAI/phone-assistant-tools. Scoring is deterministic exact match of the predicted tool-call list with no LLM judge, and the headline metric is macro exact match across seven item categories (single, multi, reasoning, lookalike, partial, no_tool, clarify); the eval.yaml at the repository root was validated at push time with tasks phone_tool_bench, config default and split test. The benchmark ships its own phonebench harness with unit tests plus tools.json and scoring.json, and a first result for klein-ml/KV1-1.2B at 12.5% macro exact match is documented with full traces in DedeProGames/kv1-1.2B-phone-tool-bench-results.", "body_md": "Hello HF team,\n\nI would like to request the registration of my dataset as a benchmark on the Hugging Face Hub:\n\nDataset: DedeProGames/phone-tool-bench  ← (link 1: [DedeProGames/phone-tool-bench · Datasets at Hugging Face](https://huggingface.co/datasets/DedeProGames/phone-tool-bench))\n\nPhoneToolBench is a held-out benchmark for phone tool calling: 520 items across\n\nEnglish, Portuguese, Spanish and German, over the full 108-tool phone catalog from\n\nLiquidAI/phone-assistant-tools. Scoring is deterministic exact match of the predicted\n\ntool-call list (no LLM judge); the headline metric is macro exact match over the\n\nseven item categories (single, multi, reasoning, lookalike, partial, no_tool, clarify).\n\nThe eval.yaml file is already at the repository root and was validated at push time\n\n(tasks: phone_tool_bench, config: default, split: test). The benchmark ships its own\n\nharness (phonebench) with unit tests, plus tools.json and scoring.json for\n\nreproduction. A first result (klein-ml/KV1-1.2B, macro 12.5%) is documented with full\n\ntraces here:  ← (link 2: [DedeProGames/kv1-1.2B-phone-tool-bench-results · Datasets at Hugging Face](https://huggingface.co/datasets/DedeProGames/kv1-1.2B-phone-tool-bench-results))\n\nCould you please add it to the benchmark allow-list? Happy to adjust the eval.yaml\n\nif any schema changes are needed.\n\nThank you!", "url": "https://wpnews.pro/news/request-to-register-dedeprogames-phone-tool-bench-as-an-official-benchmark-on", "canonical_source": "https://discuss.huggingface.co/t/request-to-register-dedeprogames-phone-tool-bench-as-an-official-benchmark-on-the-hub/192104#post_1", "published_at": "2026-11-18 17:21:49+00:00", "updated_at": "2026-11-18 17:43:39.270706+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents", "ai-research"], "entities": ["Hugging Face", "DedeProGames", "DedeProGames/phone-tool-bench", "LiquidAI", "LiquidAI/phone-assistant-tools", "klein-ml/KV1-1.2B", "phonebench", "DedeProGames/kv1-1.2B-phone-tool-bench-results"], "also_reported_by": [], "takeaway": "PhoneToolBench scores tool calling by deterministic exact match of the predicted tool-call list over 520 items in four languages and 108 tools, reporting macro exact match across seven categories; the first model tested, klein-ml/KV1-1.2B, reached 12.5% macro exact match.", "recommendation": {"advice": "Score tool-calling agents with deterministic exact match on the full predicted call list and report macro averages across categories like single, multi, lookalike, partial, no_tool and clarify, as PhoneToolBench does, rather than relying on an LLM judge.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/request-to-register-dedeprogames-phone-tool-bench-as-an-official-benchmark-on", "markdown": "https://wpnews.pro/news/request-to-register-dedeprogames-phone-tool-bench-as-an-official-benchmark-on.md", "text": "https://wpnews.pro/news/request-to-register-dedeprogames-phone-tool-bench-as-an-official-benchmark-on.txt", "jsonld": "https://wpnews.pro/news/request-to-register-dedeprogames-phone-tool-bench-as-an-official-benchmark-on.jsonld"}}