Hi HF team,
I would like to register my dataset thamilvendhan/signalbench as a Benchmark so its leaderboard can aggregate community evaluation results across the Hugging Face Hub.
Dataset: thamilvendhan/signalbench · Datasets at Hugging Face The eval.yaml file has been added to the repository root and was validated during upload.
SignalBench introduces In Band Signal Compliance (IBSC), a benchmark with a single metric called Signal Response Correctness. It evaluates both over compliance, such as prompt injection, and under compliance, including temporal blindness, access denial, memory label, and bot policy failures in LLM agents. The dataset is licensed under Apache 2.0. The DOI is 10.5281/zenodo.21223956, and the code is available at GitHub - mthamil107/signal-compliance: In-Band Signal Compliance (IBSC): one metric for prompt injection and temporal blindness. signalbench benchmark + 21-model leaderboard, action-based grading, Apache-2.0. · GitHub.
The benchmark defines the following tasks: src for the overall benchmark, along with five task families: time, access_deny, memory_label, injection, and bot_policy.
Could you please add it to the Benchmark allow list? I would be happy to update the eval.yaml file if any schema changes are needed, including completing the inspect ai field_spec, solvers, or scorers.
Thank you!