{"slug": "mca-open-source-neuro-symbolic-wrapper-that-teaches-7b-llms-to-challenge-false", "title": "MCA: open-source neuro-symbolic wrapper that teaches 7B LLMs to challenge false premises, explain paradoxes and abstain (code + dataset + Space + 2×T4 benchmark)", "summary": "Developer Mati83mon opened pull request #6 on the Mati83mon/MCA GitHub repository, adding an audit mode that runs every layer of the open-source neuro-symbolic wrapper on every item without changing the decision, plus a per-rule verdict table for uncensored threshold calibration. The change adds `audit_all_layers=True`, a `DecisionController.rule_verdicts()` method tested to reproduce `decide()` on all 300 stored rows, and `benchmarks/rule_table.py` for offline grid-search of thresholds tuned on 100 items and evaluated once on held-out probes. The author disclosed that the entropy threshold τ = 0.6 was a prior default and that the R8b evidence-answer thresholds were chosen on a censored sample covering only 20–26 of 100 items per model, and said an uncensored audit run across 3 models is underway on Kaggle with a 24-item held-out set (6 control facts) too small to detect small over-refusal differences.", "body_md": "Thanks — this is exactly right, and I checked it against the stored telemetry before replying.\n\n**Where it holds:** R8 is censored. The slow path (entropy sampling) is cancelled as soon as the fast path decides, so entropy was recorded for only 20–26 of the 100 items per model. The entropy threshold itself (τ = 0.6) was a prior default, not fitted to these logs — but the thresholds of an “evidence answer” rule (R8b) I added a few hours ago after a Space test were chosen by inspecting exactly that censored sample.\n\n**Where it’s partly already covered:** the fast-path rules (R1–R7, incl. R5) store every premise verdict, so their independent verdicts are recoverable for all items. Standalone, R5 fires on 6/25 control facts for Llama-3 and 0/25 for Qwen2.5 and DeepSeek-R1.\n\n**What I changed** ([Audit mode + per-rule verdict table for uncensored threshold calibration by Mati83mon · Pull Request #6 · Mati83mon/MCA · GitHub](https://github.com/Mati83mon/MCA/pull/6)):\n\n`audit_all_layers=True` runs every layer on every item without changing the decision`DecisionController.rule_verdicts()` returns each rule’s own verdict (tested to reproduce `decide()` on all 300 stored rows)`benchmarks/rule_table.py` builds the per-item rule table and grid-searches the thresholds offline — tune on the 100, evaluate the chosen config exactly once on the held-out probes\nAn uncensored audit run (3 models, 100 items + held-out) is running on Kaggle now; I’ll post the per-rule table and the held-out result here, whichever way it comes out.\n\nOne caveat on my side: the held-out set is only 24 items (6 control facts), so it can show a calibration that clearly fails to travel, but not a small over-refusal difference. I’ll say so with the numbers, and a larger held-out set is the next thing to build.", "url": "https://wpnews.pro/news/mca-open-source-neuro-symbolic-wrapper-that-teaches-7b-llms-to-challenge-false", "canonical_source": "https://discuss.huggingface.co/t/mca-open-source-neuro-symbolic-wrapper-that-teaches-7b-llms-to-challenge-false-premises-explain-paradoxes-and-abstain-code-dataset-space-2xt4-benchmark/191913#post_4", "published_at": "2026-11-16 03:48:55+00:00", "updated_at": "2026-11-16 03:50:13.467469+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["Mati83mon", "MCA", "GitHub", "Llama-3", "Qwen2.5", "DeepSeek-R1", "Kaggle"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/mca-open-source-neuro-symbolic-wrapper-that-teaches-7b-llms-to-challenge-false", "markdown": "https://wpnews.pro/news/mca-open-source-neuro-symbolic-wrapper-that-teaches-7b-llms-to-challenge-false.md", "text": "https://wpnews.pro/news/mca-open-source-neuro-symbolic-wrapper-that-teaches-7b-llms-to-challenge-false.txt", "jsonld": "https://wpnews.pro/news/mca-open-source-neuro-symbolic-wrapper-that-teaches-7b-llms-to-challenge-false.jsonld"}}