{"slug": "practical-model-evaluation-and-compression-tools", "title": "Practical model evaluation and compression tools", "summary": "Developer 0xSero released model-toolkit, a GitHub repository of standalone Python tools for evaluating, observing, pruning, and quantizing language models, drawn from the REAP and EXL3 experiments including the GLM-5.3 fidelity campaign and Harbor coding evaluations. The toolkit covers Terminal-Bench (Harbor, explicit 2.1 default), DeepSWE, GPQA Diamond, SciCode, EleutherAI's LM evaluation registry (MMLU, GSM8K, IFEval), KL-divergence fidelity measurement, Cerebras REAP observation and pruning, ExLlamaV3 EXL3 quantization, calibration coverage gates, runtime checks, throughput comparison, and SHA-256 evidence sealing, with each tool shipping a portable SKILL.md for coding agents. It requires Python 3.10+, with the pinned Cerebras REAP checkout requiring Python 3.12+, and bundles no model weights, benchmark questions, session logs, or credentials.", "body_md": "Practical tools for evaluating, observing, pruning, and quantizing language models. Each tool also includes a portable `SKILL.md` for coding agents. Pick a folder, install that tool's dependencies, and run it. There is no server or orchestration framework to deploy.\n\nThis repository collects the workflows used in 0xSero's REAP and EXL3 experiments, including the GLM-5.3 fidelity campaign and Harbor coding evaluations. It contains reusable Python tools plus small launchers for the original upstream implementations. Model weights, benchmark questions, private session logs, and credentials are not bundled.\n\n| I want to… | Tool | What runs | \n|---|---|---|\n| Test an agent on terminal tasks | [Terminal-Bench](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/terminal-bench) | Harbor; explicit **2.1** default, configurable dataset | \n| Test long software engineering tasks | [DeepSWE](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/deep-swe) | Official DeepSWE task set through Pier | \n| Measure graduate-level science reasoning | [GPQA Diamond](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/gpqa-diamond) | Included API runner, deterministic choice shuffle, retained responses | \n| Evaluate scientific code | [SciCode](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/scicode) | Official SciCode Inspect evaluator | \n| Run reasoning/knowledge/ instruction benchmarks | [LM evaluation](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/lm-evaluation) | EleutherAI task registry: MMLU, GSM8K, IFEval, and more | \n| Measure fidelity lost through pruning/quantization | [KL divergence](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/kl-divergence) | Included full-logit capture and aligned forward-KL comparison | \n| Collect expert activation statistics | [REAP observation](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/reap-observation) | Pinned Cerebras REAP observer; observation-only by default | \n| Select which experts to keep | [REAP pruning](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/reap-pruning) | Global, domain-balanced, and max-over-domain keep plans | \n| Convert a supported model to EXL3 | [EXL3 quantization](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/exl3-quantization) | Pinned ExLlamaV3 converter, recipes and resume support | \n| Check calibration inputs and expert coverage | [Calibration](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/calibration) | Corpus hashes, counts, per-layer/expert coverage gate | \n| Check that the intended model really works | [Runtime checks](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/runtime-checks) | Model-list check, real completion, JSON output, graph-log audit | \n| Compare inference throughput fairly | [Throughput](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/throughput) | Common-window analysis of exact token-emission traces | \n| Package reproducible evidence | [Evidence](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/evidence) | SHA-256 sealing and tamper/missing-file verification | \n\n“Intelligence” is measured through specific tasks, not a single invented composite score. DeepSWE is the concrete software benchmark included here; it is not labeled as a separate benchmark called “Deep Suite.”\n\nPython **3.10+** is required. External tools may require newer Python; the pinned Cerebras REAP checkout requires **Python 3.12+**.\n\n```\ngit clone https://github.com/0xSero/model-toolkit.git\ncd model-toolkit\npython3 -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\npython toolkit.py list\n```\n\nMost launchers and GPQA use only the Python standard library. NumPy is needed for KL, pruning plans, and route coverage. Install large GPU libraries and benchmark harnesses in separate environments using each folder's README; their dependencies may conflict.\n\n```\nexport OPENAI_API_KEY='your-local-or-provider-key'\npython toolkit.py run runtime-checks \\\n  --base-url http://localhost:8000/v1 --model your-served-model \\\n  --output runs/smoke.json\n```\n\nAfter accepting GPQA's dataset terms and downloading `gpqa_diamond.csv`:\n\n```\npython toolkit.py run gpqa-diamond \\\n  --csv data/gpqa_diamond.csv --model your-served-model \\\n  --base-url http://localhost:8000/v1 --limit 2 \\\n  --output runs/gpqa-smoke\n```\n\nRemove `--limit` for the full 198-question set. Configure sampling, reasoning effort, and output budget explicitly for your comparison. Request failures withhold the aggregate accuracy; output truncations are reported separately.\n\n```\npython toolkit.py run terminal-bench \\\n  --model openai/your-served-model --jobs-dir runs/tb21 \\\n  --limit 1 --dry-run\npython toolkit.py run exl3-quantization \\\n  --input models/source --output models/exl3-4bpw \\\n  --work runs/exl3-work --bits 4 --dry-run\n```\n\nThe previews do not require the external harness or a GPU. Actual runs do. Terminal-Bench and DeepSWE require a container environment. SciCode executes generated code: run its harness inside a disposable evaluation environment.\n\n```\nPinned model + calibration corpus\n             ↓\n       REAP observations → coverage checks → expert keep plan\n             ↓                                 ↓\n       upstream pruning / architecture-specific weight surgery\n             ↓\n        EXL3 conversion → runtime checks → KL + task benchmarks\n                                             ↓\n                                      sealed evidence\n```\n\nObservation formats, expert tensor layouts, and quantization kernels vary by architecture. The tool READMEs state the supported boundary. A keep plan does not modify weights. A completed conversion does not prove runtime support or retained model quality.\n\nFor KL, compare the quantized model against **pruned BF16** to isolate quantization loss, and separately against **original BF16** to measure total pruning-plus-quantization loss. Keep sampling, prompts, tokenization, and task sets fixed when comparing task scores.\n\n- [GLM-5.3 REAP fidelity study](https://huggingface.co/datasets/0xSero/glm-5.3-reap-fidelity-study) : criterion sweeps, keep plans, KL reports, and GPQA artifacts.\n- [REAP calibration data](https://huggingface.co/datasets/0xSero/reap-calibration-data-v1) : calibration inputs and collection methodology. Audit constituent sources before claiming a benchmark-clean evaluation.\n- [Qwen layerwise observations](https://huggingface.co/datasets/0xSero/qwen35-reap-layerwise-observations) and[GLM layerwise observations](https://huggingface.co/datasets/0xSero/glm5-layerwise-reap-observations) : prior observation artifacts.\n- [GLM-5.3 observations](https://huggingface.co/datasets/0xSero/glm-5.3-reap-observations-v1) : referenced by the fidelity study, but**not anonymously accessible when checked on 2026-09-10** . No tool here requires access to it.\n- [Cerebras REAP](https://github.com/CerebrasResearch/reap) and[PR #17](https://github.com/CerebrasResearch/reap/pull/17) : underlying observation and pruning work.\n\nSee [REAP observation](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/reap-observation) for the executable path and [provenance](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/docs/PROVENANCE.md) for the connection to earlier sessions and skills.\n\n`upstreams.json` records exact upstream commits. `python toolkit.py fetch NAME` checks out one of them under `.external/`; it does not install dependencies. Record harness/package versions, model and dataset revisions, hardware, context, sampling, timeouts, and all failed attempts with your run.\n\nThe included tools are tested with synthetic data, mocked API responses, and CLI checks. This release does **not** claim a fresh GPU conversion, full benchmark, or architecture compatibility test. See [validation](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/docs/VALIDATION.md). Historical model scores are deliberately not presented as results of this release.\n\nRun the local checks:\n\n```\npython -m unittest discover -s tests -v\n```\n\nMIT for this toolkit's code; imported MIT notices are retained in `licenses/`. External harnesses, datasets, models, and model-specific code retain their own licenses. REAP belongs to Cerebras Research; ExLlamaV3/EXL3 to turboderp and contributors; Terminal-Bench/Harbor to their authors; DeepSWE/Pier to Datacurve; SciCode to its authors; lm-evaluation-harness to EleutherAI and contributors. See [provenance](https://github.com/0xSero/model-toolkit/blob/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0/docs/PROVENANCE.md).", "url": "https://wpnews.pro/news/practical-model-evaluation-and-compression-tools", "canonical_source": "https://github.com/0xSero/model-toolkit/tree/3ee1d140efff3a3d5129fd207b9ce6eb32df88b0", "published_at": "2026-09-21 15:05:52+00:00", "updated_at": "2026-09-21 15:24:26.609946+00:00", "lang": "en", "topics": ["ai-tools", "ai-research", "large-language-models", "mlops", "developer-tools"], "entities": ["0xSero", "model-toolkit", "REAP", "EXL3", "GLM-5.3", "Harbor", "DeepSWE", "Cerebras"], "alternates": {"html": "https://wpnews.pro/news/practical-model-evaluation-and-compression-tools", "markdown": "https://wpnews.pro/news/practical-model-evaluation-and-compression-tools.md", "text": "https://wpnews.pro/news/practical-model-evaluation-and-compression-tools.txt", "jsonld": "https://wpnews.pro/news/practical-model-evaluation-and-compression-tools.jsonld"}}