{"slug": "reasoning-corpus-builder-script", "title": "Reasoning corpus builder script", "summary": "A developer published a standalone Python script that regenerates the 250-row MiMo-2.6-Pro high-reasoning corpus using a different teacher model via OpenRouter. The tool inlines the MiMo taxonomy, supports category and profile filtering (coder, indiedev, genagent), preserves original row ids, and writes JSONL in the same schema with per-row usage and latency, plus a --dry-run mode to inspect selections without spending tokens.", "body_md": "|  | #!/usr/bin/env python3 | \n|  | \"\"\"Regenerate the MiMo reasoning corpus with a different teacher via OpenRouter. | \n|  | Single file: the MiMo taxonomy is inlined, no local module is imported. Needs `typer`, | \n|  | `requests`, `rich`; `datasets` only when the source is an HF id; `transformers` only for | \n|  | `--estimate`. Run `--check` to verify the environment. | \n|  | Reads the prompts (system + user) from the 250-row MiMo set and writes a new JSONL in the | \n|  | same schema the MiMo file uses: | \n|  | {\"source\": ..., \"model\": <openrouter model>, \"messages\": [ | \n|  | {\"role\": \"system\", \"content\": ...}, | \n|  | {\"role\": \"user\", \"content\": ...}, | \n|  | {\"role\": \"assistant\", \"content\": ..., \"reasoning\": ...}]} | \n|  | Extra keys per row: `id` (original 0-based row index), `category`, `usage`, `latency_s`. | \n|  | `id` matters: categories are defined by row index, so a filtered run keeps the original ids. | \n|  | Usage: | \n|  | export OPENROUTER_API_KEY=sk-or-... | \n|  | python scripts/openrouter_reasoning_corpus.py \\ | \n|  | --model deepseek/deepseek-v4-pro \\ | \n|  | --profile coder \\ | \n|  | --out data/deepseek-v4-pro-coder.jsonl | \n|  | # one teacher, everything except stats and finance, including unlabelled rows | \n|  | python scripts/openrouter_reasoning_corpus.py -m moonshotai/kimi-k3.1 \\ | \n|  | --no-stats --no-finance --uncategorized -o data/kimi-k3.1-all.jsonl | \n|  | # inspect the selection without spending tokens | \n|  | python scripts/openrouter_reasoning_corpus.py -m deepseek/deepseek-v4-pro --dry-run | \n|  | \"\"\" | \n|  | from __future__ import annotations | \n|  | import importlib.util | \n|  | import json | \n|  | import os | \n|  | import platform | \n|  | import random | \n|  | import sys | \n|  | import threading | \n|  | import time | \n|  | from collections import defaultdict | \n|  | from concurrent.futures import ThreadPoolExecutor, as_completed | \n|  | from pathlib import Path | \n|  | from typing import Any, Optional | \n|  | import requests | \n|  | import typer | \n|  | from rich.console import Console | \n|  | from rich.table import Table | \n|  | ROOT = Path(__file__).resolve().parent.parent | \n|  | # MiMo-2.6-Pro taxonomy, judge-labelled 2026-11-07. Inlined so this file is standalone. | \n|  | CATEGORIES = { | \n|  | \"1_code\": \"0 2 30 68 96 99 117 134 137 139 143 144 145 147 160 164 169 181 184 \" | \n|  | \"188 191 194 199 201 210 218 219 231 234 243 244 246\", | \n|  | \"2_cs\": \"3 4 5 6 7 8 9 10 11 12 14 15 16 18 19 20 21 39 48 56 65 74 83 92 112 \" | \n|  | \"115 120 121 158\", | \n|  | \"4_stats\": \"23 41 59 67 76 85 94 107 141\", | \n|  | \"5_finance\": \"25 33 42 51 60 69 77 86 95 105 116 124 202\", | \n|  | \"6_business\": \"26 28 34 37 43 45 52 54 61 63 70 72 78 81 87 89 101 102 106 110 \" | \n|  | \"114 131\", | \n|  | \"8_science\": \"36 44 62 71 100 108 123 129\", | \n|  | \"9_humanities\": \"1 17 24 29 38 47 50 55 64 73 82 91 98 109 122 130 136 138 140 \" | \n|  | \"142 151 153 155 165 183 187 195 198 207 223 233 245\", | \n|  | } | \n|  | PROFILES = { | \n|  | \"coder\": [\"1_code\", \"2_cs\", \"4_stats\"], | \n|  | \"indiedev\": [\"1_code\", \"2_cs\", \"5_finance\", \"6_business\", \"9_humanities\"], | \n|  | \"genagent\": [\"5_finance\", \"6_business\", \"8_science\", \"9_humanities\"], | \n|  | } | \n|  | API_URL = \"https://openrouter.ai/api/v1/chat/completions\" | \n|  | MODELS_URL = \"https://openrouter.ai/api/v1/models\" | \n|  | HF_SOURCE = \"xtools-at/Xiaomi-MiMo-2.6-Pro-High-Reasoning-250x\" | \n|  | LOCAL_SOURCE = ROOT / \"data\" / \"mimo-2.6-pro-high-250x.jsonl\" | \n|  | # CLI flag -> category key in CATEGORIES | \n|  | FLAG_TO_CAT = { | \n|  | \"code\": \"1_code\", | \n|  | \"cs\": \"2_cs\", | \n|  | \"stats\": \"4_stats\", | \n|  | \"finance\": \"5_finance\", | \n|  | \"business\": \"6_business\", | \n|  | \"science\": \"8_science\", | \n|  | \"humanities\": \"9_humanities\", | \n|  | } | \n|  | console = Console() | \n|  | app = typer.Typer(add_completion=False, no_args_is_help=True) | \n|  | class ApiError(RuntimeError): | \n|  | def __init__(self, status: int, body: str): | \n|  | self.status = status | \n|  | self.body = body | \n|  | super().__init__(f\"HTTP {status}: {body[:300]}\") | \n|  | # ---------------------------------------------------------------- input | \n|  | def load_rows(source: str, split: str) -> list[dict]: | \n|  | path = Path(source) | \n|  | if path.exists(): | \n|  | rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()] | \n|  | console.print(f\"[dim]source: {path} ({len(rows)} rows)[/dim]\") | \n|  | return rows | \n|  | from datasets import load_dataset # imported lazily; local JSONL needs no datasets | \n|  | ds = load_dataset(source, split=split) | \n|  | rows = [dict(r) for r in ds] | \n|  | console.print(f\"[dim]source: {source} [{split}] ({len(rows)} rows)[/dim]\") | \n|  | return rows | \n|  | def category_index() -> dict[int, str]: | \n|  | \"\"\"row id -> category. Categories are disjoint in the MiMo taxonomy; first wins if not.\"\"\" | \n|  | out: dict[int, str] = {} | \n|  | for cat, ids in CATEGORIES.items(): | \n|  | for raw in ids.split(): | \n|  | out.setdefault(int(raw), cat) | \n|  | return out | \n|  | def select( | \n|  | rows: list[dict], | \n|  | profile: str, | \n|  | flags: dict[str, bool], | \n|  | uncategorized: bool, | \n|  | limit: int, | \n|  | offset: int, | \n|  | ) -> list[tuple[int, Optional[str], dict]]: | \n|  | if profile != \"all\" and profile not in PROFILES: | \n|  | raise typer.BadParameter(f\"unknown profile {profile!r}; choose all, {', '.join(PROFILES)}\") | \n|  | enabled = set(PROFILES[profile]) if profile != \"all\" else set(CATEGORIES) | \n|  | enabled = {c for c in enabled if flags.get(c, True)} | \n|  | index = category_index() | \n|  | picked: list[tuple[int, Optional[str], dict]] = [] | \n|  | for i, row in enumerate(rows): | \n|  | cat = index.get(i) | \n|  | if cat is None: | \n|  | if not uncategorized: | \n|  | continue | \n|  | elif cat not in enabled: | \n|  | continue | \n|  | picked.append((i, cat, row)) | \n|  | picked = picked[offset:] | \n|  | if limit: | \n|  | picked = picked[:limit] | \n|  | return picked | \n|  | # ---------------------------------------------------------------- request | \n|  | def build_body( | \n|  | model: str, | \n|  | messages: list[dict], | \n|  | temperature: float, | \n|  | top_p: Optional[float], | \n|  | max_tokens: int, | \n|  | stream: bool, | \n|  | reasoning: bool, | \n|  | reasoning_effort: Optional[str], | \n|  | seed: Optional[int], | \n|  | extra_body: Optional[str], | \n|  | ) -> dict[str, Any]: | \n|  | body: dict[str, Any] = { | \n|  | \"model\": model, | \n|  | \"messages\": messages, | \n|  | \"temperature\": temperature, | \n|  | \"max_tokens\": max_tokens, | \n|  | \"stream\": stream, | \n|  | \"usage\": {\"include\": True}, | \n|  | } | \n|  | if top_p is not None: | \n|  | body[\"top_p\"] = top_p | \n|  | if seed is not None: | \n|  | body[\"seed\"] = seed | \n|  | if reasoning: | \n|  | body[\"reasoning\"] = {\"effort\": reasoning_effort} if reasoning_effort else {\"enabled\": True} | \n|  | if extra_body: | \n|  | body.update(json.loads(extra_body)) | \n|  | return body | \n|  | def headers_for(api_key: str, referer: Optional[str], title: Optional[str]) -> dict[str, str]: | \n|  | headers = { | \n|  | \"Authorization\": f\"Bearer {api_key}\", | \n|  | \"Content-Type\": \"application/json\", | \n|  | } | \n|  | if referer: | \n|  | headers[\"HTTP-Referer\"] = referer | \n|  | if title: | \n|  | headers[\"X-Title\"] = title | \n|  | return headers | \n|  | def call_stream( | \n|  | body: dict, headers: dict[str, str], timeout: int | \n|  | ) -> tuple[str, str, Optional[dict]]: | \n|  | reasoning: list[str] = [] | \n|  | content: list[str] = [] | \n|  | usage: Optional[dict] = None | \n|  | finish: Optional[str] = None | \n|  | served: Optional[str] = None | \n|  | with requests.post( | \n|  | API_URL, headers=headers, json=body, stream=True, timeout=(15, timeout) | \n|  | ) as resp: | \n|  | if resp.status_code >= 400: | \n|  | raise ApiError(resp.status_code, resp.text) | \n|  | for raw in resp.iter_lines(decode_unicode=False): | \n|  | if not raw: | \n|  | continue | \n|  | line = raw.decode(\"utf-8\", errors=\"replace\") | \n|  | if line.startswith(\":\"): | \n|  | continue | \n|  | if not line.startswith(\"data:\"): | \n|  | continue | \n|  | payload = line[5:].strip() | \n|  | if payload == \"[DONE]\": | \n|  | break | \n|  | chunk = json.loads(payload) | \n|  | if chunk.get(\"error\"): | \n|  | raise ApiError(200, json.dumps(chunk[\"error\"])[:500]) | \n|  | if chunk.get(\"model\"): | \n|  | served = chunk[\"model\"] | \n|  | if chunk.get(\"usage\"): | \n|  | usage = chunk[\"usage\"] | \n|  | for choice in chunk.get(\"choices\") or []: | \n|  | if choice.get(\"finish_reason\"): | \n|  | finish = choice[\"finish_reason\"] | \n|  | delta = choice.get(\"delta\") or {} | \n|  | if delta.get(\"reasoning\"): | \n|  | reasoning.append(delta[\"reasoning\"]) | \n|  | if delta.get(\"reasoning_content\"): | \n|  | reasoning.append(delta[\"reasoning_content\"]) | \n|  | if delta.get(\"content\"): | \n|  | content.append(delta[\"content\"]) | \n|  | return \"\".join(reasoning), \"\".join(content), usage, finish, served | \n|  | def call_plain( | \n|  | body: dict, headers: dict[str, str], timeout: int | \n|  | ) -> tuple[str, str, Optional[dict]]: | \n|  | resp = requests.post(API_URL, headers=headers, json=body, timeout=(15, timeout)) | \n|  | if resp.status_code >= 400: | \n|  | raise ApiError(resp.status_code, resp.text) | \n|  | data = resp.json() | \n|  | if data.get(\"error\"): | \n|  | raise ApiError(200, json.dumps(data[\"error\"])[:500]) | \n|  | message = data[\"choices\"][0][\"message\"] | \n|  | reasoning = message.get(\"reasoning\") or message.get(\"reasoning_content\") or \"\" | \n|  | return reasoning, message.get(\"content\") or \"\", data.get(\"usage\"), data[\"choices\"][0].get(\"finish_reason\"), data.get(\"model\") | \n|  | def with_retries(fn, body: dict, headers: dict, timeout: int, retries: int) -> tuple[str, str, Optional[dict], Optional[str], Optional[str]]: | \n|  | delay = 2.0 | \n|  | for attempt in range(retries + 1): | \n|  | try: | \n|  | return fn(body, headers, timeout) | \n|  | except ApiError as exc: | \n|  | retryable = exc.status in (408, 409, 429, 500, 502, 503, 504) | \n|  | if not retryable or attempt == retries: | \n|  | raise | \n|  | sleep_for = delay * (2**attempt) + random.uniform(0, 1) | \n|  | console.print(f\"[yellow]retry {attempt + 1}/{retries} in {sleep_for:.1f}s[/yellow] [dim]{exc.body[:120]}[/dim]\") | \n|  | time.sleep(sleep_for) | \n|  | except (requests.ConnectionError, requests.Timeout) as exc: | \n|  | if attempt == retries: | \n|  | raise ApiError(0, str(exc)) | \n|  | sleep_for = delay * (2**attempt) + random.uniform(0, 1) | \n|  | console.print(f\"[yellow]network retry {attempt + 1}/{retries} in {sleep_for:.1f}s[/yellow]\") | \n|  | time.sleep(sleep_for) | \n|  | raise AssertionError(\"unreachable\") | \n|  | # ---------------------------------------------------------------- cost | \n|  | def fetch_model_entry(model: str, api_key: Optional[str]) -> Optional[dict]: | \n|  | headers = {\"Authorization\": f\"Bearer {api_key}\"} if api_key else {} | \n|  | resp = requests.get(MODELS_URL, headers=headers, timeout=30) | \n|  | resp.raise_for_status() | \n|  | for entry in resp.json().get(\"data\", []): | \n|  | if model in (entry.get(\"id\"), entry.get(\"canonical_slug\")): | \n|  | return entry | \n|  | return None | \n|  | def resolve_variant(model: str, api_key: Optional[str], pin: bool) -> dict: | \n|  | \"\"\"Resolve an OpenRouter alias to the dated variant it currently points at.\"\"\" | \n|  | entry = fetch_model_entry(model, api_key) | \n|  | if entry is None: | \n|  | console.print(f\"[yellow]{model!r} not in /models; keeping the id as-is[/yellow]\") | \n|  | return {\"model\": request_model, \"canonical\": None, \"name\": None, \"created\": None} | \n|  | canonical = entry.get(\"canonical_slug\") or entry.get(\"id\") | \n|  | return { | \n|  | \"model\": canonical if pin else model, | \n|  | \"canonical\": canonical, | \n|  | \"name\": entry.get(\"name\"), | \n|  | \"created\": entry.get(\"created\"), | \n|  | } | \n|  | def fetch_pricing( | \n|  | model: str, | \n|  | api_key: Optional[str], | \n|  | price_prompt: Optional[float], | \n|  | price_completion: Optional[float], | \n|  | ) -> tuple[float, float]: | \n|  | \"\"\"Return (USD per prompt token, USD per completion token).\"\"\" | \n|  | if price_prompt is not None and price_completion is not None: | \n|  | return price_prompt / 1e6, price_completion / 1e6 | \n|  | entry = fetch_model_entry(model, api_key) | \n|  | if entry is None: | \n|  | raise typer.BadParameter( | \n|  | f\"{model!r} not found in OpenRouter /models; pass --price-prompt/--price-completion (USD per 1M tokens)\" | \n|  | ) | \n|  | pricing = entry.get(\"pricing\") or {} | \n|  | per_prompt = float(pricing.get(\"prompt\") or 0.0) | \n|  | per_completion = float(pricing.get(\"completion\") or 0.0) | \n|  | return ( | \n|  | price_prompt / 1e6 if price_prompt is not None else per_prompt, | \n|  | price_completion / 1e6 if price_completion is not None else per_completion, | \n|  | ) | \n|  | def get_tokenizer(name: str): | \n|  | try: | \n|  | from transformers import AutoTokenizer | \n|  | return AutoTokenizer.from_pretrained(name, trust_remote_code=True) | \n|  | except Exception as exc: # noqa: BLE001 - estimation must not fail the run | \n|  | console.print(f\"[yellow]tokenizer {name!r} unavailable ({str(exc)[:120]}); falling back to chars/3.6[/yellow]\") | \n|  | return None | \n|  | def count_tokens(tok, text: str) -> int: | \n|  | if tok is None: | \n|  | return int(len(text) / 3.6) | \n|  | return len(tok(text, add_special_tokens=False).input_ids) | \n|  | def estimate_costs( | \n|  | picked: list[tuple[int, Optional[str], dict]], | \n|  | tok, | \n|  | price_in: float, | \n|  | price_out: float, | \n|  | max_tokens: int, | \n|  | assume_out: int, | \n|  | ) -> tuple[float, float]: | \n|  | per: dict[str, dict] = defaultdict(lambda: {\"rows\": 0, \"in\": 0, \"out\": []}) | \n|  | for _, cat, row in picked: | \n|  | prompt_text = \"\".join( | \n|  | m.get(\"content\", \"\") for m in row[\"messages\"] if m.get(\"role\") in (\"system\", \"user\") | \n|  | ) | \n|  | in_tok = count_tokens(tok, prompt_text) + 8 # role markers | \n|  | assistant = row[\"messages\"][-1] | \n|  | if assume_out: | \n|  | out_tok = assume_out | \n|  | else: | \n|  | out_tok = count_tokens( | \n|  | tok, (assistant.get(\"reasoning\") or \"\") + \"\\n\" + (assistant.get(\"content\") or \"\") | \n|  | ) | \n|  | bucket = per[cat or \"uncategorized\"] | \n|  | bucket[\"rows\"] += 1 | \n|  | bucket[\"in\"] += in_tok | \n|  | bucket[\"out\"].append(out_tok) | \n|  | table = Table(title=\"cost estimate\") | \n|  | for col, right in ((\"category\", False), (\"rows\", True), (\"input tok\", True), | \n|  | (\"out p50\", True), (\"out p90\", True), (\"cost p50\", True), (\"cost p90\", True)): | \n|  | table.add_column(col, justify=\"right\" if right else \"left\") | \n|  | total_rows = total_in = 0 | \n|  | total_p50 = total_p90 = 0.0 | \n|  | for cat, bucket in sorted(per.items()): | \n|  | outs = sorted(bucket[\"out\"]) | \n|  | p50 = outs[len(outs) // 2] | \n|  | p90 = outs[min(len(outs) - 1, int(len(outs) * 0.9))] | \n|  | c50 = bucket[\"in\"] * price_in + bucket[\"rows\"] * p50 * price_out | \n|  | c90 = bucket[\"in\"] * price_in + bucket[\"rows\"] * p90 * price_out | \n|  | table.add_row(cat, str(bucket[\"rows\"]), f\"{bucket['in']:,}\", f\"{p50:,}\", f\"{p90:,}\", | \n|  | f\"${c50:.2f}\", f\"${c90:.2f}\") | \n|  | total_rows += bucket[\"rows\"] | \n|  | total_in += bucket[\"in\"] | \n|  | total_p50 += c50 | \n|  | total_p90 += c90 | \n|  | table.add_row(\"TOTAL\", str(total_rows), f\"{total_in:,}\", \"\", \"\", | \n|  | f\"${total_p50:.2f}\", f\"${total_p90:.2f}\", style=\"bold\") | \n|  | console.print(table) | \n|  | ceiling = total_in * price_in + total_rows * max_tokens * price_out | \n|  | console.print( | \n|  | f\"[dim]output prior = MiMo traces tokenized with the same tokenizer; the target teacher may differ. \" | \n|  | f\"Hard ceiling at max_tokens={max_tokens}: ${ceiling:.2f}[/dim]\" | \n|  | ) | \n|  | return total_p50, total_p90 | \n|  | # ---------------------------------------------------------------- output | \n|  | HARD_DEPS = (\"requests\", \"typer\", \"rich\") | \n|  | OPTIONAL_DEPS = {\"datasets\": \"pip install datasets\", \"transformers\": \"pip install transformers\"} | \n|  | def _installed(module: str) -> bool: | \n|  | return importlib.util.find_spec(module) is not None | \n|  | def check_environment( | \n|  | *, | \n|  | source: str, | \n|  | out: Path, | \n|  | api_key: Optional[str], | \n|  | need_transformers: bool, | \n|  | need_key: bool, | \n|  | report_only: bool = False, | \n|  | ) -> None: | \n|  | \"\"\"Fail early with one actionable message instead of a traceback mid-run.\"\"\" | \n|  | problems: list[str] = [] | \n|  | if sys.version_info < (3, 9): | \n|  | problems.append(f\"Python >= 3.9 required, found {platform.python_version()}\") | \n|  | missing = [m for m in HARD_DEPS if not _installed(m)] | \n|  | if missing: | \n|  | problems.append(f\"missing packages: {', '.join(missing)} -> pip install {' '.join(missing)}\") | \n|  | src_path = Path(source) | \n|  | data_suffixes = (\".jsonl\", \".json\", \".jsonl.gz\", \".json.gz\", \".parquet\") | \n|  | is_hf = ( | \n|  | not src_path.exists() | \n|  | and source.count(\"/\") == 1 | \n|  | and not source.endswith(data_suffixes) | \n|  | ) | \n|  | if src_path.exists(): | \n|  | src_state = f\"file ({src_path})\" | \n|  | elif is_hf: | \n|  | src_state = f\"HF dataset ({source})\" | \n|  | if not _installed(\"datasets\"): | \n|  | problems.append(\"source is an HF dataset id but `datasets` is missing -> pip install datasets\") | \n|  | else: | \n|  | src_state = \"MISSING\" | \n|  | problems.append(f\"source not found: {src_path} (pass --source <jsonl\\|hf-id>)\") | \n|  | if need_transformers and not _installed(\"transformers\"): | \n|  | console.print(\"[yellow]! transformers not installed; --estimate will fall back to chars/3.6[/yellow]\") | \n|  | key_state = \"set\" if api_key else \"not set\" | \n|  | if need_key and not api_key: | \n|  | problems.append(\"missing API key: pass --api-key or set OPENROUTER_API_KEY\") | \n|  | parent = out.parent | \n|  | probe = parent | \n|  | while not probe.exists() and probe != probe.parent: | \n|  | probe = probe.parent | \n|  | if not os.access(probe, os.W_OK): | \n|  | problems.append(f\"output directory not writable: {parent}\") | \n|  | if report_only: | \n|  | console.print(f\"python {platform.python_version()} ({sys.executable})\") | \n|  | for m in HARD_DEPS: | \n|  | console.print(f\" {m:12s} {'ok' if _installed(m) else 'MISSING'}\") | \n|  | for m, hint in OPTIONAL_DEPS.items(): | \n|  | console.print(f\" {m:12s} {'ok' if _installed(m) else 'not installed -> ' + hint}\") | \n|  | console.print(f\" api key {key_state}\") | \n|  | console.print(f\" source {src_state}\") | \n|  | console.print(f\" output {out}\") | \n|  | if problems: | \n|  | for problem in problems: | \n|  | console.print(f\"[red]x {problem}[/red]\") | \n|  | raise typer.Exit(1) | \n|  | if report_only: | \n|  | console.print(\"[green]all required checks passed[/green]\") | \n|  | raise typer.Exit(0) | \n|  | def slugify(model: str) -> str: | \n|  | return \"\".join(c if c.isalnum() or c in \"-._\" else \"-\" for c in model).strip(\"-\") | \n|  | @app.command() | \n|  | def main( | \n|  | model: Optional[str] = typer.Option(None, \"--model\", \"-m\", help=\"OpenRouter model id, e.g. deepseek/deepseek-v4-pro\"), | \n|  | api_key: Optional[str] = typer.Option(None, \"--api-key\", envvar=\"OPENROUTER_API_KEY\", help=\"or set OPENROUTER_API_KEY\"), | \n|  | source: Optional[str] = typer.Option(None, \"--source\", \"-s\", help=\"JSONL path or HF dataset id (default: local file, else the MiMo HF dataset)\"), | \n|  | split: str = typer.Option(\"train\", \"--split\", help=\"split when --source is a HF dataset id\"), | \n|  | out: Optional[Path] = typer.Option(None, \"--out\", \"-o\", help=\"default data/openrouter-<model>.jsonl\"), | \n|  | profile: str = typer.Option(\"all\", \"--profile\", help=\"all \\| coder \\| indiedev \\| genagent\"), | \n|  | code: bool = typer.Option(True, \"--code/--no-code\"), | \n|  | cs: bool = typer.Option(True, \"--cs/--no-cs\"), | \n|  | stats: bool = typer.Option(True, \"--stats/--no-stats\"), | \n|  | finance: bool = typer.Option(True, \"--finance/--no-finance\"), | \n|  | business: bool = typer.Option(True, \"--business/--no-business\"), | \n|  | science: bool = typer.Option(True, \"--science/--no-science\"), | \n|  | humanities: bool = typer.Option(True, \"--humanities/--no-humanities\"), | \n|  | uncategorized: bool = typer.Option(False, \"--uncategorized/--no-uncategorized\", help=\"include the 105 rows outside the taxonomy\"), | \n|  | limit: int = typer.Option(0, \"--limit\", help=\"0 = all selected rows\"), | \n|  | offset: int = typer.Option(0, \"--offset\"), | \n|  | concurrency: int = typer.Option(4, \"--concurrency\", \"-j\"), | \n|  | temperature: float = typer.Option(0.7, \"--temperature\"), | \n|  | top_p: Optional[float] = typer.Option(None, \"--top-p\"), | \n|  | max_tokens: int = typer.Option(32768, \"--max-tokens\"), | \n|  | timeout: int = typer.Option(900, \"--timeout\", help=\"read timeout per request, seconds\"), | \n|  | retries: int = typer.Option(5, \"--retries\"), | \n|  | reasoning: bool = typer.Option(True, \"--reasoning/--no-reasoning\", help=\"send OpenRouter's reasoning parameter\"), | \n|  | reasoning_effort: Optional[str] = typer.Option(None, \"--reasoning-effort\", help=\"low \\| medium \\| high\"), | \n|  | stream: bool = typer.Option(True, \"--stream/--no-stream\"), | \n|  | seed: Optional[int] = typer.Option(None, \"--seed\"), | \n|  | extra_body: Optional[str] = typer.Option(None, \"--extra-body\", help=\"JSON merged into the request body\"), | \n|  | resume: bool = typer.Option(True, \"--resume/--no-resume\"), | \n|  | retry_truncated: bool = typer.Option(False, \"--retry-truncated\", help=\"on resume, drop finish_reason=length rows and regenerate them\"), | \n|  | dry_run: bool = typer.Option(False, \"--dry-run\"), | \n|  | list_categories: bool = typer.Option(False, \"--list-categories\"), | \n|  | check: bool = typer.Option(False, \"--check\", help=\"print the environment/dependency report and exit\"), | \n|  | referer: Optional[str] = typer.Option(None, \"--referer\", envvar=\"OPENROUTER_REFERER\"), | \n|  | title: Optional[str] = typer.Option(None, \"--title\", envvar=\"OPENROUTER_TITLE\"), | \n|  | estimate: bool = typer.Option(False, \"--estimate\", help=\"print token/cost estimate and exit\"), | \n|  | tokenizer: str = typer.Option(\"LiquidAI/LFM3-1.7B-Base\", \"--tokenizer\", help=\"HF id or local path for counting\"), | \n|  | assume_output_tokens: int = typer.Option(0, \"--assume-output-tokens\", help=\"override output tokens per row\"), | \n|  | price_prompt: Optional[float] = typer.Option(None, \"--price-prompt\", help=\"USD per 1M prompt tokens\"), | \n|  | price_completion: Optional[float] = typer.Option(None, \"--price-completion\", help=\"USD per 1M completion tokens\"), | \n|  | budget: float = typer.Option(0.0, \"--budget\", help=\"stop after this many USD of measured spend\"), | \n|  | resolve_model: bool = typer.Option(True, \"--resolve-model/--no-resolve-model\", help=\"look up the dated variant the alias points at\"), | \n|  | pin_variant: bool = typer.Option(False, \"--pin-variant/--no-pin-variant\", help=\"send the canonical dated slug instead of the alias\"), | \n|  | model_info: bool = typer.Option(False, \"--model-info\", help=\"print the resolved model entry and exit\"), | \n|  | ) -> None: | \n|  | flags = { | \n|  | FLAG_TO_CAT[k]: v | \n|  | for k, v in { | \n|  | \"code\": code, \"cs\": cs, \"stats\": stats, \"finance\": finance, | \n|  | \"business\": business, \"science\": science, \"humanities\": humanities, | \n|  | }.items() | \n|  | } | \n|  | if list_categories: | \n|  | index = category_index() | \n|  | console.print(f\"[bold]{len(CATEGORIES)} categories, {len(index)} labelled rows[/bold]\") | \n|  | for cat, ids in CATEGORIES.items(): | \n|  | rows = [int(x) for x in ids.split()] | \n|  | console.print(f\" {cat:14s} {len(rows):3d} {'enabled' if flags[cat] else 'disabled'}\") | \n|  | return | \n|  | source = source or (str(LOCAL_SOURCE) if LOCAL_SOURCE.exists() else HF_SOURCE) | \n|  | if check: | \n|  | check_environment(source=source, out=out or ROOT / \"data\" / \"openrouter-model.jsonl\", | \n|  | api_key=api_key, need_transformers=False, need_key=False, report_only=True) | \n|  | return | \n|  | if not model: | \n|  | raise typer.BadParameter(\"--model is required unless --list-categories/--check is used\") | \n|  | if profile != \"all\" and profile not in PROFILES: | \n|  | raise typer.BadParameter(f\"unknown profile {profile!r}; choose all, {', '.join(PROFILES)}\") | \n|  | if reasoning_effort and reasoning_effort not in (\"low\", \"medium\", \"high\"): | \n|  | raise typer.BadParameter(\"--reasoning-effort must be low, medium or high\") | \n|  | variant = {\"model\": model, \"canonical\": None, \"name\": None, \"created\": None} | \n|  | if resolve_model or pin_variant or model_info: | \n|  | variant = resolve_variant(model, api_key, pin_variant) | \n|  | request_model = variant[\"model\"] | \n|  | out = out or ROOT / \"data\" / f\"openrouter-{slugify(request_model)}.jsonl\" | \n|  | check_environment(source=source, out=out, api_key=api_key, | \n|  | need_transformers=estimate, | \n|  | need_key=not (dry_run or estimate or model_info)) | \n|  | if model_info: | \n|  | entry = fetch_model_entry(model, api_key) | \n|  | if entry is None: | \n|  | console.print(f\"[red]{model!r} not found in OpenRouter /models[/red]\") | \n|  | raise typer.Exit(1) | \n|  | subset = {k: entry.get(k) for k in (\"id\", \"canonical_slug\", \"name\", \"created\", | \n|  | \"hugging_face_id\", \"context_length\", \"pricing\")} | \n|  | subset[\"max_completion_tokens\"] = (entry.get(\"top_provider\") or {}).get(\"max_completion_tokens\") | \n|  | console.print_json(json.dumps(subset)) | \n|  | return | \n|  | rows = load_rows(source, split) | \n|  | picked = select(rows, profile, flags, uncategorized, limit, offset) | \n|  | if not picked: | \n|  | console.print(\"[red]no rows selected[/red]\") | \n|  | raise typer.Exit(1) | \n|  | enabled = sorted({c for _, c, _ in picked if c}) | \n|  | counts = {c: sum(1 for _, cc, _ in picked if cc == c) for c in enabled} | \n|  | if None in {c for _, c, _ in picked}: | \n|  | counts[\"uncategorized\"] = sum(1 for _, c, _ in picked if c is None) | \n|  | console.print( | \n|  | f\"[bold]{len(picked)} rows[/bold] selected profile={profile} \" | \n|  | f\"categories=\" + \", \".join(f\"{k}:{v}\" for k, v in counts.items()) | \n|  | ) | \n|  | if not uncategorized: | \n|  | unlabelled = len(rows) - len(category_index()) | \n|  | if unlabelled: | \n|  | console.print(f\"[dim]{unlabelled} unlabelled rows available; pass --uncategorized to include[/dim]\") | \n|  | if estimate: | \n|  | price_in, price_out = fetch_pricing(model, api_key, price_prompt, price_completion) | \n|  | console.print(f\"pricing: ${price_in * 1e6:.3f}/M in, ${price_out * 1e6:.3f}/M out\") | \n|  | if variant.get(\"canonical\"): | \n|  | console.print(f\"variant: {variant['canonical']} {variant.get('name') or ''}\") | \n|  | estimate_costs(picked, get_tokenizer(tokenizer), price_in, price_out, | \n|  | max_tokens, assume_output_tokens) | \n|  | return | \n|  | err_path = out.with_suffix(out.suffix + \".errors.jsonl\") | \n|  | done: set[int] = set() | \n|  | if resume and out.exists(): | \n|  | kept: list[str] = [] | \n|  | dropped = 0 | \n|  | for line in out.read_text().splitlines(): | \n|  | if not line.strip(): | \n|  | continue | \n|  | try: | \n|  | record = json.loads(line) | \n|  | rid = int(record[\"id\"]) | \n|  | except (json.JSONDecodeError, KeyError, ValueError): | \n|  | continue | \n|  | if retry_truncated and record.get(\"finish_reason\") == \"length\": | \n|  | dropped += 1 | \n|  | continue | \n|  | kept.append(line) | \n|  | done.add(rid) | \n|  | if dropped: | \n|  | tmp = out.with_suffix(out.suffix + \".tmp\") | \n|  | tmp.write_text(\"\\n\".join(kept) + (\"\\n\" if kept else \"\")) | \n|  | tmp.replace(out) | \n|  | console.print(f\"[dim]retry: dropped {dropped} truncated rows, kept {len(kept)}[/dim]\") | \n|  | if done: | \n|  | console.print(f\"[dim]resume: {len(done)} rows already in {out}[/dim]\") | \n|  | todo = [(i, c, r) for i, c, r in picked if i not in done] | \n|  | if dry_run: | \n|  | sample = todo[0] if todo else picked[0] | \n|  | msgs = [m for m in sample[2][\"messages\"] if m.get(\"role\") in (\"system\", \"user\")] | \n|  | body = build_body(request_model, msgs, temperature, top_p, max_tokens, stream, reasoning, | \n|  | reasoning_effort, seed, extra_body) | \n|  | console.print(f\"[bold]would write:[/bold] {out}\") | \n|  | canonical = f\" ({variant['canonical']})\" if variant.get(\"canonical\") and variant[\"canonical\"] != request_model else \"\" | \n|  | console.print(f\"[bold]would call:[/bold] {request_model}{canonical} x{len(todo)} concurrency={concurrency}\") | \n|  | console.print(\"[dim]first request body:[/dim]\") | \n|  | console.print_json(json.dumps(body, ensure_ascii=False)[:2000]) | \n|  | return | \n|  | if not api_key: | \n|  | console.print(\"[red]missing API key: pass --api-key or set OPENROUTER_API_KEY[/red]\") | \n|  | raise typer.Exit(1) | \n|  | headers = headers_for(api_key, referer, title) | \n|  | call = call_stream if stream else call_plain | \n|  | lock = threading.Lock() | \n|  | out.parent.mkdir(parents=True, exist_ok=True) | \n|  | stats = {\"ok\": 0, \"fail\": 0, \"skipped\": 0, \"truncated\": 0, \"reasoning_chars\": 0, \"content_chars\": 0, | \n|  | \"cost\": 0.0, \"prompt_tokens\": 0, \"completion_tokens\": 0} | \n|  | started = time.perf_counter() | \n|  | def work(item: tuple[int, Optional[str], dict]) -> None: | \n|  | i, cat, row = item | \n|  | if budget > 0: | \n|  | with lock: | \n|  | if stats[\"cost\"] >= budget: | \n|  | stats[\"skipped\"] += 1 | \n|  | return | \n|  | msgs = [m for m in row[\"messages\"] if m.get(\"role\") in (\"system\", \"user\")] | \n|  | body = build_body(request_model, msgs, temperature, top_p, max_tokens, stream, reasoning, | \n|  | reasoning_effort, seed, extra_body) | \n|  | t0 = time.perf_counter() | \n|  | try: | \n|  | reasoning_text, content, usage, finish, served = with_retries(call, body, headers, timeout, retries) | \n|  | if not content and not reasoning_text: | \n|  | raise ApiError(200, \"empty completion\") | \n|  | record = { | \n|  | \"source\": row.get(\"source\", \"\"), | \n|  | \"model\": request_model, | \n|  | \"model_requested\": model, | \n|  | \"model_canonical\": variant.get(\"canonical\"), | \n|  | \"model_name\": variant.get(\"name\"), | \n|  | \"served_model\": served, | \n|  | \"messages\": msgs + [{\"role\": \"assistant\", \"content\": content, \"reasoning\": reasoning_text}], | \n|  | \"id\": i, | \n|  | \"category\": cat, | \n|  | \"usage\": usage, | \n|  | \"finish_reason\": finish, | \n|  | \"latency_s\": round(time.perf_counter() - t0, 2), | \n|  | } | \n|  | line = json.dumps(record, ensure_ascii=False) | \n|  | with lock: | \n|  | with out.open(\"a\", encoding=\"utf-8\") as fh: | \n|  | fh.write(line + \"\\n\") | \n|  | stats[\"ok\"] += 1 | \n|  | stats[\"reasoning_chars\"] += len(reasoning_text) | \n|  | stats[\"content_chars\"] += len(content) | \n|  | if finish == \"length\": | \n|  | stats[\"truncated\"] += 1 | \n|  | if usage: | \n|  | stats[\"prompt_tokens\"] += int(usage.get(\"prompt_tokens\") or 0) | \n|  | stats[\"completion_tokens\"] += int(usage.get(\"completion_tokens\") or 0) | \n|  | if isinstance(usage.get(\"cost\"), (int, float)): | \n|  | stats[\"cost\"] += float(usage[\"cost\"]) | \n|  | flag = \" [yellow]TRUNCATED[/yellow]\" if finish == \"length\" else \"\" | \n|  | console.print( | \n|  | f\"[green]{stats['ok']:3d}/{len(todo)}[/green] id={i:3d} {str(cat):14s} \" | \n|  | f\"{record['latency_s']:7.1f}s r={len(reasoning_text):6d} c={len(content):6d}{flag}\" | \n|  | ) | \n|  | except Exception as exc: # noqa: BLE001 - one bad row must not kill the run | \n|  | with lock: | \n|  | stats[\"fail\"] += 1 | \n|  | with err_path.open(\"a\", encoding=\"utf-8\") as fh: | \n|  | fh.write(json.dumps({\"id\": i, \"category\": cat, \"model\": request_model, | \n|  | \"error\": str(exc)[:1000]}, ensure_ascii=False) + \"\\n\") | \n|  | console.print(f\"[red]FAIL[/red] id={i:3d} {str(cat):14s} {str(exc)[:160]}\") | \n|  | with ThreadPoolExecutor(max_workers=max(1, concurrency)) as pool: | \n|  | futures = [pool.submit(work, item) for item in todo] | \n|  | for future in as_completed(futures): | \n|  | future.result() | \n|  | elapsed = time.perf_counter() - started | \n|  | console.print( | \n|  | f\"\\n[bold]done[/bold] {stats['ok']} ok, {stats['fail']} failed\" | \n|  | + (f\", {stats['skipped']} skipped (budget)\" if stats[\"skipped\"] else \"\") | \n|  | + (f\", {stats['truncated']} truncated at max_tokens\" if stats[\"truncated\"] else \"\") | \n|  | + f\" in {elapsed / 60:.1f} min ({stats['ok'] / max(elapsed, 1e-9):.2f} rows/s)\" | \n|  | ) | \n|  | console.print( | \n|  | f\"tokens: {stats['prompt_tokens']:,} in, {stats['completion_tokens']:,} out\" | \n|  | + (f\" \\| measured cost ${stats['cost']:.4f}\" if stats[\"cost\"] else \"\") | \n|  | ) | \n|  | if stats[\"ok\"] and stats[\"cost\"]: | \n|  | full_count = len(select(rows, profile, flags, uncategorized, 0, offset)) | \n|  | console.print( | \n|  | f\"[dim]projected for all {full_count} selected rows: \" | \n|  | f\"${stats['cost'] / stats['ok'] * full_count:.2f}[/dim]\" | \n|  | ) | \n|  | console.print( | \n|  | f\"reasoning {stats['reasoning_chars'] / 1e6:.2f}M chars, \" | \n|  | f\"content {stats['content_chars'] / 1e6:.2f}M chars\" | \n|  | + (f\", cost ${stats['cost']:.4f}\" if stats[\"cost\"] else \"\") | \n|  | ) | \n|  | console.print(f\"wrote {out}\" + (f\" errors {err_path}\" if stats[\"fail\"] else \"\")) | \n|  | if __name__ == \"__main__\": | \n|  | app() |", "url": "https://wpnews.pro/news/reasoning-corpus-builder-script", "canonical_source": "https://gist.github.com/AJ-Gonzalez/31f1f8cfe45c233911e055d5cfed6da0", "published_at": "2026-11-11 12:24:31+00:00", "updated_at": "2026-11-11 19:24:09.861387+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-tools", "developer-tools"], "entities": ["OpenRouter", "MiMo-2.6-Pro", "Xiaomi", "DeepSeek", "Kimi", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/reasoning-corpus-builder-script", "markdown": "https://wpnews.pro/news/reasoning-corpus-builder-script.md", "text": "https://wpnews.pro/news/reasoning-corpus-builder-script.txt", "jsonld": "https://wpnews.pro/news/reasoning-corpus-builder-script.jsonld"}}