{"slug": "pi-vs-the-official-deepseek-harness-on-the-same-local-model-qwen3-8-27b", "title": "Pi vs. the official DeepSeek Harness on the same local model (Qwen3.8-27B)", "summary": "Pi 0.73.1 outperformed the official DeepSeek Harness (DSH) 0.1.1-rc.2 in a 16-run coding benchmark on the same local Qwen3.8-27B model, passing 6/8 externally checked runs versus DSH's 4/8, with mean external scores of 0.825 and 0.680. Pi also used 14.0% less total wall time (6,455.5 vs 7,356.5 seconds) and had 10.7% higher aggregate generation throughput, but the author cautions that the small sample size and single separating fixture do not support a general ranking.", "body_md": "Date: 2026-08-23 (America/Los_Angeles)\n\nIn this 16-run, coding-only benchmark, Pi led the official DeepSeek Harness\n(DSH) on the measured sample. Pi passed 6/8 externally checked runs versus\nDSH's 4/8, with mean external scores of 0.825 and 0.680 respectively. Pi won\ntwo paired task/trial comparisons, six tied, and DSH won none. Both separating\npairs were `taskflow`\n\n: Pi passed both; DSH timed out with partial scores of\n0.28 and 0.56.\n\nDSH also used 14.0% more total wall time (7,356.5 versus 6,455.5 seconds). It produced 14.8% more completed-response output tokens and had 10.7% lower aggregate generation throughput. These are whole-harness outcomes on one local model/runtime, not evidence that Pi is universally better. Four fixtures, two trials, and only two non-tied pairs are too small for a general ranking or a statistically persuasive winner claim.\n\nThis is a separate follow-up to the checked-in\n[ Pi versus Prime](/promptdriven/pdd/blob/main/research/omlx-qwen38-pi-prime/README.md) study. It does not replace\nor modify that study.\n\nOf the four tasks, two were at ceiling for both harnesses (`make-ci-green`\n\nand\n`add-feature`\n\n, 2/2 each) and one was at floor for both (`webcore`\n\n, 0/2 each,\nneither harness changing a file). Exactly one, `taskflow`\n\n, separated them. The\nentire pass-rate and mean-score margin therefore comes from a single fixture,\nand only two of eight paired cells were non-tied. That supports a lead on the\nmeasured sample and the cost/latency observations, not a general harness\nranking; no confidence interval or significance claim is warranted. The floor\ntask also argues against fixture curation in Pi's favor. See\n[Limitations](#limitations) for the full list.\n\n| Metric | Pi 0.73.1 | DSH 0.1.1-rc.2 | DSH vs Pi |\n|---|---|---|---|\n| External passes | 6/8 (75%) | 4/8 (50%) | -2 passes |\n| Mean external score | 0.825 | 0.680 | -0.145 |\n| Paired wins / losses / ties | 2 / 0 / 6 | 0 / 2 / 6 | Pi +2 wins |\n| Timeouts | 3 | 4 | +1 |\n| Total wall time | 6,455.5 s | 7,356.5 s | +14.0% |\n| Median wall time | 906.5 s | 1,055.2 s | +16.4% |\n| Endpoint requests | 86 | 80 | -7.0% |\n| Completed responses | 83 | 76 | -8.4% |\n| Completed-response input tokens | 1,009,881 | 1,130,071 | +11.9% |\n| Completed-response output tokens | 26,587 | 30,520 | +14.8% |\n| Median output tokens per run | 3,012 | 2,981.5 | -1.0% |\n| Aggregate prompt tokens/s | 387.47 | 382.55 | -1.3% |\n| Aggregate generation tokens/s | 12.74 | 11.38 | -10.7% |\n| Mean time to first token | 31.40 s | 38.87 s | +23.8% |\n\nThe token totals cover responses that delivered a final usage trailer. Seven\ntimed-out request streams (three Pi, four DSH) were cancelled and therefore\nhave no final usage record; their partial final-stream tokens are omitted. A\ntimeout does not automatically mean incorrect code: Pi's second `taskflow`\n\ncell reached the cap but its resulting workspace passed the hidden checker.\n\n| Task | Pi passes | DSH passes | Pi mean score | DSH mean score | Pi mean wall | DSH mean wall |\n|---|---|---|---|---|---|---|\n`make-ci-green` |\n2/2 | 2/2 | 1.00 | 1.00 | 221.7 s | 385.1 s |\n`add-feature` |\n2/2 | 2/2 | 1.00 | 1.00 | 608.0 s | 892.6 s |\n`taskflow` |\n2/2 | 0/2 | 1.00 | 0.42 | 1,197.4 s | 1,200.4 s |\n`webcore` |\n0/2 | 0/2 | 0.30 | 0.30 | 1,200.7 s | 1,200.2 s |\n\nOn the four pairs where both harnesses passed, Pi was faster by 144.3 to\n291.8 seconds. Both harnesses made no file changes in either `webcore`\n\ntrial,\nso its unchanged fixture score of 0.30 did not separate them.\n\n`Pass`\n\nis the external hidden checker's result. `Cap`\n\nmeans the harness process\nreached the common 1,200-second limit. Output tokens exclude a capped cell's\nunterminated final stream.\n\n| Trial | Task | Harness | Score | Pass | Cap | Wall (s) | Requests | Output tokens | Changed files |\n|---|---|---|---|---|---|---|---|---|---|\n| 1 | `make-ci-green` |\nPi | 1.00 | yes | no | 229.7 | 6 | 1,253 | 6 |\n| 1 | `make-ci-green` |\nDSH | 1.00 | yes | no | 374.0 | 7 | 2,499 | 6 |\n| 1 | `add-feature` |\nDSH | 1.00 | yes | no | 874.9 | 11 | 7,875 | 1 |\n| 1 | `add-feature` |\nPi | 1.00 | yes | no | 597.6 | 12 | 5,483 | 2 |\n| 1 | `taskflow` |\nPi | 1.00 | yes | no | 1,194.6 | 17 | 5,693 | 7 |\n| 1 | `taskflow` |\nDSH | 0.28 | no | yes | 1,200.5 | 12 | 3,586 | 0 |\n| 1 | `webcore` |\nDSH | 0.30 | no | yes | 1,200.2 | 10 | 1,619 | 0 |\n| 1 | `webcore` |\nPi | 0.30 | no | yes | 1,200.5 | 6 | 1,120 | 0 |\n| 2 | `make-ci-green` |\nDSH | 1.00 | yes | no | 396.2 | 9 | 3,012 | 6 |\n| 2 | `make-ci-green` |\nPi | 1.00 | yes | no | 213.7 | 6 | 1,518 | 6 |\n| 2 | `add-feature` |\nPi | 1.00 | yes | no | 618.5 | 14 | 5,804 | 2 |\n| 2 | `add-feature` |\nDSH | 1.00 | yes | no | 910.3 | 10 | 7,253 | 2 |\n| 2 | `taskflow` |\nDSH | 0.56 | no | yes | 1,200.3 | 12 | 2,951 | 3 |\n| 2 | `taskflow` |\nPi | 1.00 | yes | yes | 1,200.1 | 17 | 4,506 | 7 |\n| 2 | `webcore` |\nPi | 0.30 | no | yes | 1,200.8 | 8 | 1,210 | 0 |\n| 2 | `webcore` |\nDSH | 0.30 | no | yes | 1,200.1 | 9 | 1,725 | 0 |\n\nThe field-whitelisted source for this table and all aggregate calculations is\n[ results/2026-08-23-clean.json](/promptdriven/pdd/blob/main/research/omlx-qwen38-pi-deepseek-harness-2026-08-23/results/2026-08-23-clean.json). It contains\ncell metadata, scores, diff hashes/status, request identity, timing, and usage,\nbut no full prompts, model responses, checker output, credentials, or\nmachine-specific absolute paths.\n\nPrimary sources were accessed 2026-08-23. The benchmark pins the official\n`deepseek-ai/deepseek-harness`\n\nrelease\n[ dsh-v0.1.1-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.1-rc.2),\ncommit\n\n[, npm package](https://github.com/deepseek-ai/deepseek-harness/commit/b150a551b8d465e31e418e1b2eaf5e79bbb7d28e)\n\n`b150a551b8d465e31e418e1b2eaf5e79bbb7d28e`\n\n`@deepseek-ai/dsh@0.1.1-rc.2`\n\n, and\n[MIT license](https://github.com/deepseek-ai/deepseek-harness/blob/b150a551b8d465e31e418e1b2eaf5e79bbb7d28e/LICENSE). This is an RC/developer-preview release, so its behavior and configuration may change quickly.\n\nThe official one-shot invocation is:\n\n```\ndsh --profile headless --patch <benchmark.cordis.patch.yml> \"<exact task>\"\n```\n\nBefore inference, the runner resolves that same composition without booting it:\n\n```\ndsh --profile headless --patch <benchmark.cordis.patch.yml> --dump-config\n```\n\nThe shipped\n[headless bundle](https://github.com/deepseek-ai/deepseek-harness/blob/dsh-v0.1.1-rc.2/packages/bundle/headless/README.md)\ncreates a fresh persisted agent, submits the positional task as an ordinary\nuser message, waits for quiescence, and exits without starting a server. The\ncustom local route uses the official\n[ @deepseek-ai/dsh-llm-pi-ai adapter](https://github.com/deepseek-ai/deepseek-harness/blob/dsh-v0.1.1-rc.2/packages/llm/llm-pi-ai/README.md),\nbacked in this release by\n\n`@earendil-works/pi-ai`\n\n0.82.1. That dependency is a\ntransport/model adapter inside DSH; it is not the Pi coding-agent harness arm.DSH's shipped default model is `deepseek-v4-flash`\n\nthrough its native DeepSeek\nadapter. The native adapter defaults reasoning effort to high and its own\nrequest cap to 256,000. A hand-declared generic route instead has fallback\ncapacities of 262,144 context and 32,768 output, no reasoning capability unless\ndeclared, provider-default sampling when fields are omitted, and a normal\nagent-level retry policy of five retries. Those defaults were not silently\naccepted: the benchmark declared the exact oMLX model/capacities, medium\nreasoning, and zero agent-level retries to match a single Pi attempt.\n\nAn unknown OpenAI-compatible endpoint is not safely auto-detectable. DSH's\nofficial adapter documentation says it otherwise assumes OpenAI-style\n`developer`\n\nrole, `max_completion_tokens`\n\n, and bare `reasoning_effort`\n\n. oMLX\nrequires explicit compatibility values here:\n\n```\napi: openai-completions\nbaseURL: http://127.0.0.1:<per-cell-proxy>/v1\nreasoning: medium\nretryPolicy: {mode: normal, maxRetries: 0}\ncompat:\n  supportsDeveloperRole: false\n  supportsReasoningEffort: false\n  supportsUsageInStreaming: true\n  maxTokensField: max_tokens\n  thinkingFormat: qwen-chat-template\nmodels:\n  - id: Qwen3.8-27B-MLX-oQ8e-mtp\n    contextWindow: 98304\n    maxTokens: 32000\n    reasoningEfforts: {medium: medium}\n```\n\nThe real key never entered this configuration. DSH saw a dummy environment credential; the loopback proxy replaced it only on the upstream request.\n\nThe official\n[base bundle](https://github.com/deepseek-ai/deepseek-harness/blob/dsh-v0.1.1-rc.2/packages/bundle/base/cordis.patch.yml)\nships a broad native tool/control surface: bash or PowerShell, background jobs,\nfilesystem and search tools, skills, todo/goal tools, string replacement, web\nsearch, subagents/forks, workflows, and Ralph iteration, plus compaction and\nresult-pruning infrastructure. Headless defaults to `workspace-write`\n\nplus\napproval prompts.\n\nFor comparability and unattended execution, DSH used native presentation mode and native remaining tools, but the overlay disabled:\n\n- repository instruction/context-file loading and skills, matching Pi's flags;\n- web search, because the outer sandbox allowed only the local proxy;\n- LLM session-title generation, which would add unscored model traffic;\n- subagent, fork, workflow, and Ralph model fan-out, enforcing the no-concurrent oMLX rule.\n\nThe fan-out restriction is not a handicap specific to DSH. With one model loaded on a single oMLX server, concurrent subagent requests would serialize on that endpoint anyway, so fan-out buys no wall-clock parallelism here. Allowing it would also break the metering invariant that no two requests overlap inside a cell, which the analyzer enforces.\n\nDSH retained 19 model-facing tool schemas per request; Pi retained its four\nstock built-ins. DSH ran `danger-full-access`\n\nonly inside the stricter outer\nSeatbelt policy, removing headless approval prompts without broadening actual\nfilesystem or network access. This is a whole-harness comparison, not a\ntool-schema-matched ablation.\n\nThe identity correction is material: the local model is\n`Qwen3.8-27B-MLX-oQ8e-mtp`\n\n; no local oQ8c model exists. The benchmark used oQ8e\nonly.\n\n- MacBook Pro, Apple M4 Max, 16 CPU cores, 128 GB unified memory\n- macOS 26.5.2 build 25F84; Node.js 25.6.1\n- oMLX app/server 0.6.1\n- exact endpoint\n`http://127.0.0.1:8000/v1`\n\n- exact model\n`Qwen3.8-27B-MLX-oQ8e-mtp`\n\n- persisted 98,304-token context; 32,768 model output limit\n- temperature 0.6, top-p 0.95, top-k 20, repetition penalty 1.0\n- thinking enabled at medium; thinking-budget mode disabled\n- native MTP enabled at draft depth 4\n- DFlash, SpecPrefill, TurboQuant KV, VLM MTP, guided grammar, ANE prefill, and remote code disabled\n\nThe model and server settings were read and verified, not rewritten. Both\nharnesses sent `max_tokens: 32000`\n\n, omitted request-level temperature, top-p,\nand `reasoning_effort`\n\nso the same persisted runtime controls applied, and sent\nthe same Qwen chat-template kwargs (`enable_thinking: true`\n\n,\n`preserve_thinking: true`\n\n). The proxy verified every measured inference used\n`POST /v1/chat/completions`\n\nand the exact oQ8e model.\n\nThe four tasks came from\n[ minghinmatthewlam/openbench](https://github.com/minghinmatthewlam/openbench/tree/9e26c96a7df012ca9173e9725211c4cc58e11948)\nat exact commit\n\n`9e26c96a7df012ca9173e9725211c4cc58e11948`\n\n:| Task | SHA-256 of task tree | Purpose |\n|---|---|---|\n`make-ci-green` |\n`4a32fed72b90141fe985416c4b7d44d4f7b34c453fde69773e326b3994f9ce21` |\nMulti-module repair from a failing suite |\n`add-feature` |\n`d38f578795043c0f10f139647374d3d2563fbea21161ae978ce3c3297f9dbfba` |\nRecursive config-include feature |\n`taskflow` |\n`6ed568c66534b0f7839438bfd34c0696a2b6f4a07d2f9469c1541c7e0be074bb` |\nOrchestration/scheduling logic repair |\n`webcore` |\n`a5119ebfed497c749d2d0c93ea41b85dc1fbcd35d22fb0377a836ffa411f4dde` |\nConnected routing/mounting feature |\n\nEach cell received the exact fixture `instruction.md`\n\nplus the same fixed\nsuffix:\n\nWork only in the current workspace. Implement the requested coding change, run the relevant tests, and finish when the implementation is correct.\n\nThere were two trials per task and harness. Trial one alternated which harness went first by task; trial two reversed every task's order. Runs were serial, not concurrent.\n\nEvery cell had a fresh copied workspace, initialized git baseline, HOME, temp directory, Pi/DSH configuration, and session. A macOS Seatbelt profile denied external network, permitted only the per-cell loopback proxy, limited writes to the cell workspace/HOME/temp, and exposed only the exact isolated Pi and DSH installation roots plus system runtime paths. The runner denied 12 representative hidden-checker path probes before any inference.\n\nThe hidden `checker.sh`\n\nran outside the harness after every normal exit or\ntimeout, with a separate 180-second checker limit. The measured scores came\nfrom the pinned OpenBench task tree. After adversarial review, all 16 saved\nworkspaces were regraded without inference: the runner first required the\ncomplete task tree to match the hard-coded SHA-256 above, then invoked each\nchecker with the benchmark orchestrator's exact `sys.executable`\n\n, a minimal\nallowlisted environment (no orchestrator credentials), and a separate Seatbelt\nprofile that denied network, limited reads to the pinned task/workspace plus\nsystem runtimes, and limited writes to the workspace/checker temp directory.\nEvery score, pass flag, and checker exit matched the original result. This\nseparate per-run evidence is in the sanitized JSON; the published runner now\napplies the hardened boundary to every new checker execution as well. Harness\nself-reports and visible test claims were never scored. Golden solutions passed\nall four checkers; untouched fixtures produced scores 0.312, 0.400, 0.280, and\n0.300, confirming the checker path and partial-score behavior before traffic.\n\nThe metering proxy logged no message bodies. It recorded request IDs, method, path, model, selected request fields, tool count, event timing, response bytes, and usage/duration metadata. The analyzer required:\n\n- exactly one terminal response/error event per request;\n- no overlapping requests inside a cell;\n- exact endpoint and model identity;\n- the complete 4-task × 2-trial × 2-harness matrix with unique IDs;\n- exact task-tree hashes;\n- two consecutive oMLX samples with zero active and waiting requests at controlled boundaries.\n\nAll checks passed. Seven proxy errors are expected cancellation terminals for the seven timed-out final streams. No Pi, DSH, or benchmark child remained after the matrix.\n\nTwo orchestration preflights failed before any benchmark trial and are operational notes, not measurements:\n\n- oMLX's hard memory watermark blocked the first attempt.\n- After the previously resident\n`Qwen3.6-35B-A3B-bf16`\n\nmodel was unloaded (oMLX logged 65.39 GB freed), the balanced prefill guard blocked the second.\n\nGlobal oMLX Memory Guard was therefore changed from balanced to aggressive for\nbenchmark traffic. After all traffic, the runner restored it to balanced in a\n`finally`\n\npath. An independent authenticated check at 2026-08-23 15:40:16\n-0700 confirmed balanced plus two consecutive oMLX 0.6.1 samples with zero\nactive and zero waiting requests. This evidence is included in the sanitized\nJSON. No other persistent setting was changed.\n\nBefore the full protocol, the runner successfully completed a matched,\nexcluded `make-ci-green`\n\nsmoke pair:\n\n| Harness | External pass | Wall | Requests | Output tokens |\n|---|---|---|---|---|\n| DSH | yes | 404.3 s | 8 | 1,905 |\n| Pi | yes | 252.0 s | 6 | 1,283 |\n\nThe smoke gate confirmed successful process/checker exits, zero proxy errors, and identical observed request settings before allowing the full matrix.\n\nThe same loaded model, endpoint, hardware, server process, persisted sampling, reasoning mode, MTP depth, task bytes, prompt suffix, checker, isolation, and wall-clock cap were shared. That removes obvious model/runtime substitutions.\n\nWhat remains deliberately different is the harness: system-prompt assembly, tool schemas, context/history management, retry/compaction policies, editing strategy, termination decisions, and request count/shape. DSH's larger prompt and 19-tool surface likely contribute to its higher input-token total and latency, but this benchmark does not isolate a single causal mechanism. Reported token throughput is also workload-weighted: prompt length, cache reuse, and generation length differ because the harnesses chose different trajectories on the same model. It should not be read as a pure oMLX kernel benchmark.\n\nThe author has no affiliation with earendil-works (Pi) or deepseek-ai (DSH). Neither harness is a dependency of pdd, and pdd itself is not an arm of this benchmark. Pi is the harness the author uses day to day, which is why it is the reference arm here and in the Pi-versus-Prime study; readers should treat that as a bias to check rather than a disclosed-and-therefore-settled detail. The reverse experiment, the same protocol run by someone whose daily driver is DSH, is invited. The pinned locks and repro commands below exist to make that cheap.\n\n- Four fixtures and two trials are too small for a general harness ranking. With only two non-tied pairs, even both favoring Pi is weak inferential evidence; no confidence interval or significance claim is warranted.\n- Runs were balanced but not randomized. One long local session can contain thermal, cache, memory-pressure, or power drift.\n- All trials used one model, one quantization, one machine, one reasoning level, and one timeout.\n- Seven final partial streams have no token-usage trailer, so usage totals undercount work in capped cells.\n`webcore`\n\nwas too difficult for either harness at this budget and provided no separation.- DSH is an RC/developer-preview release and its generic transport is a newer, forked pi-ai package than Pi 0.73.1's transport. This is unavoidable for the pinned official release and is part of the DSH harness as shipped.\n- DSH retained more native tool schemas than Pi. The capability boundary was equalized for network/filesystem/model concurrency, not schema count.\n- Seatbelt is a best-effort local benchmark boundary, not a hardened multi-tenant security sandbox.\n\nA stronger follow-up would use more held-out repositories, randomized order, at least five trials per task, multiple budgets, and explicit thermal/power telemetry. Tool-schema and system-prompt ablations would help explain the whole-harness result.\n\nUse an isolated tool root; do not install either harness globally. The exact\ndependency graphs used for the published run are checked in under\n[ tool-lock/pi](/promptdriven/pdd/blob/main/research/omlx-qwen38-pi-deepseek-harness-2026-08-23/tool-lock/pi) and\n\n[. Their](/promptdriven/pdd/blob/main/research/omlx-qwen38-pi-deepseek-harness-2026-08-23/tool-lock/dsh)\n\n`tool-lock/dsh`\n\n`package-lock.json`\n\nSHA-256 values are respectively\n`c2c6cbeb831c9748f7071e32d81240ed017adc2a81aac7153ea5638821ca6275`\n\nand `d09b9445494e44c7f6239524acc4a82e751b67006f2b10ca3977eba94abd739a`\n\n;\nboth match the actual benchmark installations byte-for-byte. The commands\nbelow intentionally use `/private/tmp`\n\n, those locks, and exact source commits:\n\n```\nbench_root=/private/tmp/pdd-pi-dsh-bench-repro\nmkdir -p \"$bench_root/tools/pi\" \"$bench_root/tools/dsh\"\n\ncp research/omlx-qwen38-pi-deepseek-harness-2026-08-23/tool-lock/pi/package*.json \\\n  \"$bench_root/tools/pi/\"\ncp research/omlx-qwen38-pi-deepseek-harness-2026-08-23/tool-lock/dsh/package*.json \\\n  \"$bench_root/tools/dsh/\"\nnpm ci --prefix \"$bench_root/tools/pi\" --no-audit --no-fund\nnpm ci --prefix \"$bench_root/tools/dsh\" --no-audit --no-fund\n\ngit clone https://github.com/deepseek-ai/deepseek-harness.git \\\n  \"$bench_root/tools/deepseek-harness\"\ngit -C \"$bench_root/tools/deepseek-harness\" checkout \\\n  b150a551b8d465e31e418e1b2eaf5e79bbb7d28e\n\ngit clone https://github.com/minghinmatthewlam/openbench.git \\\n  \"$bench_root/openbench\"\ngit -C \"$bench_root/openbench\" checkout \\\n  9e26c96a7df012ca9173e9725211c4cc58e11948\n```\n\nBefore inference, load `Qwen3.8-27B-MLX-oQ8e-mtp`\n\nin oMLX 0.6.1 with the exact\npersisted settings above, ensure no active/waiting requests, and confirm the\nglobal Memory Guard target/rollback. The runner validates the hidden-checker\nboundary and DSH composed configuration, performs the matched smoke pair, gates\non request parity, runs the balanced matrix, and restores Memory Guard in its\nouter `finally`\n\nblock:\n\n```\neval \"$(conda shell.zsh hook)\"\nconda activate pdd\n\npython research/omlx-qwen38-pi-deepseek-harness-2026-08-23/benchmark.py \\\n  --tasks-root \"$bench_root/openbench/tasks\" \\\n  --pi \"$bench_root/tools/pi/node_modules/.bin/pi\" \\\n  --dsh \"$bench_root/tools/dsh/node_modules/.bin/dsh\" \\\n  --tool-root \"$bench_root/tools\" \\\n  --run-base \"$bench_root/run\" \\\n  --trials 2 \\\n  --timeout-seconds 1200 \\\n  --smoke-timeout-seconds 600\n\npython research/omlx-qwen38-pi-deepseek-harness-2026-08-23/revalidate_checkers.py \\\n  \"$bench_root/run/benchmark/results.jsonl\" \\\n  --tasks-root \"$bench_root/openbench/tasks\" \\\n  --raw-root \"$bench_root/run/benchmark/raw\" \\\n  --evidence-output \"$bench_root/checker-revalidation.json\"\n\npython research/omlx-qwen38-pi-deepseek-harness-2026-08-23/analyze.py \\\n  \"$bench_root/run/benchmark/results.jsonl\" \\\n  --manifest \"$bench_root/run/manifest.json\" \\\n  --checker-revalidation \"$bench_root/checker-revalidation.json\" \\\n  --clean-output \"$bench_root/clean.json\"\n\npytest -q tests/test_omlx_pi_deepseek_harness_benchmark.py\n```\n\n`--tool-root`\n\nmust contain the exact readable subdirectories `pi/`\n\nand `dsh/`\n\n;\nthe runner rejects overlap among tool, run, and hidden-task roots. Raw\ntranscripts and proxy JSONL stay in the temporary run root and must not be\npublished without a separate content review. The checked-in clean artifact was\ngenerated by [ analyze.py](/promptdriven/pdd/blob/main/research/omlx-qwen38-pi-deepseek-harness-2026-08-23/analyze.py), whose whitelist and forbidden-material\nscan fail closed on credentials and absolute user/temp paths.", "url": "https://wpnews.pro/news/pi-vs-the-official-deepseek-harness-on-the-same-local-model-qwen3-8-27b", "canonical_source": "https://github.com/promptdriven/pdd/blob/main/research/omlx-qwen38-pi-deepseek-harness-2026-08-23/README.md", "published_at": "2026-08-25 12:00:50+00:00", "updated_at": "2026-08-25 12:15:56.976395+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["Pi", "DeepSeek Harness", "Qwen3.8-27B", "taskflow", "make-ci-green", "add-feature", "webcore"], "alternates": {"html": "https://wpnews.pro/news/pi-vs-the-official-deepseek-harness-on-the-same-local-model-qwen3-8-27b", "markdown": "https://wpnews.pro/news/pi-vs-the-official-deepseek-harness-on-the-same-local-model-qwen3-8-27b.md", "text": "https://wpnews.pro/news/pi-vs-the-official-deepseek-harness-on-the-same-local-model-qwen3-8-27b.txt", "jsonld": "https://wpnews.pro/news/pi-vs-the-official-deepseek-harness-on-the-same-local-model-qwen3-8-27b.jsonld"}}