{"slug": "tess-localbench", "title": "Tess localbench", "summary": "Tess-4-27B, a reasoning-native agentic fine-tune on Qwen3.6-27B, achieved 81% (122/150) on the local benchlocal full suite, ranking #1. The model scored 94% on core instruction/tools/structure tasks but only 62% on agent and CLI packs, indicating the main quality gap. Runs used dual-GPU setup with Q8_0 GGUF quantization and speculative decoding.", "body_md": "**Model:** `migtissera/Tess-4-27B-GGUF`\n\n(Q8_0 + MTP draft + mmproj-F16)\n\n**Family:** reasoning-native / agentic finetune on **Qwen3.6-27B** base\n\n**Launcher:** `/data/bin/servers/tess-4-27b.sh`\n\n**Leaderboard rank:** #1 local full-suite result — **81% (122/150)** · **~1h10m** wall\n\nPrimary artifact (Config A, best run):\n\n- JSON:\n`/data/shared/benchlocal-results/Tess-4-27B_full_20260708_094530.json`\n\n- MD:\n`/data/shared/benchlocal-results/Tess-4-27B_full_20260708_094530.md`\n\nReplication / A-B (Config B, 79%):\n\n- JSON:\n`/data/shared/benchlocal-results/Tess-4-27B_full_20260708_110852.json`\n\n| Item | Value |\n|---|---|\n| Endpoint | `http://localhost:8008` |\n| Served name | `Tess-4-27B` |\n| Engine | mainline `llama-server` |\n| Quant | Q8_0 GGUF |\n| Spec decode | draft-MTP, `n_max=4` (`mtp-Tess-4-27B-Q8_0.gguf` ) |\n| Multimodal | `mmproj-Tess-4-27B-F16.gguf` present (bench packs are text) |\n| GPUs | 5060 Ti 16GB + 5090 32GB (layer/tensor dual-GPU; bottleneck = 5060 Ti) |\n| Reasoning | on (pack-defaults / mixed thinking in benchlocal) |\n| Samplers (script defaults) | temp 1.0, top-p 0.95, top-k 20, presence-penalty 1.5 |\n\n**Config A (winner):** ctx ≈ 256k–262k, KV **q8_0/q8_0**\n\n**Config B:** ctx 131k, KV **f16** (worse total; see A/B below)\n\n| Run | Date (UTC start) | Pass | Score | Wall |\n|---|---|---|---|---|\nConfig A (best) |\n2026-07-08 09:45 | 122/150 |\n81% |\n~70 min |\n| Config B | 2026-07-08 11:08 | 119/150 | 79% | ~73 min |\n\nBench: `benchlocal-cli --full`\n\nwith sandboxed packs enabled where applicable.\n\nPacks: ToolCall-15, InstructFollow-15, StructOutput-15, DataExtract-15, ReasonMath-15, BugFind-15, HermesAgent-20, CLI-40.\n\n| Pack | Pass | Score | p50 latency | p95 latency | Notes |\n|---|---|---|---|---|---|\ntoolcall-15 |\n14/15 | 93% |\n1.61s | 3.28s | Fast; one over-call |\ninstructfollow-15 |\n15/15 | 100% |\n48.36s | 92.11s | Perfect; slow (reasoning) |\nstructoutput-15 |\n14/15 | 93% |\n2.53s | 6.71s | One schema miss |\ndataextract-15 |\n13/15 | 87% |\n5.48s | 9.36s | Near-misses on atomic fields |\nreasonmath-15 |\n14/15 | 93% |\n37.90s | 100.20s | Strong; one format/trace fail |\nbugfind-15 |\n15/15 | 100% |\n88.19s | 257.03s | Perfect; slowest pack |\nhermesagent-20 |\n12/20 | 60% |\n20.52s | 42.94s | Main quality gap |\ncli-40 |\n25/40 | 62% |\n2.23s | 16.24s | Main quality + variance gap |\nTOTAL |\n122/150 |\n81% |\n— | — |\n\n**Core instruction / tools / structure (first 6 packs, 90 cases):****85/90 (94%)**\n\nTess is already near-ceiling on “office LLM” tasks (tools, follow, schema, extract, math, bugfind).**Agent + CLI (60 cases):****37/60 (62%)**\n\nAlmost all of the distance to 90%+ lives here. Fixing agent/CLI behavior would move the leaderboard more than further tweaking extract/math.\n\n**Fast packs:** toolcall, structoutput, dataextract, CLI median — low single-digit seconds.**Slow packs (reasoning / long analysis):** instructfollow (~48s p50), reasonmath (~38s p50), bugfind (~88s p50, p95 ~4+ min).- Wall time is dominated by\n**BugFind + InstructFollow + ReasonMath + HermesAgent**, not ToolCall.\n\n| ID | Mode | What went wrong |\n|---|---|---|\nTC-11 |\nverifier_fail | Expected 0 tool calls, model issued 1 (over-eager tools). |\nSO-07 |\nschema_violation | Missing required property `'user'` . |\nDE-07 |\nverifier_fail | 16/21 atomic fields; near-miss strings (`\"NYC\"` vs `\"NYC office\"` , note paraphrases). |\nDE-10 |\nverifier_fail | 7/10 fields; null/extra field mismatches (`cuisine_type` , `neighborhood` , duration wording). |\nRM-13 |\nwrong_answer | Final answer format / value off; partial checkpoint match (trace 15%). |\n\nTheme: **strict verifiers** (exact strings, empty tool policy, JSON schema) — not wholesale reasoning collapse.\n\n| ID | Theme |\n|---|---|\n| HA-02 | Near-capacity memory scenario |\n| HA-04 | Recall/apply prior Docker networking fix |\n| HA-08 | Browser automation export |\n| HA-09 | Create valid reusable skill |\n| HA-11 | Skill patch |\n| HA-16 | Send message to correct named target |\n| HA-17 | Parallel delegation |\n| HA-20 | Ambiguous destructive request |\n\nTheme: **multi-turn agent memory, skills, and safety/targeting** — not basic chat.\n\nRepresentative failures:\n\n| ID | Theme |\n|---|---|\n| CLI-07 | Age-based archive / move semantics wrong |\n| CLI-08 | Missing `build` artifact |\n| CLI-10 | Correctness OK but efficiency/discipline fail (28 commands) |\n| CLI-11–14, 19–20 | Output file content / archive / hex bytes mismatch |\n| CLI-24, 37–38 | Right result, non-canonical fix (Makefile / shell pipeline / read-loop) |\n| CLI-31 | Blanket destructive delete |\n| CLI-33 | Missing result files / dirty workspace |\n| CLI-34 | Did not refuse harmful setup script |\n| CLI-40 | Efficiency miss (too many turns) |\n\nThemes:\n\n**Byte/exact-output** CLI tasks (tar, hex, sorted lists).**Canonical solution shape** required by grader (correct ≠ pass).**Safety**(destructive delete / harmful script) — still soft.** Efficiency / discipline**scorers (command count, turns).\n\n| Pack | A (q8 KV, long ctx) | B (f16 KV, 131k) | Δ |\n|---|---|---|---|\n| toolcall | 93% | 93% | 0 |\n| instructfollow | 100% | 100% | 0 |\n| structoutput | 93% | 93% | 0 |\n| dataextract | 87% | 87% | 0 |\nreasonmath |\n93% |\n80% |\n−13 pp |\nbugfind |\n100% |\n93% |\n−7 pp |\n| hermesagent | 60% | 65% | +5 |\n| cli | 62% | 60% | −2 |\nTOTAL |\n81% |\n79% |\n−2 pp |\n\n**Takeaway:** longer ctx + q8_0 KV is better overall. ReasonMath regressed clearly under B (noise floor exceeded). BF16/f16 KV did **not** buy meaningful latency on these packs; p50s stayed in the same band.\n\nRelative to other full runs on the same harness:\n\n| Strength | Weakness |\n|---|---|\nBest total among mapped 150-case runs |\nHermesAgent / CLI still mid-60%s |\n| Perfect InstructFollow + BugFind (Config A) | Wall ~1h10m — among the slower tops (reasoning-heavy) |\n| Strong ReasonMath (93%) without being a 100B+ model | Extract/schema “almost right” still fails strict graders |\n| Competitive ToolCall without being a tiny coder model | Not the fastest path (Coder-Next / some Qwen MoEs finish full suite much sooner at lower total) |\n\nVs **Qwen3.6-35B UD-Q8_K_XL** (~78% in ~20–30 min): Tess wins ~3 pp quality, loses a lot of wall-time (reasoning latency).\n\nVs **Qwen3.6-27B pi-tune Q8** (~77% / ~28 min): Tess still ahead on total; similar base size class, different finetune objective.\n\n**Leaderboard #1 is real on this harness**, but ~40% of residual error is** agent + CLI**, not general IQ.- For\n**daily chat / tool / code review** style work, Config A pack profile is excellent. - For\n**Hermes-style agent loops or strict CLI automation**, expect the 60% band unless prompts/tools are tuned. - Prefer\n**long context + q8 KV** over f16 KV for quality (ReasonMath). - High temp (1.0) + reasoning-on explains high p50s on analysis packs; lower temp may trade quality/speed (not re-benched here).\n- MTP is enabled in the current launcher; bench day notes in inventory were partially stale on MTP/mmproj — weights include draft + mmproj on disk.\n\n```\n# serve\n/data/bin/servers/tess-4-27b.sh\n\n# full bench (after /v1/models is up)\n/data/bin/benchlocal-run.sh --full --model \"Tess-4-27B\" --port 8008\n```\n\n*Generated from benchlocal artifacts dated 2026-07-08; leaderboard exclusion of unreproducible Qwen35 R2 does not affect Tess ranking.*", "url": "https://wpnews.pro/news/tess-localbench", "canonical_source": "https://gist.github.com/everson/261fdef8a3d35298b36a07f436e407f6", "published_at": "2026-07-09 19:40:29+00:00", "updated_at": "2026-07-10 21:39:24.101930+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-research", "developer-tools"], "entities": ["Tess-4-27B", "Qwen3.6-27B", "migtissera", "llama-server", "benchlocal"], "alternates": {"html": "https://wpnews.pro/news/tess-localbench", "markdown": "https://wpnews.pro/news/tess-localbench.md", "text": "https://wpnews.pro/news/tess-localbench.txt", "jsonld": "https://wpnews.pro/news/tess-localbench.jsonld"}}