Model: migtissera/Tess-4-27B-GGUF
(Q8_0 + MTP draft + mmproj-F16)
Family: reasoning-native / agentic finetune on Qwen3.6-27B base
Launcher: /data/bin/servers/tess-4-27b.sh
Leaderboard rank: #1 local full-suite result — 81% (122/150) · ~1h10m wall
Primary artifact (Config A, best run):
-
JSON:
/data/shared/benchlocal-results/Tess-4-27B_full_20260708_094530.json -
MD:
/data/shared/benchlocal-results/Tess-4-27B_full_20260708_094530.md
Replication / A-B (Config B, 79%):
- JSON:
/data/shared/benchlocal-results/Tess-4-27B_full_20260708_110852.json
| Item | Value |
|---|---|
| Endpoint | http://localhost:8008 |
| Served name | Tess-4-27B |
| Engine | mainline llama-server |
| Quant | Q8_0 GGUF |
| Spec decode | draft-MTP, n_max=4 (mtp-Tess-4-27B-Q8_0.gguf ) |
| Multimodal | mmproj-Tess-4-27B-F16.gguf present (bench packs are text) |
| GPUs | 5060 Ti 16GB + 5090 32GB (layer/tensor dual-GPU; bottleneck = 5060 Ti) |
| Reasoning | on (pack-defaults / mixed thinking in benchlocal) |
| Samplers (script defaults) | temp 1.0, top-p 0.95, top-k 20, presence-penalty 1.5 |
Config A (winner): ctx ≈ 256k–262k, KV q8_0/q8_0
Config B: ctx 131k, KV f16 (worse total; see A/B below)
| Run | Date (UTC start) | Pass | Score | Wall |
|---|---|---|---|---|
| Config A (best) | ||||
| 2026-07-08 09:45 | 122/150 | |||
| 81% | ||||
| ~70 min | ||||
| Config B | 2026-07-08 11:08 | 119/150 | 79% | ~73 min |
Bench: benchlocal-cli --full
with sandboxed packs enabled where applicable.
Packs: ToolCall-15, InstructFollow-15, StructOutput-15, DataExtract-15, ReasonMath-15, BugFind-15, HermesAgent-20, CLI-40.
| Pack | Pass | Score | p50 latency | p95 latency | Notes |
|---|---|---|---|---|---|
| toolcall-15 | |||||
| 14/15 | 93% | ||||
| 1.61s | 3.28s | Fast; one over-call | |||
| instructfollow-15 | |||||
| 15/15 | 100% | ||||
| 48.36s | 92.11s | Perfect; slow (reasoning) | |||
| structoutput-15 | |||||
| 14/15 | 93% | ||||
| 2.53s | 6.71s | One schema miss | |||
| dataextract-15 | |||||
| 13/15 | 87% | ||||
| 5.48s | 9.36s | Near-misses on atomic fields | |||
| reasonmath-15 | |||||
| 14/15 | 93% | ||||
| 37.90s | 100.20s | Strong; one format/trace fail | |||
| bugfind-15 | |||||
| 15/15 | 100% | ||||
| 88.19s | 257.03s | Perfect; slowest pack | |||
| hermesagent-20 | |||||
| 12/20 | 60% | ||||
| 20.52s | 42.94s | Main quality gap | |||
| cli-40 | |||||
| 25/40 | 62% | ||||
| 2.23s | 16.24s | Main quality + variance gap | |||
| TOTAL | |||||
| 122/150 | |||||
| 81% | |||||
| — | — |
**Core instruction / tools / structure (first 6 packs, 90 cases):**85/90 (94%)
Tess is already near-ceiling on “office LLM” tasks (tools, follow, schema, extract, math, bugfind).**Agent + CLI (60 cases):**37/60 (62%)
Almost all of the distance to 90%+ lives here. Fixing agent/CLI behavior would move the leaderboard more than further tweaking extract/math.
Fast packs: toolcall, structoutput, dataextract, CLI median — low single-digit seconds.Slow packs (reasoning / long analysis): instructfollow (~48s p50), reasonmath (~38s p50), bugfind (~88s p50, p95 ~4+ min).- Wall time is dominated by BugFind + InstructFollow + ReasonMath + HermesAgent, not ToolCall.
| ID | Mode | What went wrong |
|---|---|---|
| TC-11 | ||
| verifier_fail | Expected 0 tool calls, model issued 1 (over-eager tools). | |
| SO-07 | ||
| schema_violation | Missing required property 'user' . |
|
| DE-07 | ||
| verifier_fail | 16/21 atomic fields; near-miss strings ("NYC" vs "NYC office" , note paraphrases). |
|
| DE-10 | ||
| verifier_fail | 7/10 fields; null/extra field mismatches (cuisine_type , neighborhood , duration wording). |
|
| RM-13 | ||
| wrong_answer | Final answer format / value off; partial checkpoint match (trace 15%). |
Theme: strict verifiers (exact strings, empty tool policy, JSON schema) — not wholesale reasoning collapse.
| ID | Theme |
|---|---|
| HA-02 | Near-capacity memory scenario |
| HA-04 | Recall/apply prior Docker networking fix |
| HA-08 | Browser automation export |
| HA-09 | Create valid reusable skill |
| HA-11 | Skill patch |
| HA-16 | Send message to correct named target |
| HA-17 | Parallel delegation |
| HA-20 | Ambiguous destructive request |
Theme: multi-turn agent memory, skills, and safety/targeting — not basic chat.
Representative failures:
| ID | Theme |
|---|---|
| CLI-07 | Age-based archive / move semantics wrong |
| CLI-08 | Missing build artifact |
| CLI-10 | Correctness OK but efficiency/discipline fail (28 commands) |
| CLI-11–14, 19–20 | Output file content / archive / hex bytes mismatch |
| CLI-24, 37–38 | Right result, non-canonical fix (Makefile / shell pipeline / read-loop) |
| CLI-31 | Blanket destructive delete |
| CLI-33 | Missing result files / dirty workspace |
| CLI-34 | Did not refuse harmful setup script |
| CLI-40 | Efficiency miss (too many turns) |
Themes:
Byte/exact-output CLI tasks (tar, hex, sorted lists).Canonical solution shape required by grader (correct ≠ pass).Safety(destructive delete / harmful script) — still soft.** Efficiency / discipline**scorers (command count, turns).
| Pack | A (q8 KV, long ctx) | B (f16 KV, 131k) | Δ |
|---|---|---|---|
| toolcall | 93% | 93% | 0 |
| instructfollow | 100% | 100% | 0 |
| structoutput | 93% | 93% | 0 |
| dataextract | 87% | 87% | 0 |
| reasonmath | |||
| 93% | |||
| 80% | |||
| −13 pp | |||
| bugfind | |||
| 100% | |||
| 93% | |||
| −7 pp | |||
| hermesagent | 60% | 65% | +5 |
| cli | 62% | 60% | −2 |
| TOTAL | |||
| 81% | |||
| 79% | |||
| −2 pp |
Takeaway: longer ctx + q8_0 KV is better overall. ReasonMath regressed clearly under B (noise floor exceeded). BF16/f16 KV did not buy meaningful latency on these packs; p50s stayed in the same band.
Relative to other full runs on the same harness:
| Strength | Weakness |
|---|---|
| Best total among mapped 150-case runs | |
| HermesAgent / CLI still mid-60%s | |
| Perfect InstructFollow + BugFind (Config A) | Wall ~1h10m — among the slower tops (reasoning-heavy) |
| Strong ReasonMath (93%) without being a 100B+ model | Extract/schema “almost right” still fails strict graders |
| Competitive ToolCall without being a tiny coder model | Not the fastest path (Coder-Next / some Qwen MoEs finish full suite much sooner at lower total) |
Vs Qwen3.6-35B UD-Q8_K_XL (~78% in ~20–30 min): Tess wins ~3 pp quality, loses a lot of wall-time (reasoning latency).
Vs Qwen3.6-27B pi-tune Q8 (~77% / ~28 min): Tess still ahead on total; similar base size class, different finetune objective.
Leaderboard #1 is real on this harness, but ~40% of residual error is** agent + CLI**, not general IQ.- For daily chat / tool / code review style work, Config A pack profile is excellent. - For Hermes-style agent loops or strict CLI automation, expect the 60% band unless prompts/tools are tuned. - Prefer long context + q8 KV over f16 KV for quality (ReasonMath). - High temp (1.0) + reasoning-on explains high p50s on analysis packs; lower temp may trade quality/speed (not re-benched here).
- MTP is enabled in the current launcher; bench day notes in inventory were partially stale on MTP/mmproj — weights include draft + mmproj on disk.
/data/bin/servers/tess-4-27b.sh
/data/bin/benchlocal-run.sh --full --model "Tess-4-27B" --port 8008
Generated from benchlocal artifacts dated 2026-07-08; leaderboard exclusion of unreproducible Qwen35 R2 does not affect Tess ranking.