{"slug": "jev-vs-sonnet-triage-the-9-questions-and-criteria", "title": "Jev vs Sonnet triage: the 9 questions and criteria", "summary": "A developer triaged 1,895 open GitHub issues across n8n, Supabase, Cal.com, Appwrite and Home Assistant using a fixed set of nine questions — covering issue kind, reproducibility, version reporting, expected behavior, actionability, severity, frustration, data integrity and workarounds — asked in a single call per issue. Comparing the \"kind\" classification against maintainer-applied labels on 767 issues, the Jev model scored 96.1% and Claude Sonnet 94.0%.", "body_md": "| | # The nine questions used to triage 1,895 open GitHub issues | \n| | # (n8n, Supabase, Cal.com, Appwrite, Home Assistant) with Jev and Claude Sonnet. | \n| | # | \n| | # Both models got exactly this text: same questions, same one-line criteria. | \n| | # All nine are asked in one call per issue. The issue goes in as: | \n| | # REPOSITORY: | \n| | # TITLE: | \n| | # <body> | \n| | # | \n| | # Types: \"choice\" picks one option, \"noul\" is a yes/no probability, | \n| | # \"score\" places the issue on an ordered scale. | \n| | # | \n| | # Check: \"kind\" was compared with the labels maintainers put on the issues | \n| | # themselves (767 issues). Jev 96.1%, Sonnet 94.0%. | \n| | QUESTIONS = { | \n| | \"kind\": {\"type\": \"choice\", | \n| | \"instructions\": \"What kind of thing is this issue?\", | \n| | \"criteria\": { | \n| | \"bug\": \"Reports something broken or behaving wrongly.\", | \n| | \"feature\": \"Asks for new or changed capability.\", | \n| | \"docs\": \"Is about documentation being wrong, missing or unclear.\", | \n| | \"question\": \"Asks how to do something, with no defect claimed.\"}}, | \n| | \"repro\": {\"type\": \"noul\", | \n| | \"instructions\": \"Does it contain steps another person could follow to reproduce the problem?\", | \n| | \"criteria\": {\"true\": \"Concrete steps, input, or a workflow another person could replay.\", | \n| | \"false\": \"Describes the symptom only, with no replayable steps.\"}}, | \n| | \"version\": {\"type\": \"noul\", | \n| | \"instructions\": \"Does it state which version or environment this happened on?\", | \n| | \"criteria\": {\"true\": \"Names a version, build, deployment mode or environment.\", | \n| | \"false\": \"No version or environment stated anywhere.\"}}, | \n| | \"expected\": {\"type\": \"noul\", | \n| | \"instructions\": \"Does it say what the author expected to happen instead?\", | \n| | \"criteria\": {\"true\": \"States the expected behaviour explicitly.\", | \n| | \"false\": \"Says what went wrong but never what should have happened.\"}}, | \n| | \"actionable\": {\"type\": \"noul\", | \n| | \"instructions\": \"Could an engineer start work on this as written, without asking the author anything first?\", | \n| | \"criteria\": {\"true\": \"Everything needed to begin is already here.\", | \n| | \"false\": \"Someone would have to go back to the author before starting.\"}}, | \n| | \"severity\": {\"type\": \"score\", | \n| | \"instructions\": \"How badly is this blocking the person who filed it?\", | \n| | \"criteria\": [\"cosmetic or a minor annoyance\", | \n| | \"works but is painful or needs a workaround\", | \n| | \"blocks their work entirely or loses data\"]}, | \n| | \"frustration\": {\"type\": \"noul\", | \n| | \"instructions\": \"Does the author sound frustrated enough to be at risk of leaving the product?\", | \n| | \"criteria\": {\"true\": \"Expresses frustration, lost time, or mentions leaving or switching.\", | \n| | \"false\": \"Neutral or constructive in tone.\"}}, | \n| | \"data\": {\"type\": \"noul\", | \n| | \"instructions\": \"Does this involve data being lost, corrupted or returned wrong?\", | \n| | \"criteria\": {\"true\": \"Data loss, corruption, duplication or incorrect results are involved.\", | \n| | \"false\": \"No data integrity concern.\"}}, | \n| | \"workaround\": {\"type\": \"noul\", | \n| | \"instructions\": \"Does the author already describe a workaround?\", | \n| | \"criteria\": {\"true\": \"A workaround or partial fix is described.\", | \n| | \"false\": \"No workaround is mentioned.\"}}, | \n| | } |", "url": "https://wpnews.pro/news/jev-vs-sonnet-triage-the-9-questions-and-criteria", "canonical_source": "https://gist.github.com/nicolavalenti/69ffc36f9a36c5692e83dcce7a369a6a", "published_at": "2026-09-23 10:28:14+00:00", "updated_at": "2026-09-24 04:59:42.251196+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "mlops"], "entities": ["Jev", "Claude Sonnet", "GitHub", "n8n", "Supabase", "Cal.com", "Appwrite", "Home Assistant"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/jev-vs-sonnet-triage-the-9-questions-and-criteria", "markdown": "https://wpnews.pro/news/jev-vs-sonnet-triage-the-9-questions-and-criteria.md", "text": "https://wpnews.pro/news/jev-vs-sonnet-triage-the-9-questions-and-criteria.txt", "jsonld": "https://wpnews.pro/news/jev-vs-sonnet-triage-the-9-questions-and-criteria.jsonld"}}