{"slug": "jev-and-the-problem-with-ai-that-always-has-an-answer", "title": "Jev and the Problem With AI That Always Has an Answer", "summary": "A developer integrated Jev, a decision model that returns structured probabilities instead of generated prose, into a resume-review tool after finding that LLM-written critiques produced uniform, authoritative-sounding feedback and fabricated metrics like a 25% efficiency gain. The rebuilt pipeline uses a parser, deterministic checks and bounded judgments, then lets application code decide whether any finding is worth surfacing and points users at the exact editable line. The developer's stated lesson is that the hard part of AI review is getting the model to stay quiet until it has earned the right to speak.", "body_md": "The biggest improvement was learning when to stay quiet.\n\n*This is our experience while integrating **Jev** to a use case that's both cost effective and accurate when it comes to llm. The hard part wasn't getting AI to say something smart. It was getting it to stay quiet when it wasn't sure.*\n\n**Your resume score is probably lying to you.**\n\nNobody needs another tool that gives a resume 62 out of 100 and then says:\n\nStrengthen your bullet points.\n\nI know because that's roughly what our own reviewer used to do.\n\nThe fix wasn't a better prompt.\n\nIt was changing what we asked the model to do — and, more importantly, what we allowed it to say.\n\nThe reviewer is now built around **Jev**, a decision model designed around structured judgments rather than generating another paragraph of critique.\n\nInstead of asking it to write advice, we ask it narrower questions and get probabilities back.\n\nThat distinction turned out to matter a lot more than I expected.\n\nThe honest version of this story isn't:\n\nWe added AI to our resume builder.\n\nIt's that we shipped a reviewer, watched people bounce off it, and realised that the difficult part of AI review isn't getting a model to say something intelligent.\n\nIt's getting it to **stay quiet until it has earned the right to speak**.\n\nThat's probably the one idea from this project I'd keep if I had to throw everything else away.\n\nThe obvious implementation looks something like this:\n\n```\nresume\n   ↓\nlarge prompt\n   ↓\nLLM\n   ↓\nlong critique\n```\n\nPaste in the resume.\n\nAsk for feedback.\n\nRender whatever comes back.\n\nA lot of AI review tools follow some version of this pattern, and it creates a few problems.\n\nAsk a model for prose and you get prose with roughly the same authoritative tone throughout.\n\nA 60% hunch and a 99% certainty can sound almost identical.\n\nThe person reading it has no idea which advice actually matters.\n\nSo they either trust everything or start trusting nothing.\n\nConsider:\n\nRate this resume against what a hiring manager looks for.\n\nThat's one enormous question.\n\nWhat's the correct answer?\n\nThere isn't really one.\n\nRun the same resume twice and the review may move around.\n\nIf the result changes, what do you tune?\n\nWhich part failed?\n\nWhat test should fail?\n\nYou don't really know.\n\nAsk a model to critique something and it will generally find something to critique.\n\nGive it a bullet like:\n\nResponsible for maintaining the product workflow.\n\nand a sufficiently helpful model may suggest turning it into something like:\n\nImproved the product workflow, increasing efficiency by 25%.\n\nWhere did 25% come from?\n\nNowhere.\n\nIt sounds better.\n\nIt is also fiction.\n\nThat's a particularly nasty failure mode when the output eventually ends up in a document someone sends to an employer.\n\nYour bullets under the Product role could be stronger.\n\nMaybe that's true.\n\nBut it's a weather report.\n\nThe user still has to figure out:\n\nThe machine has technically provided feedback while leaving the difficult part to the human.\n\nThe failure, I think, is structural.\n\nWe gave the model both the job of **judging** and the job of **writing**.\n\nFor this product, most of the writing turned out not to need a model at all.\n\nThe biggest change was surprisingly simple:\n\n**Stop letting the model write the feedback the user reads.**\n\nThe old shape was:\n\n```\nResume\n   ↓\nLLM\n   ↓\nOne long generated critique\n```\n\nThe new shape is closer to:\n\n```\nResume\n   ↓\nDeterministic checks + bounded judgments\n   ↓\nEvidence + uncertainty\n   ↓\nDecide whether anything is worth surfacing\n   ↓\nPoint at the exact editable line\n```\n\nThe flow today is roughly this.\n\nA parser turns the PDF or DOCX into structured resume data.\n\nNormal code handles anything that doesn't require intelligence:\n\nThen the redacted resume text goes to Jev.\n\nInstead of asking:\n\nIs this a good resume?\n\nwe ask smaller questions about particular pieces of text.\n\nJev returns structured judgments rather than writing the explanation itself.\n\nApplication code then decides whether those signals add up to something worth telling a person.\n\nIf they do, the system attaches an explanation that was written beforehand.\n\nThe UI points to the relevant line.\n\nThe person can act on it.\n\nThat's it.\n\nBecause the model gives us decisions rather than paragraphs, we can have a rule for when the system should say something at all.\n\n**That's the whole game.**\n\nWe stopped asking things like:\n\nIs this a good bullet?\n\nThat's almost impossible to calibrate.\n\nInstead, the problem gets broken into smaller judgments.\n\nFor example:\n\nThe important part isn't those particular questions.\n\nIt's the decomposition.\n\nOnce the judgments are structured, ordinary code can decide whether there's enough evidence to surface something.\n\nThat part isn't AI.\n\nIt's just logic.\n\nAnd that's exactly why it's useful.\n\nYou can write tests for it.\n\nYou can watch it fail.\n\nYou can change the rule.\n\nYou can compare the result before and after.\n\n**You can test narrow questions. Broad opinions are much harder to test.**\n\nEvery finding the user sees is built from roughly three pieces:\n\nThe model doesn't invent metrics.\n\nIt doesn't manufacture an example accomplishment.\n\nIt doesn't decide that you somehow improved revenue by 31%.\n\nIt contributes a signal.\n\nThis turned out to be one of the simplest anti-hallucination techniques in the whole system.\n\nYou can put:\n\nDo not invent metrics.\n\ninside a prompt.\n\nThat's a request.\n\nOr you can design the system so the model **cannot put invented prose in front of the user in the first place**.\n\nThat's a constraint.\n\nI trust the second one more.\n\nThis took me a while to appreciate.\n\nIf the signals don't agree strongly enough, we don't show anything.\n\nNo card.\n\nNo weak suggestion.\n\nNo:\n\nThis could potentially maybe benefit from...\n\nNothing.\n\nAt first this feels wrong.\n\nYou built an AI reviewer. Surely it should review things.\n\nBut a system that always produces feedback has a strange incentive: it has to find something wrong even when there isn't much worth saying.\n\nSo we deliberately made silence meaningful.\n\nIf the system is uncertain, the user sees less.\n\nWhen it is confident enough to surface something, we show the lines that seem most worth looking at first.\n\nThe goal isn't:\n\nHere are all the problems with your resume.\n\nIt's closer to:\n\nIf you're going to fix one thing, start here.\n\nA strong resume should produce less criticism.\n\nThat shouldn't be considered a failure of the reviewer.\n\nIt should be considered success.\n\nThe first version had three outcomes:\n\n```\nPassed\nUnclear\nCould improve\n```\n\n\"Unclear\" seemed intellectually honest.\n\nIt was awful.\n\nUsers didn't read it as:\n\nThe model is responsibly communicating uncertainty.\n\nThey read it as:\n\nThe tool doesn't know what it's doing.\n\nWorse, once they saw enough uncertain rows, they started distrusting the confident ones too.\n\nSo we removed it from the UI.\n\nNow a visible result is generally either:\n\n**Passed**\n\nor:\n\n**Could improve**\n\nIf the evaluator can't make a sufficiently useful call, the row doesn't need to exist.\n\nThe uncertainty still exists internally.\n\nWe just don't turn uncertainty into useless UI.\n\nThis was probably my favourite bug in the project.\n\nWe were evaluating one part of a resume on an ordered scale.\n\nStrong examples were occasionally being flagged while some weaker ones were getting through.\n\nMy first assumption was obvious:\n\nThe questions must be badly written.\n\nThey weren't.\n\nThe mistake was in how we interpreted confidence.\n\nSuppose the possible result is an ordered scale:\n\n```\n1  2  3  4  5\n```\n\nA model can be uncertain between level 2 and level 3.\n\nThat sounds like low confidence.\n\nBut imagine that the decision your application actually cares about is here:\n\n```\n1  2  3 | 4  5\n          ^\n       boundary\n```\n\nIf almost all of the probability is split between 2 and 3, does it really matter that the model can't decide which exact level it is?\n\nFor the application-level decision, not much.\n\nBoth answers are on the same side of the boundary.\n\nSo we stopped thinking only in terms of:\n\nHow confident is the model in the winning answer?\n\nand started thinking more about:\n\nWhere does the probability mass sit relative to the decision we actually care about?\n\nThose are not the same question.\n\nThat sounds obvious after you've found the bug.\n\nIt wasn't obvious before.\n\nIf you're building anything around graded model output, this distinction is worth remembering:\n\n**A confidence field is not necessarily a verdict.**\n\nSometimes the distribution tells you far more than the winning label.\n\nA resume reviewer has an uncomfortable property:\n\nIt needs to read fairly personal text.\n\nBut it doesn't need all of it.\n\nThe reviewer needs to see your sentences.\n\nIt doesn't need to know your name.\n\nBefore evaluation, we create a separate sanitized representation with identifying information removed.\n\nThings like:\n\nThe original is what the interface uses when it needs to show the user their own line or edit it.\n\nThe evaluator works against the redacted representation.\n\nThat separation also made another feature easier to reason about: rechecking.\n\nWhen somebody comes back after fixing a finding, the question should be:\n\nDid I fix the thing you told me was wrong?\n\nNot:\n\nCan you inspect my resume again and discover another twelve things to complain about?\n\nSo the recheck is deliberately scoped to the previous findings.\n\nIt can say the issue appears fixed.\n\nIt can say it isn't fixed.\n\nIt can say it looks partially fixed.\n\nIt can admit that it can't tell.\n\nWhat it shouldn't do is turn every revision into another endless round of criticism.\n\nThis is probably the more useful part.\n\nIt reported broad areas of resume quality.\n\nImpact.\n\nClarity.\n\nStructure.\n\nThat looked nice.\n\nIt also wasn't very actionable.\n\nTelling somebody:\n\nYour experience section could demonstrate more impact.\n\ndoesn't solve much.\n\nWhich sentence?\n\nWhy?\n\nWhat do I change?\n\nThe biggest product improvement wasn't a better model.\n\nIt was changing the **unit of feedback**.\n\nFrom:\n\nSomething in this section is weak.\n\nto:\n\nThis specific line is worth looking at.\n\n**The model wasn't necessarily the problem. The unit of feedback was.**\n\nThis produced some wonderfully wrong results.\n\nA resume summary might say that somebody led a certain type of project or worked across a particular domain.\n\nIf you evaluate that sentence alone, those claims can look unsupported.\n\nAnd the evaluator may correctly notice that.\n\nThe problem is that the evidence is sitting three inches lower on the page.\n\nOnce the summary was evaluated with the relevant experience available as context, a lot of those weird results disappeared.\n\nThe lesson wasn't:\n\nThe model needs to be smarter.\n\nIt was:\n\nWe didn't give the decision enough evidence.\n\nThere's a difference.\n\nA six-month contract, a parental leave, time spent caring for somebody, unemployment and a deliberate break can all create superficially similar timeline patterns.\n\nText alone doesn't always tell you what happened.\n\nThis is exactly the kind of area where being \"helpful\" can become harmful quickly.\n\nSo the system is deliberately conservative here.\n\nIt shouldn't tell somebody that a gap itself is a problem.\n\nIt definitely shouldn't tell them to invent experience to fill it.\n\nAt most, there are situations where a short and honest line of context may make something easier for a human reader to understand.\n\nI still consider this an area where false positives matter more than coverage.\n\nThis is the boring but important answer.\n\nWe're still going through representative resumes and recording:\n\nThe thresholds aren't sacred constants delivered from a mountain.\n\nThey're informed guesses backed by tests, and they'll move as we learn more.\n\nI'd rather say that than give the system a level of scientific authority it hasn't earned.\n\nIf I were building another system like this tomorrow:\n\nGeneration isn't automatically the valuable part.\n\nSometimes the useful intelligence is the decision.\n\nSmaller decisions are easier to inspect, test and combine.\n\nA system doesn't become smarter by always having an answer.\n\nIf you can't point to what the person can actually change, think carefully about whether the criticism is useful.\n\nDon't automatically translate uncertainty into a paragraph full of hedge words.\n\nUse it.\n\nThe system described here is the reviewer behind [FreeResume's \"What's Wrong With My Resume\" tool](https://freeresume.site/whats-wrong-with-my-resume).\n\nI'm much more interested in the cases where it gets something wrong than the ones where it gets something obviously right.\n\nBecause if you can identify exactly **why** a decision was wrong, you have something you can improve.\n\nAnd that's ultimately the difference I care about.\n\nA model that always has an answer is impressive.\n\n**A system that knows when the answer isn't good enough to show is useful.**", "url": "https://wpnews.pro/news/jev-and-the-problem-with-ai-that-always-has-an-answer", "canonical_source": "https://dev.to/999thelastpage/jev-and-the-problem-with-ai-that-always-has-an-answer-1k6f", "published_at": "2026-09-27 13:20:35+00:00", "updated_at": "2026-09-27 13:31:14.570314+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-tools"], "entities": ["Jev"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/jev-and-the-problem-with-ai-that-always-has-an-answer", "markdown": "https://wpnews.pro/news/jev-and-the-problem-with-ai-that-always-has-an-answer.md", "text": "https://wpnews.pro/news/jev-and-the-problem-with-ai-that-always-has-an-answer.txt", "jsonld": "https://wpnews.pro/news/jev-and-the-problem-with-ai-that-always-has-an-answer.jsonld"}}