{"slug": "my-llm-kept-making-stuff-up-on-resumes-heres-how-i-shut-it-down", "title": "My LLM Kept Making Stuff Up on Resumes. Here’s How I Shut It Down.", "summary": "A developer building NextStep, an AI resume scorer and rewriter, solved GPT-4o's hallucination and inconsistency problems by computing scores deterministically in Python, using typed diffs for rewrites, and validating all model output server-side. The approach also made caching trivial, reducing API costs.", "body_md": "I'm building NextStep — an AI thing that scores your resume against a job and rewrites it to fit. Two features, both completely dependent on the model not lying. And GPT-4o loves to lie.\n\nTwo bugs made me stop trusting it:\n\nScore the same resume twice → 87, then 79. Cool, so the number means nothing.\n\nAsk it to \"optimize for this DevOps role\" → it adds \"Managed production Kubernetes clusters\" to a guy who's never opened a terminal. That's not a typo, that's getting someone caught lying in an interview.\n\nSo I stopped treating the model like it knows things. I treat its output like a request body from some random client: assume it's garbage until I've checked it.\n\nStop asking the model for the score\n\nThe score was drifting because I was asking the model to do math, which is the one thing it's bad at. So I just compute the numbers myself first, in boring Python:\n\nkeyword_match = 100.0 * sum(1 for k in keywords if _has_word(resume_text, k)) / len(keywords)\n\ndenom = len(required) + 0.5 * len(nice)\n\nskills_match = 100.0 * (required_hits + 0.5 * nice_hits) / denom\n\n_has_word is just a word-boundary regex that doesn't choke on ci-cd and friends. These numbers are the same every single run.\n\nThen I give them to the model — but not as the answer. As hints it's allowed to argue with, as long as it says why:\n\nhint_block = (\n\n\"\\nDeterministic hints (you may override but must justify in summary):\"\n\nf\"\\n- keyword_match={hints.keyword_match}\"\n\nf\"\\n- skills_match={hints.skills_match}\"\n\nf\"\\n- embedding_similarity={hints.embedding_similarity}\"\n\n)\n\nNow the number is stable, but I still get the stuff a regex can't see — like \"you wrote React but the role wants Server Components and your bullets don't back that up.\" I also hardcode the weighting in the prompt so it can't freelance: score = keyword(0.35) + skills(0.35) + experience(0.20) + format(0.10).\n\nMake it annoying to lie\n\nThe rewriter is the scary one because its whole job is editing text. So the prompt is blunt, and I don't let it write prose — it has to return typed diffs:\n\nHard rules:\n\nThe actual enforcement is on the server\n\nThis is the part that matters. After the model answers, I loop its diffs and throw out anything aimed at a role or bullet that doesn't exist:\n\nfor change in raw.changes:\n\nif change.section == \"experience\" and change.experience_index is not None:\n\nif change.experience_index >= len(resume.content.experience):\n\nlog.warning(\"optimize_drop_invalid_experience_change\"); continue\n\nexp = resume.content.experience[change.experience_index]\n\nif change.bullet_index is not None and change.bullet_index > len(exp.bullets):\n\nlog.warning(\"optimize_drop_invalid_bullet_change\"); continue\n\ncleaned_changes.append(change)\n\nModel tries to edit a 4th job on a 3-job resume? Gone, logged, user never sees it. And I never touch the original resume — every optimize writes a new row that points back at the source:\n\nsb.table(\"resumes\").insert({\n\n\"content\": new_content.model_dump(mode=\"json\"),\n\n\"source\": \"optimized\",\n\n\"source_resume_id\": source.id,\n\n}).execute()\n\nWorst case is now \"here's a draft you can delete,\" not \"the AI wrecked your resume.\"\n\nSide effect: it got cheap\n\nBecause the inputs are deterministic, caching is trivial — the key is just their hash:\n\nf\"ats:v1:{sha256(resume_id + resume_updated_at + job_description)[:32]}\"\n\nEdit the resume, updated_at changes, cache busts, re-score. Otherwise you pay OpenAI once and reopen the result as many times as you want for free.\n\nWhat I'd tell past me\n\nAnything you can compute, compute it. Don't make the model guess a number.\n\nLet it override your math, but force it to explain itself.\n\nStructured output isn't a nice-to-have — typed diffs make lying awkward and validation easy.\n\nThe prompt is a suggestion. The server is the law.\n\nNever mutate the source. Generate a variant.\n\nBasically: the model is just another untrusted client. Validate its JSON like you'd validate anyone else's.\n\n— built this in nextstep-today.com", "url": "https://wpnews.pro/news/my-llm-kept-making-stuff-up-on-resumes-heres-how-i-shut-it-down", "canonical_source": "https://dev.to/risola_me_a79eac9d2622b19/my-llm-kept-making-stuff-up-on-resumes-heres-how-i-shut-it-down-15d2", "published_at": "2026-07-27 20:36:50+00:00", "updated_at": "2026-07-27 21:03:18.416389+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "developer-tools"], "entities": ["NextStep", "GPT-4o", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/my-llm-kept-making-stuff-up-on-resumes-heres-how-i-shut-it-down", "markdown": "https://wpnews.pro/news/my-llm-kept-making-stuff-up-on-resumes-heres-how-i-shut-it-down.md", "text": "https://wpnews.pro/news/my-llm-kept-making-stuff-up-on-resumes-heres-how-i-shut-it-down.txt", "jsonld": "https://wpnews.pro/news/my-llm-kept-making-stuff-up-on-resumes-heres-how-i-shut-it-down.jsonld"}}