I'm building NextStep — an AI thing that scores your resume against a job and rewrites it to fit. Two features, both completely dependent on the model not lying. And GPT-4o loves to lie.
Two bugs made me stop trusting it:
Score the same resume twice → 87, then 79. Cool, so the number means nothing.
Ask it to "optimize for this DevOps role" → it adds "Managed production Kubernetes clusters" to a guy who's never opened a terminal. That's not a typo, that's getting someone caught lying in an interview.
So I stopped treating the model like it knows things. I treat its output like a request body from some random client: assume it's garbage until I've checked it.
Stop asking the model for the score
The score was drifting because I was asking the model to do math, which is the one thing it's bad at. So I just compute the numbers myself first, in boring Python:
keyword_match = 100.0 * sum(1 for k in keywords if _has_word(resume_text, k)) / len(keywords)
denom = len(required) + 0.5 * len(nice)
skills_match = 100.0 * (required_hits + 0.5 * nice_hits) / denom
_has_word is just a word-boundary regex that doesn't choke on ci-cd and friends. These numbers are the same every single run.
Then I give them to the model — but not as the answer. As hints it's allowed to argue with, as long as it says why:
hint_block = ( "\nDeterministic hints (you may override but must justify in summary):"
f"\n- keyword_match={hints.keyword_match}"
f"\n- skills_match={hints.skills_match}"
f"\n- embedding_similarity={hints.embedding_similarity}"
)
Now the number is stable, but I still get the stuff a regex can't see — like "you wrote React but the role wants Server Components and your bullets don't back that up." I also hardcode the weighting in the prompt so it can't freelance: score = keyword(0.35) + skills(0.35) + experience(0.20) + format(0.10).
Make it annoying to lie
The rewriter is the scary one because its whole job is editing text. So the prompt is blunt, and I don't let it write prose — it has to return typed diffs:
Hard rules:
The actual enforcement is on the server
This is the part that matters. After the model answers, I loop its diffs and throw out anything aimed at a role or bullet that doesn't exist:
for change in raw.changes:
if change.section == "experience" and change.experience_index is not None:
if change.experience_index >= len(resume.content.experience):
log.warning("optimize_drop_invalid_experience_change"); continue
exp = resume.content.experience[change.experience_index]
if change.bullet_index is not None and change.bullet_index > len(exp.bullets):
log.warning("optimize_drop_invalid_bullet_change"); continue
cleaned_changes.append(change)
Model tries to edit a 4th job on a 3-job resume? Gone, logged, user never sees it. And I never touch the original resume — every optimize writes a new row that points back at the source:
sb.table("resumes").insert({
"content": new_content.model_dump(mode="json"),
"source": "optimized",
"source_resume_id": source.id,
}).execute()
Worst case is now "here's a draft you can delete," not "the AI wrecked your resume."
Side effect: it got cheap
Because the inputs are deterministic, caching is trivial — the key is just their hash:
f"ats:v1:{sha256(resume_id + resume_updated_at + job_description)[:32]}" Edit the resume, updated_at changes, cache busts, re-score. Otherwise you pay OpenAI once and reopen the result as many times as you want for free.
What I'd tell past me
Anything you can compute, compute it. Don't make the model guess a number.
Let it override your math, but force it to explain itself. Structured output isn't a nice-to-have — typed diffs make lying awkward and validation easy.
The prompt is a suggestion. The server is the law.
Never mutate the source. Generate a variant.
Basically: the model is just another untrusted client. Validate its JSON like you'd validate anyone else's.
— built this in nextstep-today.com